How can the BOINC client allocate customized CPU/Memory resources to specific applications


Advanced search

Message boards : News : How can the BOINC client allocate customized CPU/Memory resources to specific applications

1 · 2 · 3 · Next
Author Message
Profile Wenjing Wu
Project administrator
Project developer
Project tester
Project scientist
Avatar
Send message
Joined: 23 Jun 14
Posts: 31
Credit: 2,849,678
RAC: 18
      
Message 4729 - Posted: 14 Jul 2016, 8:42:51 UTC
Last modified: 14 Jul 2016, 8:48:45 UTC

Some of the users might only want to give a certain ratio of their CPU/Memory to a specific application. Take the ATLAS@home Multiple core application ATLAS_MCORE for example, if you want to this app to use less than the CPU/Memory that you allow the BOINC client to use, you can do it through creating a configuration file app_config.xml in your project directory:


You can limit the MultiCore-App by using an app_config.xml.

Below is an example to limit ATLAS_MCORE to use only 4 Cores:

<app_config> <app_version> <app_name>ATLAS_MCORE</app_name> <avg_ncpus>4.000000</avg_ncpus> <plan_class>vbox_64_mt_mcore</plan_class> <cmdline>--memory_size_mb 5300</cmdline> </app_version> </app_config>


You should change these two lines to your needs:

<avg_ncpus>4.000000</avg_ncpus> <cmdline>--memory_size_mb 5300</cmdline>


Memory usage calculated by the ATLAS_MCORE app is by this formula:

memory = 1300 + (1000* NumerOfCores)

so it is 5300MB for 4 cores.


Thanks to Yeti for giving this recipe.

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5023 - Posted: 16 Aug 2016, 17:06:30 UTC - in response to Message 4729.

The formula for Memory-Calculation is no longer valid.

Please use:

David Cameron wrote:

We noticed that many tasks using 2 or 3 cores were failing because not enough memory was available so we have changed the formula to

Memory = 2.5 GB + 0.8 x no of cores

This gives more memory when there are fewer cores but keeps roughly the same memory as before for the higher core numbers.

Please adjust app_config.xml if you have set one up.

RussellG
Send message
Joined: 17 May 16
Posts: 3
Credit: 31,658
RAC: 0
    
Message 5150 - Posted: 31 Aug 2016, 21:12:17 UTC - in response to Message 5023.

How do write an app.config file for Atlas single core and multicore? I want to restrict the number of concurrent Atlas tasks run by single core and multicore, and also the number of processors used by multicore - on my 8GB 4core i5 PC set to use 75% of cores for BOINC I find that >1 single core task makes the machine unresponsive for long periods, and so far any multicore task fails to complete. My attempt below doesn't seem to work, is there something wrong with it?

<app_config>
<app>
<name>ATLAS</name>
<max_concurrent>1</max_concurrent>
</app>
<app_version>
<app_name>ATLAS_MCORE</app_name>
<max_concurrent>1</max_concurrent>
<avg_ncpus>2.000000</avg_ncpus>
<plan_class>vbox_64_mt_mcore</plan_class>
<cmdline>--memory_size_mb 3300</cmdline>
</app_version>
</app_config>

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5151 - Posted: 31 Aug 2016, 21:27:12 UTC - in response to Message 5150.

My attempt below doesn't seem to work, is there something wrong with it?

What does it or does it not ?

You noticed that changes of number of cores only work for new downloaded WUs?

RussellG
Send message
Joined: 17 May 16
Posts: 3
Credit: 31,658
RAC: 0
    
Message 5152 - Posted: 31 Aug 2016, 21:44:22 UTC - in response to Message 5151.

It doesn't change the number of cores in newly downloaded WUs. I applied it yesterday and have restarted the machine but today I was still getting 3 core jobs. It restricts the total no of concurrent Atlas jobs to 1 successfully - I was doing that for single core. I was trying to extend the app.config to tune multicore.

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5153 - Posted: 31 Aug 2016, 21:55:38 UTC

you can try following

force BOINC to read config-files
check logs what is to find there regarding Atlas app_config.xml

If you can wait until I'm next in the office I can post one of mine

computezrmle
Send message
Joined: 29 Oct 14
Posts: 54
Credit: 1,137,404
RAC: 35
      
Message 5156 - Posted: 1 Sep 2016, 6:35:28 UTC - in response to Message 5150.

Check the filename:
You wrote app.config but it must be app_config.xml.

Check the file format:
<max_concurrent>N</max_concurrent> is not allowed in the <app_version> section.
Strictly follow the format that is given in the BOINC documentation.

Then:
Follow Yeti┬┤s advice.

David Cameron
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 13 May 14
Posts: 251
Credit: 2,028,082
RAC: 30
      
Message 5157 - Posted: 1 Sep 2016, 7:45:12 UTC - in response to Message 5156.

--memory_size_mb 3300


Please note that the memory formula has changed and it should be 4100 for 2 cores.

On my Linux desktop I configure BOINC to use 5 cores, running 1 single core and 1 4-core multicore task. To do this I have the following app_config.xml:

<app_config> <app> <name>ATLAS</name> <max_concurrent>2</max_concurrent> </app> <app_version> <app_name>ATLAS_MCORE</app_name> <avg_ncpus>4.000000</avg_ncpus> <plan_class>vbox_64_mt_mcore</plan_class> <cmdline>--memory_size_mb 5700</cmdline> </app_version> </app_config>


and cc_config.xml in the BOINC main directory:

<cc_config> <options> <ncpus>5</ncpus> </options> </cc_config>


However this affects other projects too. What I would like is to have 8 cores for BOINC, but 5 cores for ATLAS in the configuration above but I haven't found a way to do that.

computezrmle
Send message
Joined: 29 Oct 14
Posts: 54
Credit: 1,137,404
RAC: 35
      
Message 5159 - Posted: 1 Sep 2016, 8:52:48 UTC - in response to Message 5157.

If you run only one project, e.g. ATLAS, I would expect this settings should work:


in cc_config.xml:

<ncpus>5</ncpus>



app_config.xml (slightly different from your app_config.xml):

<app_config> <app> <name>ATLAS</name> <max_concurrent>1</max_concurrent> </app> <app> <name>ATLAS_MCORE</name> <max_concurrent>1</max_concurrent> </app> <app_version> <app_name>ATLAS_MCORE</app_name> <plan_class>vbox_64_mt_mcore</plan_class> <avg_ncpus>4</avg_ncpus> <cmdline>--memory_size_mb 5700</cmdline> </app_version> <project_max_concurrent>2</project_max_concurrent> </app_config>



However this affects other projects too. What I would like is to have 8 cores for BOINC, but 5 cores for ATLAS in the configuration above but I haven't found a way to do that.

May be possible with a very complex setup where you run more than one BOINC client or BOINC inside a couple of VMs or ...

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5162 - Posted: 1 Sep 2016, 12:37:24 UTC - in response to Message 5157.
Last modified: 1 Sep 2016, 12:38:51 UTC

What I would like is to have 8 cores for BOINC, but 5 cores for ATLAS in the configuration above but I haven't found a way to do that.

There are two way to do this:

  1. Remove NCPUs from cc_config.xml
    Allow BOINC to use 100% of CPUs
    Use this app_config.xml for Atlas:
    <app_config> <app> <name>ATLAS</name> <max_concurrent>1</max_concurrent> </app> <app_version> <app_name>ATLAS_MCORE</app_name> <avg_ncpus>4.000000</avg_ncpus> <max_ncpus>4.000000</max_ncpus> <plan_class>vbox_64_mt_mcore</plan_class> <cmdline>--memory_size_mb 7500</cmdline> <dont_throttle/> <is_wrapper/> <needs_network/> </app_version> </app_config>

    This should give you 1x Atlas-Single-Core, 1x Atlas-Multi-Core with 4 Cores and 3 Cores free for other projects.

    What can happen is, that BOINC tries to start 2x Atlas-Multi-Core e.g. when Single-Tasks doesn't have work or the other project don't have work.

  2. Do the same as 1) but set BOINC to use only 87,5 % of available CPUs; this gives you 7 Cores.
    As you are running 4-Core-WUs with Multicore BOINC can't start 2x Multi-Core because it would need 8 cores for this


If this doesn't work for you then you are hit by the memory-bug of the scheduler I mentioned already in other threads.

Edit: Memory is a little bit high but I like to spent a little bit more memory

Profile Harm Joris ten Napel
Send message
Joined: 27 Nov 14
Posts: 40
Credit: 1,696,115
RAC: 65
      
Message 5170 - Posted: 1 Sep 2016, 22:57:18 UTC - in response to Message 4729.

I gave BOINC 4 cores and the VM has allocated 5700 MB... It's not a problem my system has 32G.

David Cameron
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 13 May 14
Posts: 251
Credit: 2,028,082
RAC: 30
      
Message 5171 - Posted: 2 Sep 2016, 13:44:53 UTC - in response to Message 5162.

Thanks a lot Yeti! This recipe worked perfectly.

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5172 - Posted: 2 Sep 2016, 15:13:17 UTC - in response to Message 5171.

Thanks a lot Yeti! This recipe worked perfectly.

Glad that I could help :-)

Toby Broom
Send message
Joined: 1 Jul 14
Posts: 70
Credit: 12,032,688
RAC: 88
      
Message 5175 - Posted: 2 Sep 2016, 21:12:05 UTC
Last modified: 2 Sep 2016, 21:13:48 UTC

<app_config>
<app>
<name>ATLAS</name>
<max_concurrent>6</max_concurrent>
</app>
<app_version>
<app_name>ATLAS_MCORE</app_name>
<max_concurrent>1</max_concurrent>
<avg_ncpus>6.000000</avg_ncpus>
<plan_class>vbox_64_mt_mcore</plan_class>
<cmdline>--memory_size_mb 7300</cmdline>
</app_version>
</app_config>

I tried to have this but it didn't work, can I have a 6x1 or 1x6 config?

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5179 - Posted: 3 Sep 2016, 5:35:46 UTC - in response to Message 5175.

... can I have a 6x1 or 1x6 config?

Hm, it is not clear what you really want:
So far as I know it is not really possible.

But you can do a trick: On Project-Preferences of Atlas you could allow only MultiCore and then trigger "If the choosen app doesn't have work, send work from other application".

This will focus you on MultiCore but it still can happen that some SingleCore-WUs make it to your PC

Is there anything more that make you say your app_config doesn't work ?

Toby Broom
Send message
Joined: 1 Jul 14
Posts: 70
Credit: 12,032,688
RAC: 88
      
Message 5181 - Posted: 3 Sep 2016, 9:09:22 UTC
Last modified: 3 Sep 2016, 9:10:50 UTC

I want it to run six ATLAS tasks if I got those and 1 MCORE with six cpus if I got one of those.

So I'm always using 6 cores either in one or six VM depending on what project sends.

I think I'll just block the single core and go with the multi for now.

You think this would be valid?

<app_config>
<app>
<name>ATLAS</name>
<max_concurrent>6</max_concurrent>
</app>
<app>
<name>ATLAS_MCORE</name>
<max_concurrent>1</max_concurrent>
</app>
<app_version>
<app_name>ATLAS_MCORE</app_name>
<plan_class>vbox_64_mt_mcore</plan_class>
<avg_ncpus>6</avg_ncpus>
<cmdline>--memory_size_mb 5700</cmdline>
</app_version>
</app_config

Profile Yeti
Avatar
Send message
Joined: 20 Jul 14
Posts: 699
Credit: 22,597,832
RAC: 211
      
Message 5183 - Posted: 3 Sep 2016, 15:42:00 UTC - in response to Message 5181.

Toby, I found this statement from you in another thread:

Toby wrote:

My expectation is I use 6cores for the project either as 6x1 cores or 1x6 core, however I see it kicked my machine running 6x6core


This can only happen if you have a 36 core machine.

At the moment I don't know any way to configure BOINC to run either 1x6 or 6x1 Cores for the same project.

Much is depending on the number of cores available and your concret wishes.

I can not tell you if your app_config.xml will work but it is easy to find it out.

Install it, force BOINC to read config files and then check logs. If something is wrong in app_config.xml you will find a hint in the logs. If it is okay, then you see only a note that Atlas has an app_config.xml

Toby Broom
Send message
Joined: 1 Jul 14
Posts: 70
Credit: 12,032,688
RAC: 88
      
Message 5198 - Posted: 5 Sep 2016, 17:36:45 UTC - in response to Message 5183.

I messed with it quite a bit, just with the MCORE tasks, I can't really get it to work the way I want so I'll do some work for other projects with my computer.

RussellG
Send message
Joined: 17 May 16
Posts: 3
Credit: 31,658
RAC: 0
    
Message 5227 - Posted: 7 Sep 2016, 6:41:48 UTC - in response to Message 5157.

thanks to you and others above for advice. Removing the <max_concurrent>1</max_concurrent> from the app_version section seemed to do the trick and I adjusted the memory to the correct value. With these settings my PC has successfully run a couple of multi-core jobs using 2 cores.

However most abort with messages such as 'Aborting task XXXX: exceeded disk limit: 6258.07MB > 5722.05MB'. This is despite there being about 500GB of disk space available to BOINC. I tried looking this up but can't find anything I can do about it.

In addition when a multi-core job is running the machine is frequently unresponsive when I try to use it for other purposes. Seems to me that Atlas multi-core needs a beefier machine than I have available, so I have regretfully stopped trying to run multi-core jobs.

adam laptop
Send message
Joined: 13 Nov 16
Posts: 1
Credit: 0
RAC: 0
Message 5623 - Posted: 13 Nov 2016, 6:50:34 UTC - in response to Message 4729.

What should I put in the 2 lines. I'm unfamiliar with writing in code.

Adam

1 · 2 · 3 · Next

Message boards : News : How can the BOINC client allocate customized CPU/Memory resources to specific applications