Message boards : Number crunching : limiting number of tasks downloaded
Message board moderation

To post messages, you must log in.

AuthorMessage
greg_be

Send message
Joined: 28 Dec 08
Posts: 318
Credit: 4,148,677
RAC: 2,010
Message 45383 - Posted: 23 Sep 2021, 7:45:24 UTC

I came back here to pick your brains about this.
For another project, when using max_concurrent, I am getting flooded with over 800-900 tasks at one time which equals over 1 year of work.

Do you guys have any idea what command if any can be used to limit the number of tasks downloaded?

Say I want to set it at 10 in queue from this project, how can I do that?
The projects page is the most basic and does allow you to set how many CPU's or anything else related to your computer. It's all or nothing.
ID: 45383 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,951,161
RAC: 137,233
Message 45384 - Posted: 23 Sep 2021, 8:20:50 UTC - in response to Message 45383.  

... what command if any can be used to limit the number of tasks downloaded?

There's no command available doing this.
BOINC client sends a work request to the server that includes a number of work seconds not covered by tasks in it's work buffer.
Based on that the server estimates how many tasks can be send.


As mentioned in another thread there might be a bug in the BOINC client code in connection with <max_concurrent> and <project_max_concurrent>:
https://github.com/BOINC/boinc/issues/4322

You can either wait until a new BOINC client is published without that bug or you dig into the BOINC source code, write a patch and deliver it to the community.
ID: 45384 · Report as offensive     Reply Quote
greg_be

Send message
Joined: 28 Dec 08
Posts: 318
Credit: 4,148,677
RAC: 2,010
Message 45385 - Posted: 23 Sep 2021, 10:53:42 UTC - in response to Message 45384.  
Last modified: 23 Sep 2021, 10:59:22 UTC

... what command if any can be used to limit the number of tasks downloaded?

There's no command available doing this.
BOINC client sends a work request to the server that includes a number of work seconds not covered by tasks in it's work buffer.
Based on that the server estimates how many tasks can be send.


As mentioned in another thread there might be a bug in the BOINC client code in connection with <max_concurrent> and <project_max_concurrent>:
https://github.com/BOINC/boinc/issues/4322

You can either wait until a new BOINC client is published without that bug or you dig into the BOINC source code, write a patch and deliver it to the community.


I was hoping NCPUS would do something, but that command does not seem to work.

MaxMaximus in that post is describing word for word what I experience.

I could only do modifications in BASIC long ago. I don't understand todays languages.
I never understood PASCAL or FORTRAN back in the day either.
So I guess I will just have to wait and manually limit the work in that project.



Oh well......thanks anyway
ID: 45385 · Report as offensive     Reply Quote
Harri Liljeroos
Avatar

Send message
Joined: 28 Sep 04
Posts: 674
Credit: 43,152,472
RAC: 15,698
Message 46581 - Posted: 6 Apr 2022, 8:08:52 UTC

I have some observations on server side limits for downloaded tasks. I am using <max_concurrent> tag on my app_config files so the bug kicks in and I get (Boinc client requests) more tasks than my cache settings (1 d + 0.1 d) should say. My 'Max number of Tasks' is set to 'No limit' as the maximum numerical value you can select is 8 and that is too low for my 8/16 CPU host. I am running all but CMS tasks on Windows computers, so VBox is used for Atlas and Theory.

The limits I see on server side seem to be:
- Atlas 16 tasks
- Theory 80 tasks
- sixtrack 80 tasks
- sixtracktest limit seems higher but I haven't reached that limit yet
- CMS limit used to be several hundreds but I haven't run that recently so it might have changed
- All combined (at least without CMS) 200 tasks

The values may depend on number of CPUs a host has available but this is for my 4/8 and 8/16 host.

Please share if you see different values.
ID: 46581 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 46582 - Posted: 6 Apr 2022, 14:20:11 UTC

I'm not using No limit for Max # of jobs, but can imagine what's going on.
Atlas' run times are rather stable, so BOINC can calculate very well how many ATLAS-tasks it needs to fill your cache,
but the runtimes for Theory and sixtrack are varying very strongly. After a bunch of short tasks, you suddenly could get too much tasks.
You could reduce the 1 day + 0.1 day cache buffer to the half. Deadlines should then not be a problem.

One other thing you could consider, already using an app_config.xml, is to increase the number of used cores for your ATLAS-tasks to e.g. 4.
ID: 46582 · Report as offensive     Reply Quote
Harri Liljeroos
Avatar

Send message
Joined: 28 Sep 04
Posts: 674
Credit: 43,152,472
RAC: 15,698
Message 46583 - Posted: 6 Apr 2022, 14:43:12 UTC - in response to Message 46582.  

With my current settings I do not have problems with deadlines as I limit the the number of Atlas tasks in my cache on the slower computer. I try to keep the amount in max 7 by switching Atlas on and off in a few day interval. It means I have to do some micro managing but that's OK. And the server seems to resend Atlas tasks to Agile Boincers already after 5 days even there are still 2 days left to deadline (which by the way is one day shorter in Boinc client than on server).

The faster computer can handle the deadlines for all subprojects I am doing for LHC (i.e. no CMS).
ID: 46583 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,137,814
RAC: 105,274
Message 46584 - Posted: 6 Apr 2022, 16:56:32 UTC - in response to Message 46583.  
Last modified: 6 Apr 2022, 17:08:19 UTC

Atlas, CMS and Theory enabled.
max. Tasks 8 - CPU's 1.
Some waiting after finishing one task, because in this moment, only 7 are running.
No problems always.
Not tested with more than 8 CPU's.
No HyperThreading, but RedHat - Squid inside a CentOS8 or CentOS9 VM.
ID: 46584 · Report as offensive     Reply Quote

Message boards : Number crunching : limiting number of tasks downloaded


©2024 CERN