Message boards : Number crunching : Balance between experiments
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile jjv

Send message
Joined: 28 Jun 06
Posts: 5
Credit: 43,565,090
RAC: 7,761
Message 28836 - Posted: 9 Feb 2017, 9:42:18 UTC

I've been wondering what determines which WUs my machines receive. All of my computers seem to run pure LHCb with occasional dashes of sixtrack. I can't recall the last time I've seen a CMS unit let alone any others. My settings should allow all. I've even set up my app_config in a way that limits simultaneous work from a single experiment but not several.
Looking at the status page there should be work available under multiple experiments...

JJ
ID: 28836 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 708
Credit: 4,336,250
RAC: 0
Message 28837 - Posted: 9 Feb 2017, 14:55:04 UTC
Last modified: 9 Feb 2017, 14:55:29 UTC

I am getting mostly LHCb tasks on the Windows PC and the 64-Bit Linux Sun. On the 32-bit Linux box I get also Theory tasks but they all fail after 10 minutes with a kernel panic.
Tullio
ID: 28837 · Report as offensive     Reply Quote
Profile jjv

Send message
Joined: 28 Jun 06
Posts: 5
Credit: 43,565,090
RAC: 7,761
Message 28845 - Posted: 12 Feb 2017, 10:45:08 UTC

It kind of seems that when requesting work it fills the work buffer from the first experiment with work available. As I run a multiday buffer it makes sense that I get lots of WUs from a single experiment. This however doesn't explain why it's always the same experiment. Personally I would prefer a spread of WUs. This is however not really an issue since according to the status page work is getting done on all experiments.
What with all CERN BOINC projects being assimilated under a single project I'm kind of wondering if everything is working as intended.

Basically I'm asking what actually happens when I tick the 'all applications' box in my preferences.

JJ
ID: 28845 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 28846 - Posted: 12 Feb 2017, 13:35:21 UTC - in response to Message 28836.  
Last modified: 12 Feb 2017, 14:05:35 UTC

All of my computers seem to run pure LHCb with occasional dashes of sixtrack.

There are explanations on that subject somewhere around here; probably on the LHCb topic. In short, if you check LHCb and also enable "Run test applications", then you will get only LHCb until the well runs dry, and only then will you get the others. So if you want the other projects, do not enable "Run test applications". (But you won't get LHCb in that case, until they are out of beta and have reached release status.)
ID: 28846 · Report as offensive     Reply Quote
Profile jjv

Send message
Joined: 28 Jun 06
Posts: 5
Credit: 43,565,090
RAC: 7,761
Message 28857 - Posted: 13 Feb 2017, 18:24:29 UTC - in response to Message 28846.  

That sounds kind of counter-intuitive. So if I allow test application I get nothing but? In other projects it certainly doesn't work that way.
Although now that you mentioned it I vaguely recall discussion about this.

JJ
ID: 28857 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 798
Credit: 644,764,770
RAC: 232,088
Message 28858 - Posted: 13 Feb 2017, 19:13:39 UTC

I wrote about this in the past.

I assume the project team does the balancing (if any) on there side. We don't have any balancing option other than the options to disable a project.

I see the same as you if left to it's own devices I get mostly LHCb tasks.
ID: 28858 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,951,690
RAC: 137,033
Message 29659 - Posted: 26 Mar 2017, 19:53:56 UTC

As with the consolidation of the different CERN projects a great disadvantage of BOINC becomes important.
It´s the balance control between subprojects.

Example:
If I try to run ATLAS and CMS on the same host at the same time with equal priority I expect to use the following settings:

<app_config>
  <app>
    <name>ATLAS</name>
    <max_concurrent>1</max_concurrent>
  </app>
  <app>
    <name>CMS</name>
    <max_concurrent>1</max_concurrent>
  </app>
  <project_max_concurrent>2</project_max_concurrent>
</app_config>


Together with the preferences on the project´s webpage:
Max # of jobs: 2


I would expect that the server sends 1 ATLAS WU and 1 CMS WU to run them both on my host.


Instead on one host the server sent me a second CMS WU that is now waiting until the first one has finished.
As I don´t know any control within the BOINC client that has an influence on the selected WU type I would like to forward this as a question to the audience or a request to be solved by the project developers.
ID: 29659 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 29660 - Posted: 26 Mar 2017, 20:04:19 UTC - in response to Message 29659.  

As I don´t know any control within the BOINC client that has an influence on the selected WU type I would like to forward this as a question to the audience or a request to be solved by the project developers.

That is the way it is on World Community Grid. You can select the projects, but not the balance between them. And it is now also the case for the most part on Climate Prediction Network. So I think it is by their design; they get to chose the ratios. I can live with that, since they presumably know which ones are the most urgent/important/etc.
ID: 29660 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,951,690
RAC: 137,033
Message 29661 - Posted: 26 Mar 2017, 20:25:50 UTC - in response to Message 29660.  

... I can live with that, since they presumably know which ones are the most urgent/important/etc.

I agree from the project´s perspective.
I do not agree from the user´s perspective as I try to find the individual balance between ressources and reliability/performance on each of my hosts.
ID: 29661 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 29663 - Posted: 26 Mar 2017, 21:39:45 UTC - in response to Message 29661.  
Last modified: 26 Mar 2017, 21:41:27 UTC

I do not agree from the user´s perspective as I try to find the individual balance between ressources and reliability/performance on each of my hosts.

Yes, it can cause problems. I sometimes use a ramdisk, and have to limit the number of certain projects to fit in it, though I am OK on LHC now.
ID: 29663 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 29669 - Posted: 27 Mar 2017, 8:58:43 UTC - in response to Message 29661.  

... I can live with that, since they presumably know which ones are the most urgent/important/etc.

I agree from the project´s perspective.
I do not agree from the user´s perspective as I try to find the individual balance between ressources and reliability/performance on each of my hosts.

App_config is not designed for sub-project balancing and not for balancing at all.
It's designed by the BOINC-team (and not any project team) to reduce the maximum number of cores to use for a (sub-)project.
Main reason was/is the available RAM or also some projects use huge disk i/o's like WCG's Clean Energy Project.
Meanwhile the purpose is extented for RAM-usage and # of threads and other command line switches.

A workaround for your balancing problem is first to load 12 CMS and thereafter 12 ATLAS (mind the deadline) and then set 'No New Tasks' until one of the sub-project tasks in your queue get low.
ID: 29669 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,951,690
RAC: 137,033
Message 29672 - Posted: 27 Mar 2017, 11:42:19 UTC - in response to Message 29669.  

App_config is not designed for sub-project balancing and not for balancing at all.
I know.


A workaround for your balancing problem is first to load 12 CMS and thereafter 12 ATLAS (mind the deadline) and then set 'No New Tasks' until one of the sub-project tasks in your queue get low.
Doing this needs a lot of babysitting and has also influence on other projects on the same host, i.e. "cache full" message.
ID: 29672 · Report as offensive     Reply Quote

Message boards : Number crunching : Balance between experiments


©2024 CERN