Message boards : Number crunching : ATLAS Simulation.
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 30955 - Posted: 22 Jun 2017, 20:11:00 UTC
Last modified: 22 Jun 2017, 20:22:41 UTC

I have an ATLAS simulation task running on my system. I was alerted to the issue I'm seeing because I could see a lot of waiting to run tasks, and eventually came across the ATLAS, which says it is running on 8 CPU's, ie. all of them, (multithreaded). I have removed this task class from my profile, as I am not happy with the way it works. Question is, is that the only task class which does this, or am I better off just dropping LHC from my projects list?

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 30955 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 421
Credit: 105,157,144
RAC: 63,993
Message 30958 - Posted: 22 Jun 2017, 22:55:05 UTC - in response to Message 30955.  

Sixtrack is easy to run

All other Subprojects need VirtualBox.

At the moment so far as I know is Atlas the only one with MulticoreApps, but others are preparing MultiCoreApps.

If you want to run these Apps, you can work through my checklist for Atlas and other VB-Apps: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4161&postid=29359#29359


Supporting BOINC, a great concept !
ID: 30958 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 625
Credit: 392,376,593
RAC: 163,076
Message 30960 - Posted: 23 Jun 2017, 5:55:21 UTC

The Theory task are less intensive on computer resources, somewhere in between Altas and Sixtrack.

http://lhcathome.web.cern.ch/what-specification-each-application
ID: 30960 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 30962 - Posted: 23 Jun 2017, 6:09:34 UTC - in response to Message 30960.  

I have only removed ATLAS from my sub-projects list, so far. I need to find out more about how BOINC handles multithreaded applications. If it treats a task running on 8 cores for an hour the same as a task running on one core for 8 hours for example.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 30962 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 625
Credit: 392,376,593
RAC: 163,076
Message 30978 - Posted: 23 Jun 2017, 16:41:58 UTC

You can set your web prefernces to only allow 1 core, to test once your happy then you can get into more advanced if you desire.
ID: 30978 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31006 - Posted: 24 Jun 2017, 7:01:00 UTC - in response to Message 30978.  

That sounds good, I'll look for that, maybe I can continue the project.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31006 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1343
Credit: 25,169,706
RAC: 23,353
Message 31007 - Posted: 24 Jun 2017, 8:17:46 UTC - in response to Message 30962.  

... I need to find out more about how BOINC handles multithreaded applications. If it treats a task running on 8 cores for an hour the same as a task running on one core for 8 hours for example.

What exactly do you mean by "how BOINC handles ..."

I have had single-core and multi-core tasks running - from a BOINC point of view, I couldn't notice any difference.
ID: 31007 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31011 - Posted: 24 Jun 2017, 10:27:12 UTC
Last modified: 24 Jun 2017, 10:29:59 UTC

You might not notice any difference, but I might. The example I gave earlier in the thread, which you quote, illustrates a difference I want to find out more about, the task runs for an hour on 8 CPU's, stopping all other projects - does the work scheduler class that as an hour or eight. I can think of numerous situations which I would like to know more about.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31011 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1666
Credit: 98,713,396
RAC: 86,730
Message 31014 - Posted: 24 Jun 2017, 11:41:28 UTC - in response to Message 31011.  

ATLAS WUs have (simply spoken) 3 main phases.

Phase 1
VM setup and download of the ATLAS job.
This phase uses 1 core although your BOINC client marks all configured cores as busy and therefore not usable by other projects.
This phase needs a walltime between 5 min and >30 min depending on your internet bandwidth.

Phase 2
Calculation phase
This phase runs (normaly) 50 subjobs and uses all configured cores.
Each subjob runs between 60 s and (very rare) 1200 s.
At the end of this phase 1 or more cores may be idle as there are no more subjobs to be calculated.

Phase 3
Stage Out
This phase uses only 1 core and prepares the upload file.
Other cores are not usable by other projects.


Conclusion
ATLAS WUs with a low core setting are more efficient.
ATLAS WUs with a high core setting are faster.
The initial phase highly depends on your internet bandwidth.


NB: Other LHC subproject (CMS, LHCb, Theory) use a time based calculation scheme and are not comparable to ATLAS.
ID: 31014 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31038 - Posted: 25 Jun 2017, 11:25:47 UTC

I have changed the project to use at most 25% CPU's. I'll watch that and see if it is acceptable.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31038 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31041 - Posted: 25 Jun 2017, 12:32:00 UTC
Last modified: 25 Jun 2017, 12:32:50 UTC

Sorry, it would not let me edit my earlier post.

Why does the project make nultithreaded app? The deadlines are fairly long, so it can't be a time issue. The way the app works there are times when it cannot possibly use all the resources it has allocated, therefore wasting them and denying them to other projects. There are folk out there that see the issue I've seen and rather than resolving it, just drop the project.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31041 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1666
Credit: 98,713,396
RAC: 86,730
Message 31054 - Posted: 25 Jun 2017, 19:44:43 UTC - in response to Message 31041.  

Why does the project make nultithreaded app? ...

The ATLAS multithreaded app (x cores) needs less RAM compared to x 1-core apps.

Example:
8 1-core apps request 8x3.4 GB => 27.2 GB => needs a computer with 32 GB RAM
2 4-core apps request 2x5.8 GB => 11.6 GB => runs on a computer with 16 GB RAM

That is a very short explanation as there are a couple of additional pitfalls.
You may read the MB and the project documentation to get more information.
Yeti's checklist would be a good starting point.

In general it can be said that the multithreaded app allows volunteers to optimize ATLAS for the use on their computer.


The way the app works there are times when it cannot possibly use all the resources it has allocated, therefore wasting them and denying them to other projects. There are folk out there that see the issue I've seen and rather than resolving it, just drop the project.

I'm sure that the project team knows this and tries to satisfy both, the project's needs to solve the scientific problems as well as the volunteer's request for an easy to use application.
ID: 31054 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31073 - Posted: 26 Jun 2017, 8:54:25 UTC
Last modified: 26 Jun 2017, 9:39:25 UTC

>>> Example:
>>> 8 1-core apps request 8x3.4 GB => 27.2 GB => needs a computer with 32 GB RAM
>>> 2 4-core apps request 2x5.8 GB => 11.6 GB => runs on a computer with 16 GB RAM

No. You are making a totally incorrect comparison here, you are assuming that the 8 1 core apps are all running at the same time. With a regular BOINC set up with a collection of projects running at the same priority, that would not happen.

I have the checklist open in another tab and have been working my way through it.

I have limited the project to 2 cores, at least, I believe I have understood the control mechanism sufficiently to have done that. I'll see. The ATLAS job class is re-enabled.

>>> Use at most 25% of the CPUs

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31073 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1666
Credit: 98,713,396
RAC: 86,730
Message 31075 - Posted: 26 Jun 2017, 9:57:57 UTC - in response to Message 31073.  

With a regular BOINC set up with a collection of projects ...

What if I decide to setup my computer exclusively for ATLAS?
Who/What will deny it?

... you are assuming that the 8 1 core apps are all running at the same time ...

Why not?

The original question was:
Why does the project make nultithreaded app?

My example wasn't thought to be a complete real setup ("additional pitfalls").
It should point out that ATLAS can be configured in different ways to satisfy different needs.
The multicore app is just an option that supports flexibility.
ID: 31075 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 421
Credit: 105,157,144
RAC: 63,993
Message 31076 - Posted: 26 Jun 2017, 10:04:06 UTC

You can easily make Atlas running 1-Core-WUs:

Go to Project-Preferences and edit the used venue

Change "Max # of CPUs for this project" to 1 and new downloaded WUs will be 1-Core-WUs


Supporting BOINC, a great concept !
ID: 31076 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31079 - Posted: 26 Jun 2017, 11:42:56 UTC - in response to Message 31076.  

As you can see, I have already done that by changing it to use at most 25% of CPU's, it should only use 2 cores, I can live with that.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31079 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1343
Credit: 25,169,706
RAC: 23,353
Message 31081 - Posted: 26 Jun 2017, 11:45:51 UTC - in response to Message 31073.  

>>> Example:
>>> 8 1-core apps request 8x3.4 GB => 27.2 GB => needs a computer with 32 GB RAM
>>> 2 4-core apps request 2x5.8 GB => 11.6 GB => runs on a computer with 16 GB RAM

No. You are making a totally incorrect comparison here, you are assuming that the 8 1 core apps are all running at the same time. With a regular BOINC set up with a collection of projects running at the same priority, that would not happen.


Why is the comparison totally incorrect?
I have had 8 1-core ATLAS tasks besides 2 GPUGRID tasks.
It worked well.
ID: 31081 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 31082 - Posted: 26 Jun 2017, 11:50:36 UTC

>>> Who/What will deny it?

Nobodys is trying to deny you anything. My gripe is that it seemed to be assumed that everyone would want to be like that. Like probably 99% of BOINC users, I am attached to multiple projects. Makes sense, sometimes a server is down, or job generation was interrupted, etc. Always makes sense to run a group of different projects from different servers. The alternative can be periods running nothing just consuming electricity for no reason.

>>> Why not?

Because on systems with multiple projects attached running at comparable quota levels the schedulaer doesn't do that. You tell me how or why it should.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 31082 · Report as offensive     Reply Quote
Sid

Send message
Joined: 26 Jul 12
Posts: 18
Credit: 2,388,108
RAC: 1,638
Message 31459 - Posted: 17 Jul 2017, 13:12:05 UTC
Last modified: 17 Jul 2017, 13:13:02 UTC

I have couple of 16/24 cores computers. I'd like to enforce Atlas application to use almost all of them - say, 14 cores or 22 depending of a host. However, I'm receiving only 8C WUs. Is it kind of limit for Atlas? I've enough RAM and I have set everything I have found on preference pages to maximum except number of CPUs.
ID: 31459 · Report as offensive     Reply Quote
gyllic

Send message
Joined: 9 Dec 14
Posts: 202
Credit: 2,533,390
RAC: 0
Message 31461 - Posted: 17 Jul 2017, 13:29:27 UTC - in response to Message 31459.  

I have couple of 16/24 cores computers. I'd like to enforce Atlas application to use almost all of them - say, 14 cores or 22 depending of a host. However, I'm receiving only 8C WUs. Is it kind of limit for Atlas? I've enough RAM and I have set everything I have found on preference pages to maximum except number of CPUs.

The maximum core number per task, i.e. atlas vm, is 8. you may consider running e.g. 2 or 3 8 core vms at the same time (if you have enough ram). But the efficiency of 8 core wus is not the best (as far as i can remember) so you may also consider running for example 4 4core tasks (again if there is enough ram in your system) instead. There are a lot of posts here how u can achive that (app_config.xml)
ID: 31461 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : ATLAS Simulation.


©2021 CERN