Message boards :
ATLAS application :
why only 3 concurrent 1-core ATLAS tasks possible on 32GB machine?
Message board moderation
Author | Message |
---|---|
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
In the recent past, I have tried all kinds of things in the various settings to make it possible to process 4 1-core ATLAS tasks; however only 3 tasks can be run concurrently, the fourth one remains in "ready to start" status. The PC has 32GB RAM, processor 6+6HT cores. In the BOINC computing preferences, RAM usage is set to 100% now (before it was 95%). Disk space also more than sufficient. In the app_config.xml the max concurrent ATLAS jobs is set to 4; RAM is set to 6400MB (given the fact that from console 3 I could see that towards the end of a task, it uses up to 6GB, sometimes even a little more). Right now, where 3 1-core ATLAS tasks are running, MemInfo shows a total RAM usage of about 20GB (there are also 2 GPUGRID tasks running, which take some 300MB each). So there should be plenty of free RAM available for a 4th ATLAS task. Also, I remember that at times of the former ATLAS project (when it was seperate from LHC) I could run as many ATLAS tasks as possible concurrently, until the 32GB RAM was pretty much used up. Why is this not possible now? Is there something wrong in my settings? Do I overlook something? Would be nice to get some advice from the experts. |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
and what's most interesting, almost unbelievable: although I cannot run 4 1-core ATLAS tasks, I can run 4 2-core ATLAS tasks conurrently. Does anyone understand this? For me, a conondrum |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
and what's most interesting, almost unbelievable: although I cannot run 4 1-core ATLAS tasks, I can run 4 2-core ATLAS tasks conurrently.unfortunately, the bad thing is that a 2-core task uses up close to 7GB RAM (in contrast to a 1-core task which takes slightly below 6GB), thus (with 2 GPUGRID tasks running, the system itself, and the usual minimum other apps like antivirus etc.) bringing the total RAM usage to approx. 31,5 GB which is not good at all. |
Send message Joined: 15 Jun 08 Posts: 2386 Credit: 222,989,970 RAC: 136,507 |
... a 2-core task uses up close to 7GB RAM (in contrast to a 1-core task which takes slightly below 6GB), ... A 2-core WU should not request such high amount of RAM. Did you get that number from your host or from the VM? Be so kind as to post the output from top at console 2. |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
I got the number from console 3, line 4 (dealing with RAM): the value shown under "used". Unfortunately, I cannot make a screenshot now, since at the moment I am running 1-core tasks. |
Send message Joined: 15 Jun 08 Posts: 2386 Credit: 222,989,970 RAC: 136,507 |
I got the number from console 3... Sorry, my fault. Console 3 of course. You'll have to look at: Line 4 total: ?? used: ?? free: ?? Line 5 used: ?? cached: ?? Process overview Lines with athena.py -> value(s) in column RES: ?? |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
Below are the values for a currently running 1-core task (running for about 23 1/2 hours, progress about 89%, 197 events processed so far: console 3... |
Send message Joined: 15 Jun 08 Posts: 2386 Credit: 222,989,970 RAC: 136,507 |
Let's see how the numbers would look when we change the total amount of RAM the VM is allowed to use. RAM configured 6GB 3.5GB 4.4GB total 6269324 3709324 4630924 used 6148004 3588004 4509604 free 121320 121320 121320 cached 3392000 832000 1753600 athena.py 1.9g 1.9g 1.9g 6GB: Configured now, but a waste of RAM for a 1-core setup as most of it is used by the VM's cache. In addition the VM never gives allocated RAM back to the host. Your host and other tasks suffer from a lack of RAM. 3.5GB: Configured for a 1-core setup according to the project's RAM formula. May work for some types of ATLAS WUs, but out of experience most WU types need more RAM during the setup phase. Not recommended. 4.4GB: Configured for a 2-core setup according to the project's RAM formula. Works with most WU types. Recommended minimum. 4.6-4.8GB: RAM setting for a 1-core or a 2-core setup if you want to be on the safe side. Configure it via app_config.xml and also set your web preferences accordingly. |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
Let's see how the numbers would look when we change the total amount of RAM the VM is allowed to use.very interesting information, unknown to me in that detail. So I will change my app_config settings accordingly. |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
I am now running three 2-core tasks concurrently, with RAM setting 4800MB in the config_app. In console_3, cache shows approx. 1,5GB; total free RAM of the system is about 9GB (out of 32GB) - which seems okay. |
Send message Joined: 18 Dec 15 Posts: 1686 Credit: 100,432,038 RAC: 102,819 |
computezrmle wrote: In fact, I remember to have seen a variety of "RAM formulas" during the course of time. Hence, my question now: which RAM values would you recommend for a 3-core and for a 4-core task; "minimum" and "to be on the safe side" ? |
Send message Joined: 15 Jun 08 Posts: 2386 Credit: 222,989,970 RAC: 136,507 |
Erich56 wrote: computezrmle wrote:In fact, I remember to have seen a variety of "RAM formulas" during the course of time. Right. I also remember long discussions about the best formula. David Cameron explained that a while ago. The highest RAM usage can be seen during the startup phase of a VM while the data structures are uncompressed. Once this is done, the RAM requirement goes down. In the past we had WUs with typically 50 events, now we have WUs with 200 events. In addition there are different sets of input parameters that lead to different sizes of input files. 200-400 MB can be seen. VMs configured for 3 cores or more have enough RAM to deal with the largest input files, at least until now. Thus you may configure them with default settings. 2-core and (especially) 1-core VMs MAY be able to run WUs with smaller input files, but there's a higher risk that they fail with larger input files. Thus they should be configured with "safe side" RAM values. |
©2024 CERN