Message boards : ATLAS application : why only 3 concurrent 1-core ATLAS tasks possible on 32GB machine?
Message board moderation

To post messages, you must log in.

AuthorMessage
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35233 - Posted: 12 May 2018, 5:44:10 UTC

In the recent past, I have tried all kinds of things in the various settings to make it possible to process 4 1-core ATLAS tasks; however only 3 tasks can be run concurrently, the fourth one remains in "ready to start" status.
The PC has 32GB RAM, processor 6+6HT cores.
In the BOINC computing preferences, RAM usage is set to 100% now (before it was 95%). Disk space also more than sufficient.
In the app_config.xml the max concurrent ATLAS jobs is set to 4; RAM is set to 6400MB (given the fact that from console 3 I could see that towards the end of a task, it uses up to 6GB, sometimes even a little more).

Right now, where 3 1-core ATLAS tasks are running, MemInfo shows a total RAM usage of about 20GB (there are also 2 GPUGRID tasks running, which take some 300MB each).
So there should be plenty of free RAM available for a 4th ATLAS task.
Also, I remember that at times of the former ATLAS project (when it was seperate from LHC) I could run as many ATLAS tasks as possible concurrently, until the 32GB RAM was pretty much used up.
Why is this not possible now?
Is there something wrong in my settings? Do I overlook something?
Would be nice to get some advice from the experts.
ID: 35233 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35240 - Posted: 12 May 2018, 17:53:57 UTC
Last modified: 12 May 2018, 18:03:08 UTC

and what's most interesting, almost unbelievable: although I cannot run 4 1-core ATLAS tasks, I can run 4 2-core ATLAS tasks conurrently.
Does anyone understand this?
For me, a conondrum
ID: 35240 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35248 - Posted: 13 May 2018, 16:34:35 UTC - in response to Message 35240.  

and what's most interesting, almost unbelievable: although I cannot run 4 1-core ATLAS tasks, I can run 4 2-core ATLAS tasks conurrently.
unfortunately, the bad thing is that a 2-core task uses up close to 7GB RAM (in contrast to a 1-core task which takes slightly below 6GB), thus (with 2 GPUGRID tasks running, the system itself, and the usual minimum other apps like antivirus etc.) bringing the total RAM usage to approx. 31,5 GB which is not good at all.
ID: 35248 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1965
Credit: 139,430,403
RAC: 86,163
Message 35249 - Posted: 13 May 2018, 16:48:03 UTC - in response to Message 35248.  

... a 2-core task uses up close to 7GB RAM (in contrast to a 1-core task which takes slightly below 6GB), ...

A 2-core WU should not request such high amount of RAM.
Did you get that number from your host or from the VM?

Be so kind as to post the output from top at console 2.
ID: 35249 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35254 - Posted: 14 May 2018, 4:21:08 UTC

I got the number from console 3, line 4 (dealing with RAM): the value shown under "used".

Unfortunately, I cannot make a screenshot now, since at the moment I am running 1-core tasks.
ID: 35254 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1965
Credit: 139,430,403
RAC: 86,163
Message 35255 - Posted: 14 May 2018, 5:07:38 UTC - in response to Message 35254.  

I got the number from console 3...

Sorry, my fault.
Console 3 of course.

You'll have to look at:

Line 4
total: ??
used: ??
free: ??

Line 5
used: ??
cached: ??


Process overview
Lines with athena.py -> value(s) in column RES: ??
ID: 35255 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35260 - Posted: 14 May 2018, 16:49:15 UTC - in response to Message 35255.  

Below are the values for a currently running 1-core task (running for about 23 1/2 hours, progress about 89%, 197 events processed so far:

console 3...

You'll have to look at:

Line 4
total: ?? 6.269.324k
used: ?? 6.148.004k
free: ?? 121.320k

Line 5
used: ?? 0 (=zero) k
cached: ?? 3.392.000k


Process overview
Lines with athena.py -> value(s) in column RES: ?? 1.9g
ID: 35260 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1965
Credit: 139,430,403
RAC: 86,163
Message 35261 - Posted: 14 May 2018, 18:18:10 UTC - in response to Message 35260.  

Let's see how the numbers would look when we change the total amount of RAM the VM is allowed to use.

RAM configured        6GB      3.5GB      4.4GB
total             6269324    3709324    4630924
used	          6148004    3588004    4509604
free               121320     121320     121320
cached            3392000     832000    1753600
athena.py            1.9g       1.9g       1.9g

6GB:
Configured now, but a waste of RAM for a 1-core setup as most of it is used by the VM's cache.
In addition the VM never gives allocated RAM back to the host.
Your host and other tasks suffer from a lack of RAM.

3.5GB:
Configured for a 1-core setup according to the project's RAM formula.
May work for some types of ATLAS WUs, but out of experience most WU types need more RAM during the setup phase.
Not recommended.

4.4GB:
Configured for a 2-core setup according to the project's RAM formula.
Works with most WU types.
Recommended minimum.

4.6-4.8GB:
RAM setting for a 1-core or a 2-core setup if you want to be on the safe side.
Configure it via app_config.xml and also set your web preferences accordingly.
ID: 35261 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35262 - Posted: 14 May 2018, 19:17:09 UTC - in response to Message 35261.  

Let's see how the numbers would look when we change the total amount of RAM the VM is allowed to use.

RAM configured        6GB      3.5GB      4.4GB
total             6269324    3709324    4630924
used	          6148004    3588004    4509604
free               121320     121320     121320
cached            3392000     832000    1753600
athena.py            1.9g       1.9g       1.9g
very interesting information, unknown to me in that detail.
So I will change my app_config settings accordingly.
ID: 35262 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35281 - Posted: 16 May 2018, 9:02:55 UTC - in response to Message 35261.  


4.6-4.8GB:
RAM setting for a 1-core or a 2-core setup if you want to be on the safe side.
Configure it via app_config.xml and also set your web preferences accordingly.
I am now running three 2-core tasks concurrently, with RAM setting 4800MB in the config_app. In console_3, cache shows approx. 1,5GB;
total free RAM of the system is about 9GB (out of 32GB) - which seems okay.
ID: 35281 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 42,351,745
RAC: 41,065
Message 35288 - Posted: 16 May 2018, 20:11:37 UTC - in response to Message 35261.  

computezrmle wrote:

4.4GB:
Configured for a 2-core setup according to the project's RAM formula.
Works with most WU types.
Recommended minimum.

4.6-4.8GB:
RAM setting for a 1-core or a 2-core setup if you want to be on the safe side.
In fact, I remember to have seen a variety of "RAM formulas" during the course of time.
Hence, my question now: which RAM values would you recommend for a 3-core and for a 4-core task; "minimum" and "to be on the safe side" ?
ID: 35288 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1965
Credit: 139,430,403
RAC: 86,163
Message 35290 - Posted: 16 May 2018, 21:08:03 UTC - in response to Message 35288.  

Erich56 wrote:
computezrmle wrote:

4.4GB:
Configured for a 2-core setup according to the project's RAM formula.
Works with most WU types.
Recommended minimum.

4.6-4.8GB:
RAM setting for a 1-core or a 2-core setup if you want to be on the safe side.
In fact, I remember to have seen a variety of "RAM formulas" during the course of time.
Hence, my question now: which RAM values would you recommend for a 3-core and for a 4-core task; "minimum" and "to be on the safe side" ?

Right.
I also remember long discussions about the best formula.

David Cameron explained that a while ago.
The highest RAM usage can be seen during the startup phase of a VM while the data structures are uncompressed.
Once this is done, the RAM requirement goes down.

In the past we had WUs with typically 50 events, now we have WUs with 200 events.
In addition there are different sets of input parameters that lead to different sizes of input files.
200-400 MB can be seen.

VMs configured for 3 cores or more have enough RAM to deal with the largest input files, at least until now.
Thus you may configure them with default settings.

2-core and (especially) 1-core VMs MAY be able to run WUs with smaller input files, but there's a higher risk that they fail with larger input files.
Thus they should be configured with "safe side" RAM values.
ID: 35290 · Report as offensive     Reply Quote

Message boards : ATLAS application : why only 3 concurrent 1-core ATLAS tasks possible on 32GB machine?


©2022 CERN