Message boards : Number crunching : Memory requirements for LHC applications
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
pls

Send message
Joined: 22 Oct 07
Posts: 27
Credit: 808,821
RAC: 0
Message 37189 - Posted: 3 Nov 2018, 6:53:11 UTC

I'm trying to assemble an authoritative list of memory requirement formulas for the multithreaded app. The bottom of this post is what I have so far. Please let me know of any additions for corrections. When complete, perhaps someone with the appropriate authority can make this a pinned post.

Thanks,
++PLS

=======================================================================

LHC application multithreaded memory requirements
Single threaded applications are not listed here.

App: ATLAS
Forumla: 3900 + 900 * nCPU, megabyte
Command line: --memory_size_mb megabytes
Source: Yeti setup guide, version 3

App: CMS


App: LHCb
Formula: 2048 + 1300 * nCPU, megabytes
Command line: --memory_size_mb megabytes
Source: LHCb Application board, message 37105

App: Theory
Formula: 730 + 100 * nCPU, megabytes
Command line: --memory_size_mb megabytes
Source: Number crunching, message 37161
ID: 37189 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37192 - Posted: 3 Nov 2018, 7:52:51 UTC - in response to Message 37188.  

Your formula for Theory is wrong. The mistake is VERY subtle. Please read carefully and think.

Theory requires 730 MB for the first task plus 100 MB for each additional task. I call that the word expression and I think we agree it is correct. The problem is where you translate the word expression into an algebraic expression. It translates into 630 + 100 * nCPUs. Your algebraic expression adds an additional 100 MB for nCPUs > 1. It's not a huge error. But if you make the same error for ATLAS and other apps then it becomes a huge error > 1 GB.

If you don't believe me then work out RAM for a 3 CPU task using both methods (the word expression and your algebraic equation) and note that you get 2 different results.
ID: 37192 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 37193 - Posted: 3 Nov 2018, 8:22:35 UTC

The 'Command line: --memory_size_mb' in app_config.xml is ignored by the Theory vbox64_mt_mcore applicfation.

It will use the setting coming from the server: 630MB + nCPUs * 100MB.
ID: 37193 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37194 - Posted: 3 Nov 2018, 8:36:28 UTC - in response to Message 37189.  
Last modified: 3 Nov 2018, 8:37:01 UTC

App: ATLAS
Forumla: 3900 + 900 * nCPU, megabyte
Command line: --memory_size_mb megabytes
Source: Yeti setup guide, version 3

You seem to have misquoted Yeti. In item #6 of Yeti setup guide, version 3 it says 3,0 GB + 0,9 GB * number of cores. Again you are adding the amount for each additional core twice.
ID: 37194 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37195 - Posted: 3 Nov 2018, 8:42:07 UTC - in response to Message 37193.  

The 'Command line: --memory_size_mb' in app_config.xml is ignored by the Theory vbox64_mt_mcore applicfation.

It will use the setting coming from the server: 630MB + nCPUs * 100MB.
Do ATLAS and LHCb also ignore the --memory_size_mb command line arg?
ID: 37195 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 37196 - Posted: 3 Nov 2018, 8:51:07 UTC - in response to Message 37195.  

Do ATLAS and LHCb also ignore the --memory_size_mb command line arg?

ATLAS and CMS are obeying the settings from app_config.xml.
I'm not sure about LHCb. I did not run LHCb lately, cause I think the application is not doing something useful at the moment.
However I'll test it and report here.
ID: 37196 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 37200 - Posted: 3 Nov 2018, 10:19:23 UTC - in response to Message 37196.  

Do ATLAS and LHCb also ignore the --memory_size_mb command line arg?

ATLAS and CMS are obeying the settings from app_config.xml.
I'm not sure about LHCb. I did not run LHCb lately, cause I think the application is not doing something useful at the moment.
However I'll test it and report here.

LHCb is also using the memory settings, one could have configured in an app_config.xml.
ID: 37200 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,919,374
RAC: 138,073
Message 37201 - Posted: 3 Nov 2018, 10:37:39 UTC - in response to Message 37193.  

The way an app_config.xml works is that the local client simply overwrites the parameter values sent by the server with the corresponding values from your app_config.xml.
In case of Theory you may check if you still use the old singletask plan_class.
ID: 37201 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 37208 - Posted: 3 Nov 2018, 14:27:34 UTC - in response to Message 37201.  

The way an app_config.xml works is that the local client simply overwrites the parameter values sent by the server with the corresponding values from your app_config.xml.
In case of Theory you may check if you still use the old singletask plan_class.

I was using the right plan_class, but did not retested a divergent MB-size for Theory,
cause I was still in the belief that the server setting for Theory could still not be overruled (as it was in the past).
ID: 37208 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,919,374
RAC: 138,073
Message 37209 - Posted: 3 Nov 2018, 14:44:33 UTC - in response to Message 37208.  

I forgot to mention:
I'm using a slightly higher RAM setting for Theory (to spend a bit more cache inside the VM) and I can confirm that it works as expected.
ID: 37209 · Report as offensive     Reply Quote
pls

Send message
Joined: 22 Oct 07
Posts: 27
Credit: 808,821
RAC: 0
Message 37225 - Posted: 4 Nov 2018, 12:26:45 UTC - in response to Message 37209.  

Corrections are incorporated. I couldn't find requirements for CMS, does anyone have them?

=====================================

LHC application multithreaded memory requirements
Single threaded applications are not listed here.

App: ATLAS
Forumla: 3000 + 900 * nCPU, megabyte
Command line: --memory_size_mb megabytes
Source: Yeti setup guide, version 3

App: CMS


App: LHCb
Formula: 2048 + 1300 * nCPU, megabytes
Command line: --memory_size_mb megabytes
Source: LHCb Application board, message 37105

App: Theory
Formula: 630 + 100 * nCPU, megabytes
Command line: not needed, memory is computed by server
Source: Number crunching, message 37193
ID: 37225 · Report as offensive     Reply Quote
BITLab Argo

Send message
Joined: 16 Jul 05
Posts: 24
Credit: 35,251,537
RAC: 0
Message 37226 - Posted: 4 Nov 2018, 13:37:54 UTC - in response to Message 37225.  

Corrections are incorporated.


I think your LHCb formula is wrong - it doesn't match what I posted in the other thread
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4867&postid=37168, and the latter matches what I see running.

Also, your list is for VirtualBox jobs, not multi-threaded - multi-threaded native Atlas seems to have more frugal requirements.
ID: 37226 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37227 - Posted: 4 Nov 2018, 14:26:10 UTC - in response to Message 37225.  

Looking good. Sorry no info for CMS

You may as well include "plan class" as it is required in app_config.xml

Your App: LHCb looks to be correct so just add the plan class to get:
App: LHCb
Plan class: vbox64_mt_mcore_lhcb
Formula: 2048 + 1300 * nCPU, megabytes
Command line: --memory_size_mb megabytes
Source: LHCb Application board, message 37105


Your App: ATLAS looks to be correct for the VBox version so just add the plan class to get:
App: ATLAS
Plan class: vbox64_mt_mcore_atlas
Forumla: 3000 + 900 * nCPU, megabyte
Command line: --memory_size_mb megabytes
Source: Yeti setup guide, version 3


A new section for ATLAS native:
App: ATLAS
Plan class: native_mt
Formula: 2100 + ? * nCPU  (verify this line)
Command line: --memory_size_mb megabytes
Source: ???


Theory is more complicated as there are 32 bit and 64 bit versions, one of them ignores the memory parameter from app_config.xml while the other does not, not sure which is which.
App: Theory
Plan class: vbox32
Formula: ?
Command line: 
Source: ???


App: Theory
Plan class: vbox64_mt_mcore
Formula:  630 + 100 * nCPU, megabytes
Command line:
Source: ???
ID: 37227 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,919,374
RAC: 138,073
Message 37229 - Posted: 4 Nov 2018, 15:23:01 UTC

The current default RAM requirements are as follows


Theory vbox32 (only singlecore)
320 (in megabytes)


Theory vbox64 (multicore)
630 + 100 * #cpus (in megabytes)


LHCb vbox64 (multicore)
748 + 1300 * #cpus (in megabytes)
based on the old singlecore app that required 2 GB


CMS vbox64 (old singlecore is currently not available, multicore is still under development)
2048 (in megabytes)


ATLAS vbox64
3000 + 900 * #cpus (in megabytes)


ATLAS native
roughly 2 GB per thread + RAM required by auxiliary apps like CVMFS



RAM that is allocated by a native app can be returned to the OS.
Thus those requirements are less critical than vbox requirements which can't be returned.


All default RAM settings are more or less suggestions based on experience.
VMs with less RAM configured via an app_config.xml may be able to run successfully.
On the other hand there may be future jobs with higher requirements.
ID: 37229 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 37230 - Posted: 4 Nov 2018, 15:48:18 UTC - in response to Message 37229.  

Theory vbox32 (only singlecore)
320 (in megabytes)

That's how it is set in the project file Theory32_2017_01_25.xml, but the VM is created with 'only' 256MB.
You may set a different RAM-size in app_config.xml and also could run 2 or more jobs concurrently inside the VM.
E.g.
 <app_version>
  <app_name>Theory</app_name>
  <plan_class>vbox32</plan_class>
  <avg_ncpus>2</avg_ncpus>
  <cmdline>--nthreads 2</cmdline>
  <cmdline>--memory_size_mb 384</cmdline>
 </app_version>
ID: 37230 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37250 - Posted: 6 Nov 2018, 7:50:06 UTC

Sixtrack has 2 plan classes, avx and sse2. What is the memory requirement for each plan class?
ID: 37250 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,919,374
RAC: 138,073
Message 37251 - Posted: 6 Nov 2018, 8:56:29 UTC - in response to Message 37250.  

Sixtrack has 2 plan classes, avx and sse2. What is the memory requirement for each plan class?

The size of the (linux) app itself is roughly 7 MB.
AVX and SSE2 are fed from the same task pool. Thus the RAM requirements are the same.

The recent SixTrack tasks over at LHC-dev had a peak working set size of roughly 350 MB each.
This may vary depending on the task's input data. Thus I would not recommend to set a RAM value via app_config.xml.
ID: 37251 · Report as offensive     Reply Quote
pls

Send message
Joined: 22 Oct 07
Posts: 27
Credit: 808,821
RAC: 0
Message 37258 - Posted: 7 Nov 2018, 5:31:16 UTC - in response to Message 37251.  

Thank you all. This has become far more interesting than I thought it would be.

Here is the current file, incorporating these changes. I'm not listing the sixtrack entries. It seem the native jobs really don't need assistance in managing memory, whereas VBox requires specifying memory size when the VM is created.

======================================
LHC application multithreaded memory requirements,
Single threads applications, VirtualBox or not, are
not mentioned here.


App: ATLAS VirtualBox64
Forumla: 3000 + 900 * nCPU, megabyte
Plan class: vbox64_mt_mcore_atlas
Command line: --memory_size_mb megabytes
Source: Yeti setup guide, version 3

App: ATLAS native (not VirtualBox)
Formula 2100 + 2000 * nCPU, megabytes
Plan class: native_mt
Command line: --memory_size_mb megabytes
Source: Number Cruching message 37225

App: CMS


App: LHCb VirtualBox64
Formula: 748 + 1300 * nCPU, megabytes
Plan class: vbox64_mt_mcore_lhcb
Command line: --memory_size_mb megabytes
Source: LHCb Application board, message 37105

App: Theory VirtualBox64
Formula: 630 + 100 * nCPU, megabytes
Plan class: vbox64_mt_mcore
Command line: not needed, memory is computed by server
Source: Number crunching, message 37193

App: Theory VirtualBox32
Formula: 256 + 64 * nCPU, megabytes
Plan class: vbox32
Command line: --memory_size_mb megabytes
Source: Number crunching message 37229
ID: 37258 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 37259 - Posted: 7 Nov 2018, 6:55:04 UTC - in response to Message 37258.  

App: ATLAS native (not VirtualBox)
Formula 2100 + 2000 * nCPU, megabytes
Plan class: native_mt
Command line: --memory_size_mb megabytes
Source: Number Cruching message 37225


The formula is wrong. It should be 100 + 2000 * nCPU.
Also it might be worth mentioning it's Linux only else the unwary might try to run it on Windows or Mac.

App: ATLAS native (not VirtualBox, Linux only)
Formula 100 + 2000 * nCPU, megabytes
Plan class: native_mt
Command line: --memory_size_mb megabytes
Source: Number Cruching message 37225
ID: 37259 · Report as offensive     Reply Quote
gyllic

Send message
Joined: 9 Dec 14
Posts: 202
Credit: 2,533,875
RAC: 0
Message 37262 - Posted: 7 Nov 2018, 9:01:56 UTC - in response to Message 37259.  

The formula is wrong. It should be 100 + 2000 * nCPU.
The formula 100 + 2000*nCPU is also wrong. I don't know the correct one but since my PC with 6GB RAM can crunch a 4 core native ATLAS task without any problem, the mentioned formula can't be correct (according to the formula the native task would need 100 + 2000*4 which is 8100MB on a 6000MB RAM machine which obviously would not work).
It is also possible to crunch two 2-core native ATLAS tasks concurrently with 6GB RAM, so the correct formula has to be another one.
ID: 37262 · Report as offensive     Reply Quote
1 · 2 · 3 · Next

Message boards : Number crunching : Memory requirements for LHC applications


©2024 CERN