Message boards : Theory Application : Theory not utilizing all cores
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · Next

AuthorMessage
CCPLogibro

Send message
Joined: 27 Apr 07
Posts: 2
Credit: 548,665
RAC: 0
Message 36349 - Posted: 9 Aug 2018, 0:48:33 UTC

As you can see here, the Theory Simulation jobs claim to take all 16 cores, but really only take around 2-3's worth. How can I get these theory jobs to either utilize all 16 cores, or release unused cores for other jobs?

ID: 36349 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 36350 - Posted: 9 Aug 2018, 1:20:53 UTC - in response to Message 36349.  
Last modified: 9 Aug 2018, 1:21:52 UTC

The stderr output for https://lhcathome.cern.ch/lhcathome/result.php?resultid=203757358 shows "Setting CPU Count for VM. (8)" so it appears to me Task Manager is mis-reporting?

Also, that task shows Run Time = 9,910.30 secs and CPU Time = 57,289.13 secs. If the task had used only 2 cores then CPU Time would be a little less than double the run time. Instead CPU Time is close to 8X Run Time (actually about 6.1 X Run Time but factor in idle time and it's close enough to 8X). So it's not the 16 you were expecting but it's more than 2.
ID: 36350 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 442
Credit: 12,069,870
RAC: 8,080
Message 36351 - Posted: 9 Aug 2018, 1:22:44 UTC - in response to Message 36349.  

How can I get these theory jobs to either utilize all 16 cores, or release unused cores for other jobs?

How do you have "Max # CPUs" set in your preferences? I set it to 2, and have no problem using the cores on my i7-3770 (Ubuntu 16.04).
ID: 36351 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 947
Credit: 40,295,249
RAC: 7,108
Message 36352 - Posted: 9 Aug 2018, 2:32:51 UTC - in response to Message 36349.  

Set your preferences to run 8 - 2-core tasks



And you will be running all 16 cores to run those 8 tasks and you have enough ram for Theory tasks.
ID: 36352 · Report as offensive     Reply Quote
CCPLogibro

Send message
Joined: 27 Apr 07
Posts: 2
Credit: 548,665
RAC: 0
Message 36353 - Posted: 9 Aug 2018, 3:20:30 UTC - in response to Message 36352.  

Woohoo!! Now we're cookin'

Thanks for the tip! I thought the # of CPU dictated how much to use for the project as a whole.
ID: 36353 · Report as offensive     Reply Quote
Luigi R.
Avatar

Send message
Joined: 7 Feb 14
Posts: 99
Credit: 5,027,000
RAC: 0
Message 36572 - Posted: 27 Aug 2018, 21:48:05 UTC - in response to Message 36352.  

Set your preferences to run 8 - 2-core tasks

https://i80.servimg.com/u/f80/19/36/49/57/lhc_pr10.png

And you will be running all 16 cores to run those 8 tasks and you have enough ram for Theory tasks.

It doesn't look a solution

Isn't vbox64_mt_mcore application supposed to use all cores?
ID: 36572 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 36574 - Posted: 27 Aug 2018, 23:12:55 UTC - in response to Message 36572.  

Isn't vbox64_mt_mcore application supposed to use all cores?

It can use all cores but it will not do it automatically. You need to have sufficient RAM and also have some options configured properly.
Unforunately your computers are hidden so it is difficult to determine what you are doing wrong. Expose your computers to make it easier for others to help you.
ID: 36574 · Report as offensive     Reply Quote
Luigi R.
Avatar

Send message
Joined: 7 Feb 14
Posts: 99
Credit: 5,027,000
RAC: 0
Message 36577 - Posted: 28 Aug 2018, 9:35:51 UTC - in response to Message 36574.  

It can use all cores but it will not do it automatically. You need to have sufficient RAM and also have some options configured properly.
Unforunately your computers are hidden so it is difficult to determine what you are doing wrong. Expose your computers to make it easier for others to help you.

This is my host
ID: 36577 · Report as offensive     Reply Quote
Luigi R.
Avatar

Send message
Joined: 7 Feb 14
Posts: 99
Credit: 5,027,000
RAC: 0
Message 36578 - Posted: 28 Aug 2018, 14:04:36 UTC
Last modified: 28 Aug 2018, 14:14:51 UTC

Now 5-10% of only 1 core! Absurd.

Edit: task reported after 1 hour idle... this is a waste.
ID: 36578 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1461
Credit: 77,713,757
RAC: 88,933
Message 36579 - Posted: 28 Aug 2018, 14:20:26 UTC - in response to Message 36578.  

You may navigate to the project's web preferences:
https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project

Set the "max #cores" of the venue that is connected to your host to "1".
Then request fresh Theory work.
ID: 36579 · Report as offensive     Reply Quote
Luigi R.
Avatar

Send message
Joined: 7 Feb 14
Posts: 99
Credit: 5,027,000
RAC: 0
Message 36592 - Posted: 31 Aug 2018, 11:06:59 UTC - in response to Message 36579.  

It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores!

If I run LHC, Cosmology, Milkyway and GPUGRID multicore apps, I need all the tasks are mt and use 8 cores.
Mixing mt and non-mt tasks is a mess and it doesn't work well.
ID: 36592 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 36593 - Posted: 31 Aug 2018, 22:47:19 UTC - in response to Message 36592.  

VBox tasks seem to run best if they run start to finish uninterrupted. If they get interrupted they do weird things. In 2 of the 3 tasks that completed and validated the stderr output indicates they are are stopping and restarting, especially https://lhcathome.cern.ch/lhcathome/result.php?resultid=206286457.

What do you have in preferences for "switch between tasks every _ minutes" and "Suspend when non-boinc CPU usage is above _ %"? Maybe try setting those to 2000 minutes and 90%.
ID: 36593 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 960
Credit: 6,339,151
RAC: 1,355
Message 36594 - Posted: 1 Sep 2018, 7:04:14 UTC - in response to Message 36592.  

It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores!
...

You're right. The 8-core Theory VM is only using a max of 4 cores. I tested the 8-core VM and saw this:
08/31/18 22:37:36 Allocating auto shares for slot type 0: Cpus: auto, Memory: auto, Swap: auto, Disk: auto
slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00%
slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00%
slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00%
slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00%
08/31/18 22:37:36 slot1: New machine resource allocated
08/31/18 22:37:36 Setting up slot pairings
08/31/18 22:37:36 slot2: New machine resource allocated
08/31/18 22:37:36 Setting up slot pairings
08/31/18 22:37:36 slot3: New machine resource allocated
08/31/18 22:37:36 Setting up slot pairings
08/31/18 22:37:36 slot4: New machine resource allocated
08/31/18 22:37:36 Setting up slot pairings
08/31/18 22:37:36 CronJobList: Adding job 'multicore'

Maybe only physical cores are counted (4 in my case). Have to be confirmed by someone with more physical cores.
Anyway, the most efficient way to run Theory when you have enough RAM (you have) is the single core VM.
ID: 36594 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 947
Credit: 40,295,249
RAC: 7,108
Message 36595 - Posted: 1 Sep 2018, 10:19:50 UTC - in response to Message 36594.  

I have been running these thousands of tasks since I tested them over at -dev in all versions as far as 8-core computers running all the versions of multi-task from 8 down to 2-core and they all work.

I ran many 8-core multi's over at dev and they always worked (unless the server was having a problem)

After I tested all those versions I just switched to running them with 3-core on a couple old Phenom 3-core pc's (in fact have been testing 3-core LHCb's for a while on my older Phenom)

So when we moved these Theory multi's over here I just set all my 8-core pc's to run four 2-core tasks.
And I even have the old X86 running two of the 2-core multi's here and as you probably know you only have less than 4GB of Ram with a X86. (and they all work perfect)
As far as Memory on the four 8-core pc's I have here three of them run the 2-core X4 tasks with either 16GB ram or 24GB ram and the one I am on right now runs 2-core X4 tasks and it only has 8GB ram.

Of course when we have CMS tasks that is not enough Ram.
Same with Atlas.

Of all the versions of VB tasks Theory is the easiest one to run.

(3:20am.....goodnight)
ID: 36595 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 960
Credit: 6,339,151
RAC: 1,355
Message 36596 - Posted: 1 Sep 2018, 12:43:12 UTC - in response to Message 36595.  

MAGIC Quantum Mechanic wrote:
I have been running these thousands of tasks since I tested them over at -dev in all versions as far as 8-core computers running all the versions of multi-task from 8 down to 2-core and they all work.

I ran many 8-core multi's over at dev and they always worked (unless the server was having a problem)

After I tested all those versions I just switched to running them with 3-core on a couple old Phenom 3-core pc's (in fact have been testing 3-core LHCb's for a while on my older Phenom)

So when we moved these Theory multi's over here I just set all my 8-core pc's to run four 2-core tasks.
....

Magic, could you run an 8-core Theory once again on one of your i7-3770's to check it's running 8 jobs within the VM or only max 4?
ID: 36596 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1284
Credit: 23,110,348
RAC: 2,496
Message 36598 - Posted: 1 Sep 2018, 13:54:58 UTC - in response to Message 36595.  

And I even have the old X86 running two of the 2-core multi's here and as you probably know you only have less than 4GB of Ram with a X86. (and they all work perfect)
same here with two of my machines (although I have them run 1-core tasks only)
ID: 36598 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 36599 - Posted: 1 Sep 2018, 15:50:24 UTC

Luigi R completed 3 tasks configured for 8 cores:
A: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206194396
B: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206286457
C: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206302937

Open stderr output for each of the above tasks in your browser and search (Ctrl-F) for "New Job Starting in slot8". You won't find it in any of his tasks, A, B, or C.
In A you'll see the highest slot # in "New Job Starting..." is 7
In B the highest is 5
In C the highest is 2
Why do they not go up to 8? I thought maybe it's because the tasks are getting paused/preempted before they have enough time to "rev up" to 8. Maybe it's a different cause, maybe the tasks themselves are broken.
ID: 36599 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 960
Credit: 6,339,151
RAC: 1,355
Message 36600 - Posted: 1 Sep 2018, 16:29:10 UTC - in response to Message 36599.  

Questions, questions . . .

One possible answer: After the first start of the VM initially the first 2 jobs are starting. The next 2 will start 20 minutes later.
Maybe there is some miscalculation when (one of) the first two jobs have finished before no. 3 and 4 should start.
ID: 36600 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 36601 - Posted: 1 Sep 2018, 17:27:03 UTC - in response to Message 36600.  

Questions, questions . . .

One possible answer: After the first start of the VM initially the first 2 jobs are starting. The next 2 will start 20 minutes later.
Maybe there is some miscalculation when (one of) the first two jobs have finished before no. 3 and 4 should start.

Now that you mention it I see the pattern too... jobs 1 & 2 ending before 3 and 4 should start. Interesting. Yes, maybe some miscalculation going on there.

Then the garbled lines eg.
2018-08-27 23:51:13 (10268): Guest Log: [[INIFNOF]O ]C oCnodnodro rJ oJboIbDI:D :  4 5435836826.21.91 5i ni ns lsolto3t1
2018-08-27 23:51:13 (10268): Guest Log: [[INIFNOF]O ]C oCnodnodro rJ oJboIbDI:D :  4 5435836826.21.91 5i ni ns lsolto3t1

... not a good sign either. Looks like 2 processes writing to the same output buffer(s) simultaneously.

Maybe just an overloaded host... 8 cores busy on a Theory task while trying to feed a GPU at the same time? If so then plausible some calculs could go amiss. Is that 8 real cores or just 4 real + 4 virtual? What really goes on in the pipeline when a cache misses?
ID: 36601 · Report as offensive     Reply Quote
Luigi R.
Avatar

Send message
Joined: 7 Feb 14
Posts: 99
Credit: 5,027,000
RAC: 0
Message 36602 - Posted: 1 Sep 2018, 20:25:03 UTC - in response to Message 36594.  
Last modified: 1 Sep 2018, 20:31:31 UTC

It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores!
...

...
Maybe only physical cores are counted (4 in my case). Have to be confirmed by someone with more physical cores.
Anyway, the most efficient way to run Theory when you have enough RAM (you have) is the single core VM.

My VMs use up to 7 threads, not constantly though.


I have monitored a VM process for 2-3 hours. I just left my computer alone and I did not use internet connection.

Bash script
LANG=C
counter=1
while true; do echo "$counter "$( top -n 1 | grep VBoxHeadless | awk '{print $9}' ) >> lhc.txt; counter=$((counter+1)); sleep 1; done


Then I drew a graph of the CPU usage by ROOT.

ROOT Macro
void apriTxt() {

	ca = new TCanvas( "canvas1", "Grafico", 800, 600);
	ca->SetFillColor( 0 );
	ca->SetGrid();
	ca->GetFrame()->SetFillColor( 21 );
	ca->GetFrame()->SetBorderSize( 12 );
	TGraph *tg = new TGraph( "$ROOTSYS/macros/lhc.txt" );
	tg->SetTitle( "" );
	tg->GetXaxis()->SetTitle( "Time (s)" );
	tg->GetXaxis()->SetLabelSize( 0.030 );
	tg->GetYaxis()->SetTitle( "CPU (%)" );
	tg->GetYaxis()->SetLabelSize( 0.030 );
	tg->SetLineStyle( 0 );
	tg->SetEditable( kFALSE );
	tg->SetMarkerStyle( 8 );
	tg->SetMarkerColor( kBlue );
	tg->SetMarkerSize( 0.7 );
	tg->Draw( "AP" );
}

Graph
ID: 36602 · Report as offensive     Reply Quote
1 · 2 · 3 · Next

Message boards : Theory Application : Theory not utilizing all cores


©2020 CERN