Message boards :
Theory Application :
Theory not utilizing all cores
Message board moderation
Author | Message |
---|---|
Send message Joined: 27 Apr 07 Posts: 2 Credit: 864,546 RAC: 0 |
As you can see here, the Theory Simulation jobs claim to take all 16 cores, but really only take around 2-3's worth. How can I get these theory jobs to either utilize all 16 cores, or release unused cores for other jobs? |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 |
The stderr output for https://lhcathome.cern.ch/lhcathome/result.php?resultid=203757358 shows "Setting CPU Count for VM. (8)" so it appears to me Task Manager is mis-reporting? Also, that task shows Run Time = 9,910.30 secs and CPU Time = 57,289.13 secs. If the task had used only 2 cores then CPU Time would be a little less than double the run time. Instead CPU Time is close to 8X Run Time (actually about 6.1 X Run Time but factor in idle time and it's close enough to 8X). So it's not the 16 you were expecting but it's more than 2. |
Send message Joined: 15 Nov 14 Posts: 602 Credit: 24,371,321 RAC: 0 |
How can I get these theory jobs to either utilize all 16 cores, or release unused cores for other jobs? How do you have "Max # CPUs" set in your preferences? I set it to 2, and have no problem using the cores on my i7-3770 (Ubuntu 16.04). |
Send message Joined: 24 Oct 04 Posts: 1127 Credit: 49,749,586 RAC: 10,234 |
Set your preferences to run 8 - 2-core tasks And you will be running all 16 cores to run those 8 tasks and you have enough ram for Theory tasks. |
Send message Joined: 27 Apr 07 Posts: 2 Credit: 864,546 RAC: 0 |
Woohoo!! Now we're cookin' Thanks for the tip! I thought the # of CPU dictated how much to use for the project as a whole. |
Send message Joined: 7 Feb 14 Posts: 99 Credit: 5,180,005 RAC: 0 |
Set your preferences to run 8 - 2-core tasks It doesn't look a solution Isn't vbox64_mt_mcore application supposed to use all cores? |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 |
Isn't vbox64_mt_mcore application supposed to use all cores? It can use all cores but it will not do it automatically. You need to have sufficient RAM and also have some options configured properly. Unforunately your computers are hidden so it is difficult to determine what you are doing wrong. Expose your computers to make it easier for others to help you. |
Send message Joined: 7 Feb 14 Posts: 99 Credit: 5,180,005 RAC: 0 |
It can use all cores but it will not do it automatically. You need to have sufficient RAM and also have some options configured properly. This is my host |
Send message Joined: 7 Feb 14 Posts: 99 Credit: 5,180,005 RAC: 0 |
Now 5-10% of only 1 core! Absurd. Edit: task reported after 1 hour idle... this is a waste. |
Send message Joined: 15 Jun 08 Posts: 2411 Credit: 226,297,445 RAC: 131,772 |
You may navigate to the project's web preferences: https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project Set the "max #cores" of the venue that is connected to your host to "1". Then request fresh Theory work. |
Send message Joined: 7 Feb 14 Posts: 99 Credit: 5,180,005 RAC: 0 |
It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores! If I run LHC, Cosmology, Milkyway and GPUGRID multicore apps, I need all the tasks are mt and use 8 cores. Mixing mt and non-mt tasks is a mess and it doesn't work well. |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 |
VBox tasks seem to run best if they run start to finish uninterrupted. If they get interrupted they do weird things. In 2 of the 3 tasks that completed and validated the stderr output indicates they are are stopping and restarting, especially https://lhcathome.cern.ch/lhcathome/result.php?resultid=206286457. What do you have in preferences for "switch between tasks every _ minutes" and "Suspend when non-boinc CPU usage is above _ %"? Maybe try setting those to 2000 minutes and 90%. |
Send message Joined: 14 Jan 10 Posts: 1280 Credit: 8,487,876 RAC: 1,781 |
It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores! You're right. The 8-core Theory VM is only using a max of 4 cores. I tested the 8-core VM and saw this: 08/31/18 22:37:36 Allocating auto shares for slot type 0: Cpus: auto, Memory: auto, Swap: auto, Disk: auto slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00% slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00% slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00% slot type 0: Cpus: 1.000000, Memory: 375, Swap: 25.00%, Disk: 25.00% 08/31/18 22:37:36 slot1: New machine resource allocated 08/31/18 22:37:36 Setting up slot pairings 08/31/18 22:37:36 slot2: New machine resource allocated 08/31/18 22:37:36 Setting up slot pairings 08/31/18 22:37:36 slot3: New machine resource allocated 08/31/18 22:37:36 Setting up slot pairings 08/31/18 22:37:36 slot4: New machine resource allocated 08/31/18 22:37:36 Setting up slot pairings 08/31/18 22:37:36 CronJobList: Adding job 'multicore' Maybe only physical cores are counted (4 in my case). Have to be confirmed by someone with more physical cores. Anyway, the most efficient way to run Theory when you have enough RAM (you have) is the single core VM. |
Send message Joined: 24 Oct 04 Posts: 1127 Credit: 49,749,586 RAC: 10,234 |
I have been running these thousands of tasks since I tested them over at -dev in all versions as far as 8-core computers running all the versions of multi-task from 8 down to 2-core and they all work. I ran many 8-core multi's over at dev and they always worked (unless the server was having a problem) After I tested all those versions I just switched to running them with 3-core on a couple old Phenom 3-core pc's (in fact have been testing 3-core LHCb's for a while on my older Phenom) So when we moved these Theory multi's over here I just set all my 8-core pc's to run four 2-core tasks. And I even have the old X86 running two of the 2-core multi's here and as you probably know you only have less than 4GB of Ram with a X86. (and they all work perfect) As far as Memory on the four 8-core pc's I have here three of them run the 2-core X4 tasks with either 16GB ram or 24GB ram and the one I am on right now runs 2-core X4 tasks and it only has 8GB ram. Of course when we have CMS tasks that is not enough Ram. Same with Atlas. Of all the versions of VB tasks Theory is the easiest one to run. (3:20am.....goodnight) |
Send message Joined: 14 Jan 10 Posts: 1280 Credit: 8,487,876 RAC: 1,781 |
MAGIC Quantum Mechanic wrote: I have been running these thousands of tasks since I tested them over at -dev in all versions as far as 8-core computers running all the versions of multi-task from 8 down to 2-core and they all work. Magic, could you run an 8-core Theory once again on one of your i7-3770's to check it's running 8 jobs within the VM or only max 4? |
Send message Joined: 18 Dec 15 Posts: 1688 Credit: 103,721,252 RAC: 121,886 |
And I even have the old X86 running two of the 2-core multi's here and as you probably know you only have less than 4GB of Ram with a X86. (and they all work perfect)same here with two of my machines (although I have them run 1-core tasks only) |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 |
Luigi R completed 3 tasks configured for 8 cores: A: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206194396 B: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206286457 C: https://lhcathome.cern.ch/lhcathome/result.php?resultid=206302937 Open stderr output for each of the above tasks in your browser and search (Ctrl-F) for "New Job Starting in slot8". You won't find it in any of his tasks, A, B, or C. In A you'll see the highest slot # in "New Job Starting..." is 7 In B the highest is 5 In C the highest is 2 Why do they not go up to 8? I thought maybe it's because the tasks are getting paused/preempted before they have enough time to "rev up" to 8. Maybe it's a different cause, maybe the tasks themselves are broken. |
Send message Joined: 14 Jan 10 Posts: 1280 Credit: 8,487,876 RAC: 1,781 |
Questions, questions . . . One possible answer: After the first start of the VM initially the first 2 jobs are starting. The next 2 will start 20 minutes later. Maybe there is some miscalculation when (one of) the first two jobs have finished before no. 3 and 4 should start. |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 |
Questions, questions . . . Now that you mention it I see the pattern too... jobs 1 & 2 ending before 3 and 4 should start. Interesting. Yes, maybe some miscalculation going on there. Then the garbled lines eg. 2018-08-27 23:51:13 (10268): Guest Log: [[INIFNOF]O ]C oCnodnodro rJ oJboIbDI:D : 4 5435836826.21.91 5i ni ns lsolto3t1 2018-08-27 23:51:13 (10268): Guest Log: [[INIFNOF]O ]C oCnodnodro rJ oJboIbDI:D : 4 5435836826.21.91 5i ni ns lsolto3t1 ... not a good sign either. Looks like 2 processes writing to the same output buffer(s) simultaneously. Maybe just an overloaded host... 8 cores busy on a Theory task while trying to feed a GPU at the same time? If so then plausible some calculs could go amiss. Is that 8 real cores or just 4 real + 4 virtual? What really goes on in the pipeline when a cache misses? |
Send message Joined: 7 Feb 14 Posts: 99 Credit: 5,180,005 RAC: 0 |
It's not what I'm looking for. I don't want 8 tasks, I want only 1 tasks that uses 8 cores! My VMs use up to 7 threads, not constantly though. I have monitored a VM process for 2-3 hours. I just left my computer alone and I did not use internet connection. Bash script LANG=C counter=1 while true; do echo "$counter "$( top -n 1 | grep VBoxHeadless | awk '{print $9}' ) >> lhc.txt; counter=$((counter+1)); sleep 1; done Then I drew a graph of the CPU usage by ROOT. ROOT Macro void apriTxt() { ca = new TCanvas( "canvas1", "Grafico", 800, 600); ca->SetFillColor( 0 ); ca->SetGrid(); ca->GetFrame()->SetFillColor( 21 ); ca->GetFrame()->SetBorderSize( 12 ); TGraph *tg = new TGraph( "$ROOTSYS/macros/lhc.txt" ); tg->SetTitle( "" ); tg->GetXaxis()->SetTitle( "Time (s)" ); tg->GetXaxis()->SetLabelSize( 0.030 ); tg->GetYaxis()->SetTitle( "CPU (%)" ); tg->GetYaxis()->SetLabelSize( 0.030 ); tg->SetLineStyle( 0 ); tg->SetEditable( kFALSE ); tg->SetMarkerStyle( 8 ); tg->SetMarkerColor( kBlue ); tg->SetMarkerSize( 0.7 ); tg->Draw( "AP" ); } Graph |
©2024 CERN