Message boards :
CMS Application :
Why Such Varied Runtimes?
Message board moderation
Author | Message |
---|---|
![]() Send message Joined: 17 Sep 04 Posts: 104 Credit: 32,530,127 RAC: 6,491 ![]() ![]() |
I'm just curious why work unit runtimes vary so much for CMS (on one consistent machine)? Thanks. Regards, Bob P. |
Send message Joined: 27 Sep 08 Posts: 817 Credit: 679,144,285 RAC: 222,332 ![]() ![]() ![]() |
I assume they are somewhat similar to Sixtrac in that if the particles crash in to the wall then that the end of the WU, I can't say for sure if that true. For me they are also +/- 2 hours for the same computer. |
![]() Send message Joined: 29 Aug 05 Posts: 1044 Credit: 7,221,207 RAC: 11,854 ![]() |
I'm just curious why work unit runtimes vary so much for CMS (on one consistent machine)? Well, you are getting errors on your one machine that runs CMS tasks. I can't be much more specific at the moment as I'm at home with poor connectivity -- and my uni has hobbled my laptop and desktop. A CMS "task" is an instance of a virtual machine -- that 1.5 GB file that you downloaded when you started up -- running inside the VirtualBox supervisor (hypervisor). If the VM doesn't start up cleanly, then the task will fail. When the VM _does_ start up, it requests a CMS "job" from an HTCondor batch job server. This job may fail for a number of reasons, most commonly network problems. Unfortunately these failures aren't always relayed to the BOINC task, so one task may suffer multiple job failures in its lifetime. Now, on most modern machines, each properly-running job takes about two hours to complete, generating 10,000 simulated proton-proton collisions within the CMS detector system. At this point, the job is terminated and its result and log files are sent back to CERN for storage (and later merging into larger files on dedicated storage). Jobs can also fail in this "stage-out" step. Once the job is finished, if the VM is less than 12 hours old, it requests another job from the condor pool, otherwise it terminates and your BOINC task ends. An easy way to see if your tasks are misbehaving is to look at you work-unit result page on the LHC@Home site. If you see a large discrepancy between the CPU time and running time, you probably have a problem. If your tasks consistently take less than 12 hours to run, you probably also have a problem. ![]() |
![]() Send message Joined: 17 Sep 04 Posts: 104 Credit: 32,530,127 RAC: 6,491 ![]() ![]() |
Thank you for the detailed reply. It’s interesting to know that jobs will keep coming up to a 12-hour age for the VM. Those recent CMS failures were due to a reboot. I’ll be more careful next time!👠Regards, Bob P. |
![]() Send message Joined: 29 Aug 05 Posts: 1044 Credit: 7,221,207 RAC: 11,854 ![]() |
Thank you for the detailed reply. It’s interesting to know that jobs will keep coming up to a 12-hour age for the VM. Yes. It's important that you shut BOINC down carefully with boincmgr ot boinccmd, and give the VM time to store its state (2 minutes should be enough) before you switch off or reboot. Then, as long as the interruption isn't too long, the task will resume from where it was when BOINC is restarted. ![]() |
Send message Joined: 15 Nov 14 Posts: 602 Credit: 24,371,321 RAC: 0 ![]() ![]() |
Could you take a look at mine? I see a large variation too, as compared to what it used to be with 50.00, when it used to run for 13 1/2 hours and was done with it. Now they vary from 1/2 hour to 12 1/2 hours, though it does not seem to be due to errors to any great extent. https://lhcathome.cern.ch/lhcathome/results.php?hostid=10697859&offset=0&show_names=0&state=4&appid= I assumed the new app was allowing the experimenters to do different things, though what they were doing all along is a mystery to me anyway,. |
![]() Send message Joined: 15 Jun 08 Posts: 2473 Credit: 244,957,259 RAC: 171,022 ![]() ![]() ![]() |
I also see those short runtimes (<12 h). The BOINC clients in question are running only CMS 24/7 without any interruption. Uploads (stage-out) work fine. The logfiles do not show any hints that point out any errors nor do my other logs. @Ivan Be so kind as to investigate whether a script deeper in the process or a configuration independent from BOINC (Condor/WMAgent) causes a task not to get follow-up jobs (=subtasks). |
![]() Send message Joined: 29 Aug 05 Posts: 1044 Credit: 7,221,207 RAC: 11,854 ![]() |
I also see those short runtimes (<12 h). The 21st Oct was when we were updating the WMAgent, so no jobs would have been available then. [Edit] Oops, no, I misread the chart... [/Edit] Jim seems to have had task times >12 hrs in recent times. Yours are still a little short, I hope your 28 machines /430 cores aren't taxing your network bandwidth. 🙂 ![]() |
![]() Send message Joined: 15 Jun 08 Posts: 2473 Credit: 244,957,259 RAC: 171,022 ![]() ![]() ![]() |
... I hope your 28 machines /430 cores aren't taxing your network bandwidth. 🙂 Since I run multiple BOINC clients on the same box those numbers are not real. I'm usually running around 35-37 CMS and 35-37 ATLAS tasks concurrently which results in a download saturation of <5% and an upload saturation of <15%. Unfortunately this afternoon my ISP had a major DSLAM outage that affected a couple of households. 3rd time within a few weeks :-( May cause some tasks to fail. |
Send message Joined: 15 Nov 14 Posts: 602 Credit: 24,371,321 RAC: 0 ![]() ![]() |
Yes, they have been well-behaved for me the last week or so, averaging a little over 12 hours. But I am learning new things about running on 23 cores of a Ryzen 3900X (reserving one core for a GPU). I received an "out of disk" space error message, and saw that my 250 GB SSD was in fact a little low, which was a surprise. So I upgraded to 500 GB, which should be plenty, but recently got the error message again. You have to set the BOINC "Disk usage" parameters manually, and not rely on the defaults. So I am using 500 GB max, and 100% of total disk space. That seems to have satisfied it, for the moment. |
![]() Send message Joined: 29 Aug 05 Posts: 1044 Credit: 7,221,207 RAC: 11,854 ![]() |
Yes, they have been well-behaved for me the last week or so, averaging a little over 12 hours. Good to hear. In the early days of BOINC (which mostly meant SETI@Home) you could let your install "fly blind", but with much more sophisticated applications these days (including such as ours, using VMs and/or GPUs), a little bit of oversight is often necessary. As you've found out, this is especially true in terms of file sizes and network bandwidth. Within CMS we now routinely deal with data files of at least 2 GB (in fact, our sister site*, T3_CH_CMSAtHome, which runs at CERN, does all the "post-production" tasks for CMS@Home, and its major task is JobMerge where it takes your 60-80 MB result files and gathers them into files ~2 GB and stores them into our central storage system. * In case you didn't know, our production pseudo-site is known in CMS as T3_CH_Volunteer, which translates as "Tier 3" (Tier 0 is CERN, Tier 1 are national facilities, Tier 2 are regional centres such as London, and Tier 3 are seen as institutional set-ups), and CH means Switzerland (obviously...). ![]() |
©2024 CERN