LHC@home

Good evening,

I keep realizing again and again that Atlas Simulation tasks never finish. They are on 100% after, lets say 8h, but 4d later are still in the same stage and count as "running" until I manually abort them.

Any tips on how to troubleshoot?

Example of tasks that was on 100% for days:

Application
ATLAS Simulation 2.00 (vbox64_mt_mcore_atlas)
Name
ADlNDm6BAsyn9Rq4apoT9bVoABFKDmABFKDmw8qYDmABFKDmVnJmnm
State
Aborted by project
Received
18/04/2021 04:31:59
Report deadline
25/04/2021 04:31:59
Resources
8 CPUs
Estimated computation size
43,200 GFLOPs
CPU time
---
Elapsed time
---
Executable
vboxwrapper_26198ab7_windows_x86_64.exe

Thanks,
Rene

It looks like you don't have enough memory. The CMS take about 3 GB each, and the ATLAS probably almost as much on VBox.
But even the native ATLAS (on Linux) takes 2 GB.

Vbox-Projects (Atlas, CMS and Theory) need some tuning to optimize a successful task.
Yeti's Checklist is very useful.
You can use for the first experience Theory and Sixtrack. 16 GByte RAM is ok.
Atlas and CMS-Tasks must be running complete, before you stop the PC.

oh, so restarts due to OS updates can cause this behaviour?
Do you maybe even have a link to that checklist?

Thanks,
Rene

oh, so restarts due to OS updates can cause this behaviour?
Do you maybe even have a link to that checklist?

Thanks,
Rene

On this same forum, one of the sticky threads.

---

I have a couple of these also.

Going on the 5th day and always approaching 100% but never reaching it.
One of them gives a flashing numluck/scrolllock LED's when connecting to the VM using VBox manager.
The other just shows the login screen and is still using 1 core of the client.

I remember the tasks eventually ending and giving credit for the days of time consumed.
Something has changed?

Seriously do not want to abort 10 days of core usage for no credit if these will eventually end with credit.

I have more than enough memory and 16 cores.
I have found that if BOINC runs more than one instance of ATLAS the task will stall at around 90% and drag on forever. It will only increase in finishing by .001% every few seconds.
You need to limit the number of simultaneous tasks by LHC. I even tried running Theory and ATLAS together and ATLAS stalled out all the time.

Now if you run ATLAS alone and set your cpu number to 4 in the preferences, then it will make ATLAS run only one task at a time. It should take under 8 hours to complete a task.
If you run any other LHC projects, then you will have to write a special script to force BOINC to run only 1 task from LHC at a time.

I have more than enough memory and 16 cores.
I have found that if BOINC runs more than one instance of ATLAS the task will stall at around 90% and drag on forever. It will only increase in finishing by .001% every few seconds.
You need to limit the number of simultaneous tasks by LHC. I even tried running Theory and ATLAS together and ATLAS stalled out all the time.

Now if you run ATLAS alone and set your cpu number to 4 in the preferences, then it will make ATLAS run only one task at a time. It should take under 8 hours to complete a task.
If you run any other LHC projects, then you will have to write a special script to force BOINC to run only 1 task from LHC at a time.

I have traced this behavior to ATLAS job not saving it state properly. (There maybe other causes but this is certainly one).

In BOINC advanced options, computing preferences, computing tab, set "switch between tasks every" to 9999 minutes so that ATLAS is never suspended when BOINC decides to swap WU tasks to accommodate resource share of multiple projects. (Or isolate your other projects from ATLAS).

If you need to shut down BOINC then suspend your ATLAS WU's 1 at a time so they get saved properly in VBox manager.

I have a couple of these also.

Going on the 5th day and always approaching 100% but never reaching it.

Found a solution.

Suspend the WU manually in BOINC.

Open VBox Manager and find the newly saved state ATLAS VM.
Delete the saved state,

Now the WU has to start over and the corrupt execution state is gone.

Although, the credit was pitiful and, since the WU's started from scratch, would probably have been just as well to abort them.
BUT, nothing new is learned unless you experiment.

314464634	162960124	22 Apr 2021, 11:19:02 UTC	28 Apr 2021, 19:06:30 UTC	Completed and validated	497,012.25	553,831.60	265.24	ATLAS Simulation v2.00 (vbox64_mt_mcore_atlas)
windows_x86_64

313918637	162691404	21 Apr 2021, 9:24:05 UTC	29 Apr 2021, 2:26:32 UTC	Completed and validated	536,236.41	500,751.30	408.92	ATLAS Simulation v2.00 (vbox64_mt_mcore_atlas)
windows_x86_64

Maybe you could experiment with my xml-file for ATLAS. No savings to disk when BOINC stops/ system shutdown, but instead taking regular snapshots.of the ATLAS task.

Replace the contents of the ATLAS_vbox_2.00_job.xml with the example in the thread ATLAS using VirtualBox with snapshots.

In the options part of cc-config.xml you have to add a line
<dont_check_file_sizes>1</dont_check_file_sizes> to avoid overwriting tha adjusted file by the project.

You could change the snapshot (checkpoint) interval or increase the 'write to disk' in BOINC's preferences.

Thanks for all the tips; I've now gone with adjusting the settings of swapping projects first and will see how that goes, else I work through the other tips :)

cc-config.xml? Where is that located? I don't see a file named that anywhere under C:\program files or c:\programdata\ and none of the XML's located under C:\ProgramData\BOINC\projects\ I found seem to have that option line.

---

It must be "cc_config.xml" instead of "cc-config.xml".
See the BOINC manual:
https://boinc.berkeley.edu/wiki/Client_configuration

well, I still have had some in the last couple of days, but slapped more RAM into the device now, let see if that stops them from happening with no reboot disturbing them :)

Will update again in a week or three.

Update: Happens less, but still happens.

I have the exact same issue. I keep on opening the computer expecting the tasks to be complete. The last 'hours' take days and then typically fails.
Elapsed time keeps ticking but reamaining time doesn't change.
I have 8 cores and 64GBs of RAM so that's not the issue.
I have changed my preferences as suggested above but to no avail.
All apps are up to date. I have tried re-loading the project as well. Nothing.

You may make your computers visible for other volunteers here:
https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project

In addition it is necessary to get some information from a typical stderr.txt, either
- from a reported tasks (valid or invalid) or
- post that log from a currently running task (see BOINC's slots dir)

Welcome Philip,
you can work thru Yeti's Checklist first.