LHC@home

The best we can do as crunchers is accept that reality and work with it patiently. If you can't find the patience then you need to decide whether or not crunching is for you.

... OS as unstable and poorly designed as Windoze you're just asking for tons of trouble. I would consider formatting the drive and reinstalling everything from scratch and this time going with a real OS instead of Windoze. Then learn how to walk before you run. Turning on ALL the applications at this project is likely a mistake. Setting "unlimited cores" would be another mistake.

On the Top500 list IBM has reached the top with its Summit computer, which makes large use of nVidia Tesla GPU boards at 200 petaflops. The second is a Chinese supercomputer which was first last time, the third is another American computer, Sierra. The Piz Daint Swiss computer which was third is now sixth, the best Italian is thirteenth. Things change rapidly in the supercomputer world.
Tullio

But what used to work is not working so well now, as everything is falling apart in various degrees. I think the fact of life is that LHC is the most advanced physics project in the world, and it has the most complicated computer and network structure as a necessary part of it. Furthermore, it was not developed for the home users running BOINC, as are most BOINC projects, but was developed for the advanced computing capabilities of similar large institutions (Fermilab, etc.) around the world. They probably don't use VBox at all. So we are sort of an afterthought. It is not that they don't appreciate our efforts, but we are the tail of the dog, not the head of it.

ATLAS now has an elapsed time of almost ten hours - taking so far the last three hours for 10 seconds of remaining run time !
At the same time theory WU is predicting 1 to 2 days (!) of remaining time.
LHCb has upped the estimated time to around 20 hours !

Don't rely on the times.
They are more or less fake as they are based on fixed input parameters.
ATLAS: has longer or shorter batches
Theory, LHCb (,CMS): designed to run 12h+ but time calculation is based on the watchdog limit of 18h

Theory special: Today a new app version has been introduced. It needs a couple of days until your BOINC client corrects the times.



Some stderr.txt files are now available.
Most of them show that your hosts are much too busy (for whatever reason).
Thus your BOINC client, VirtualBox, vboxwrapper and VMs run into timing/priority problems.


There are lots of blank lines in your logs plus messages like this:

2018-06-28 20:54:21 (2308): Powering off VM.
2018-06-28 20:59:22 (2308): VM did not power off when requested.
2018-06-28 20:59:22 (2308): VM was successfully terminated.


2018-06-28 21:13:37 (5104): Powering off VM.
2018-06-28 21:18:38 (5104): VM did not power off when requested.
2018-06-28 21:18:38 (5104): VM was NOT successfully terminated.


2018-06-28 11:19:33 (3948): ERROR: Vboxwrapper lost communication with VirtualBox, rescheduling task for a later time.

ATLAS 1-core may run with only 3500 MB but to be on the save side you may configure 4800 MB via an app_config.xml:

2018-06-28 08:35:04 (3904): Setting Memory Size for VM. (3500MB)
2018-06-28 08:35:05 (3904): Setting CPU Count for VM. (1)

Your BOINC client was terminated before it could save all VMs and other relevant files.
How many VMs (all together) do you run concurrently? Seems to be too much.
Same problem occurs when you restart too many VMs concurrently.
This is what Erich56 already mentioned.

09:57:41 (3984): BOINC client no longer exists - exiting
09:57:41 (3984): timer handler: client dead, exiting
09:57:52 (3984): BOINC client no longer exists - exiting
09:57:52 (3984): timer handler: client dead, exiting
09:58:03 (3984): BOINC client no longer exists - exiting
09:58:03 (3984): timer handler: client dead, exiting
09:58:04 (4896): Can't acquire lockfile (32) - waiting 35s
09:58:13 (3984): BOINC client no longer exists - exiting
09:58:13 (3984): timer handler: client dead, exiting
09:58:23 (3984): BOINC client no longer exists - exiting
09:58:23 (3984): timer handler: client dead, exiting
09:58:33 (3984): BOINC client no longer exists - exiting
09:58:33 (3984): timer handler: client dead, exiting
09:58:39 (4896): Can't acquire lockfile (32) - exiting
09:58:39 (4896): Error: The process cannot access the file because it is being used by another process.

The problem with the lockfile acquisition is indeed indicative of a crash or not giving the VM enough time to shutdown properly before shutting down the OS. BOINC will not fix that problem for you. You must find out which slot dir that lockfile is in then delete it manually from an admin account though I doubt that action alone will cure all your problems. Sounds like you need to remove the LHC project and then re-add it.

there are 2 thoughs on this:
...
So, my guess would be that either one of the above points, or both, are the reason for your problems.

There is, unfortunately, not yet a recent stderr.txt from your WUs that can be analysed.

The older logs as well as the error messages
1. "postponed: VM job unmanageable ..."
2. "Postponed: Waiting to acquire slot directory lock. Another instance may be running"
point out a local problem rather than a general problem.

(1.) could be caused by the RAM setting you configured in your BOINC client.
(2.) could be caused by remains of older crashes.

Did you recently
- restarted your hosts
- reinstalled BOINC
- cleaned your VBox environment
- reset the project?

How many cores do you use for your ATLAS WUs?
How much RAM is configured?

hello, San Fernando

while for me, too, it's not clear by what your problems are caused, one of your lines jumped into my eye:

-- fast SSDs (NVMe Samsung 500GB)

Are you really sure you want to crunch LHC VM tasks with a SSD? Particularly Atlas writes tons of data to the disk.
When I started crunching Atlas with my new PC which was equipped with a SSD, I quickly figured that 4 (or was it even only 3?) concurrently running ATLAS tasks were writing up to 200GB data per day (!). So, it was clear to me that the TBW value of the SSD would be reached within a year, if not earlier (although meanwhile one could read in various forums that some people's SSDs have reached a manyfold of the indicated TBW).

So, once your ATLAS tasks will run well again, you might give a thought to operate VM crunching on a separate HDD.
Just my advice - whatever it's worth :-)

Have this also,
thinking it is a RAM-problem.
Atlas-tasks are dynamicly growing to use more RAM.
When there is no more RAM avalaible in the PC than postponed...
Every better answer is welcome for us volunteers.

If you do nothing the job will resume 86400 seconds (1 day) later.
Restarting BOINC will try a resume immediately.

Make sure that the BOINC "Computing preferences" in the "Disk and memory" tab allow for the use of sufficient memory and disk space. Simply having sufficient installed capacity is not enough.

... ANY idea what I'm doing wrong?

As far as I can see only CMS tasks failed on your computers.
The error log tells you what happened:
"207 (0x000000CF) EXIT_NO_SUB_TASKS"

This means that the project wasn't able to deliver a workpackage to run inside a fresh VM and is mostly caused by a WMAgent outage.
You may check the CMS MB to see if other users report the same problem or if the project team (Ivan) suggests to stop pulling CMS tasks for a while.