LHC@home

3. a script that periodically (once per minute) checks for recently started VMs/vboxwrappers and renices their nice level.

If that script is not too long could you post it here, maybe in a separate thread? Or PM it to me? I would love to have a look at it and maybe incorporate it into the LHC babysitter script I've been working on for these past months.

If that script is not too long ...

It's just 1 line in /etc/crontab

* * * * *	root for VWRAP in $(pgrep -u boinc -l| egrep -i 'wrapper' |cut -d" " -f1); do renice -n 10 $VWRAP; done >/dev/null 2>&1

bash is best

it would appear that <process_priority_special> doesn't do anything for this project, the wrapper still runs at low which is contrary to the documentation. the normal process priorty boost it up as documented.

not off to a good start, with all processes running as normal then I still got some unmanageable after a few hours but wiil give some time.

it's maybe better with the change in Prio but still not 0 so same impact with no more tasks as it's blocks the queue.

Given this I quit my work for ATLAS project as the effort to babysit the project is too much, with the poor scheduling, ram configuration and now this.

If there is some changes then I will try again in the future.

I just got back yesterday, hence delay. I have not seen anything that addresses the issue that I see and am unhappy with. I have detatched my machines from LHC.

---

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

Since I'm on windows, I made a Powershell script that runs and aborts the stuck tasks automatically.

I downgraded to the 5.1.x Vbox and this improved things as well I think.

Since I'm on windows, I made a Powershell script that runs and aborts the stuck tasks automatically.

Then you run the risk of a scenario where the host is doing only 1 of 2 things: it's either 1) aborting a stuck task and downloading another one OR 2) in the process of turning another viable task into a stuck task. It's a vicious cycle... get a new task, hope to get lucky, blow it, get a new task, hope to get lucky, blow it, get a new task, hope to get lucky, blow it, and on and on and on.

I downgraded to the 5.1.x Vbox and this improved things as well I think.

We all want to believe that our shiny new box with the CPU we've been drooling over for months coupled with more RAM than we've ever crammed onto a mobo before, fast drives, etc. can do anything we throw at it. Then reality sets in. And what do we do? Human nature is to deny and blame everybody but ourselves. I've been caught up in the "oh, it's gotta be a flaky version of this or that app" game before and sometimes it is. More often than not I am simply expecting too much of a rig I thought was absolutely invincible but instead we torture ourselves with weeks of re-installing this, rolling back to that, searching for the magic combination of X, Y and Z and other fruitless endeavor.

Not sure how many cores you're trying to run ATLAS on but sometimes we get more done by simply trying to do less than the max and doing that well. Sometimes we need to try doing just the minimum until we can do that each and every time consistently and only once we reach that level should we kick it up a small notch.

I guess my previous actions were the best. As if we don't try and work around the issues and stop working for the project then they would address the request that we have.

I only run one core task on my machines as it the most efficient, I have one that run dual core as there is a limit on the number of tasks that you can run. compared to the other projects I don't contribute much to ATLAS as it's not possible to configure my computers to do more when shared with the other projects.

I expected it to work well as it was working well before so that set my expectations, hence the roll back.

If the task is postponed then in my experience it's never come back from that state, I think if you restart Boinc, then it will come back. The biggest problem with that state is it stops boinc from getting new tasks so when left un-managed it drains the queue

Same error msg here after a day of crunching. Happens on Win10 and Win7 with lastest vbox and I did downgrade to 5.1.30 on the win7 machine, testing it now. Win10 machine will be downgraded now too. Stopping boinc and restarting works, for about 4%, then stops again. Will see what happens with a downgrade on the win10 machine now.

How do you filter out just the ATLAS tasks?
(Indeed, these constantly seem to be "unmanageable" and "postponed" a long time, blocking other jobs.)
Thank you.

---

Go to your project settings at https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project&cols=1. There you will have up to 4 venues configured (default, home, work, school) or, depending on what you have configured in the past you might have only the default venue configured. For each of the venues you have configured click "edit" then deselect ALL of the following:

ATLAS
Run test applications
If no work for selected applications is available, accept work from other applications

Don't forget to scroll to the bottom of the page and click "Update preferences".

Hi Toby, All,

I'm encountering the same issue for about 2 months now. (Postponed: VM job unmanageable, restarting later.)

I run BOINC version 7.14.2 (x64), version 3.0.1.
VB v6.0.0 r127566.

I have 32 GB of RAM
6 core CPU with hyperthreading activated (12 virtual cores)
2 x Asus GTX 1070Ti
Ample disk space, dedicated drive to BOINC of 3 TB with 2.5 TB still unused capacity.
Win 10 Pro (10.017134), all latest patches installed. All drivers kept up-to-date (mobo, drives, LAN, bluetooth, WiFi, etc.).
Latest up-to-date Anti-Virus & Firewall software.

I usually see RAM usage at 12-15 GB out of 32, 15-20 GB available as reported by Win 10.

I've been running multiple projects over time simultaneously (not all of them at once of course, but a select number of them at any
one time).
On GPU level I'm mainly running GPUGrid, in absence of GPUGrid, I have MilkyWay as backup (and Einstein, but only if
MilkyWay is out of WU's)
Each GPU card gets one virtual CPU attributed.

On CPU level, I run LHC and as backup I have WorldComGrid, Rosetta, ClimatePrediction. (only one selected as backup at a time)

I allow LHC to run virtually all projects : LHCb, Atlas, Theory, CMS, etc.

The WU's run in 5-core setups and recently in 7-core setups.

Total core usage : 2 for GPU's, 7 for CPU or 9 cores in total. Adding one core for BOINC and VirtualBox, I'm running on average 10
virtual cores out of 12.

Apart from doing some mail or surfing on the internet, nothing else is happening on this system.
Average CPU load is 80 %
The system runs 24/7, 320/365 days for 4 years now.

On LHC in total, I racked up 3.537.241 credits, I modestly think you could say I'm no longer a rookie :-).

The odd thing is that this error kicks in also in the middle of the night, when I'm not active, subsequently, there should be ample space, processing
power and RAM available.
Unless of course I'm hacked and someone's using my rig to mine on cryptocurrencies... but didn't find any trace just yet.

So far I only spotted Atlas WU's being affected. And it errors not only at the beginning of the WU, but at any possible time (e.g. I have one now at 81.47 %)

The only way to unblock it, is to suspend all CPU & GPU activity, give BOINC the time to stop all crunching. Exit orderly from BOINC, restart BOINC
and then it works again when all WU's are activated once more. Unfortunately, quite soon afterwards, it errors out again. Sometimes still the same
WU, sometimes another one.

LHC is the only project using VB, most likely, there might be some interaction causing an issue. The odd thing is that it seems to occur on both
v6 from VB as well as v5.2.x Moreover, BOINC being at the very latest version does not seem to offer any alleviation.

Actually, when checking VB, I get 2 instances which give an error "Inaccessible" :
Runtime error opening 'F:\BOINC DATA\slots\2\boinc_a10398caff860c3b\boinc_a10398caff860c3b.vbox' for reading: -103 (Path not found.).
F:\tinderbox\win-rel\src\VBox\Main\src-server\MachineImpl.cpp[745] (long __cdecl Machine::i_registeredInit(void)).
Result Code: E_FAIL (0x80004005)
Component: MachineWrap
Interface: IMachine {5047460a-265d-4538-b23e-ddba5fb84976}

Then I have a third instance which is "Powered Off"

And a forth one which is Running (the current Atlas-7cores WU).

Should I reinstall VB ?

Any suggestions ?

---

Hi all and thanks in advance for any proposals

When trying to start jobs from LHC@home , after some working time and arounf 30% done, the job stops saying "Reported.: VM job unmanageable , restarting later. "
I'm working on an Gigabyte PC, 8 core Intel proc, 16 GO RAM. Don't know what to do with last updates Boinc and VBox..
With my apologies if disturbing you but I would like to understand…
Many thanks again, have good days.

---

I am not running any LHC tasks but I have recently come across this error running the Virtual Box tasks on cosmology@home.

I had my system set to run 4 tasks with 2 cpus each and the error was occurring. I switched the tasks to a single task using 8 cpus and, so far
the tasks are running fine. I have been running this way for at least 24 hours now.

The errors started after I set no new tasks, returned all work units and upgraded to Virtual Box 6.0.4. Boinc is 7.14.2. I have SMT on but
everything else should show for my computer info. Computer is the AMD processor one

I don't know if this may help anyone with this error or not but I figured I would post it here.

I don't know if this may help anyone with this error or not but I figured I would post it here.

Thanks Jonathan!

I also noticed that VirtualBox 6.0.2 and even more v6.0.4 is more sensible for the famous error "Postponed: VM job unmanageable, restarting later",
caused by missing heartbeat by vboxwrapper.

Posting this again to this thread:

There seems to be a specific problem with ATLAS's vboxwrapper executable where it loses control of the VM periodically: 5/7 such tasks eventually missed their deadline due to this on my machine (1 failed validation, 1 passed). Since this was just wasting slots/time for me, I disabled ATLAS tasks. Now I have a Theory Simulation VM task and it has not yet aborted due to the VM unmanageable issue and seems to be progressing and checkpointing (I hibernate the machine when not in use and the progress is not reseting). Hopefully it will finish.

There have not been any schedule/policy changes other than blocking ATLAS tasks: now this new VM task seems to actually work and progress normally. This is with VirtualBox VM 6.0.8 (was using a version 5 before with the same issue).

From the task properties and From looking at the files in C:\ProgramData\BOINC\projects\lhcathome.cern.ch_lhcathome :
Theory is using vboxwrapper_26198ab7_windows_x86_64.exe
ATLAS appears have been using vboxwrapper_26196_windows_x86_64.exe

I assume that the major difference in the tasks is the .vdi used for each (I can see those in the same folder), so possibly this issue can be fixed (very easily?) by the developer updating the vboxwrapper for ATLAS.

I think this has been noted before (maybe not in this thread though): is it too simple to fix the issue?

Hi Everyone,

I'm experiencing exactly the same error with the Atlas project, none others.

I have them on a quite frequent basis, pausing, exiting BOINC, rebooting and restarting again
does the job. Not just on one, but now also on 2 machines.

I'm running VBox 5.2.26 r 128414 (Qt5.6.2)
BOINC Manager 7.14.2, Widgets 3.0.1

Both machines have 32 GB of RAM, swap file of 48 GB (1.5 x RAM size).
Memory usage is peaking at 40 % of 32 GB (or around 13-14 GB of actual RAM usage).
So by far not coming near even 50 % of the allowed maximum (setting of 80 % of RAM max usage for BOINC).

It is quite annoying since it also interferes with other projects such as GPUGrid or WorldComGrid.

When reading through the above discussion, there's mention that wrapper v5.2 seems to be the cause of the issues.
Are there any plans to start to support it ? (Oracle wants me to upgrade to 5.2.30 in the mean time...)

Thanks for any updates :-)

BE.

Hi, 5.1.38 is reliable for me with ATLAS, all of the versions after that are not with 100% load, If I run my computer at 70% then I can use 6.0.x

Hi Toby,

Thanks for the quick response !

I understood from above discussions that apparently v5.1 would work with Atlas, so your
statement confirms that.
Which is a little bit odd if I may say so : when downloading new version of BOINC + VBOX
you automatically get 5.2 something.

So, if I install 3 new PC's I have to install BOINC + VBOX first, then downgrade VBOX 5.1
on all 3 PC's afterwards. That's a bit of a hassle...

Are there any plans in the relatively short term to make Atlas compatible with v5.2 ?

By the way, Oracle is already pushing to install v6. smth in the meantime...
Or is it the purpuse to leapfrog v5.2 and support straight away v6 ?

Wish you a nice weekend !

Friendly Greetings,
K.