Message boards : Number crunching : VM environment need to be cleaned up.
Message board moderation

To post messages, you must log in.

AuthorMessage
marmot

Send message
Joined: 5 Nov 15
Posts: 90
Credit: 4,370,848
RAC: 34,240
Message 33178 - Posted: 30 Nov 2017, 16:42:48 UTC
Last modified: 30 Nov 2017, 16:44:32 UTC

This error in BOINC manager sometimes shows up.
Also, probably related since I noticed the job with that message had a collision with an undeleted previous vdi in its claimed slot, eventually hundreds of broken links to vdi files in the BOINC data\slot directories will be in the VBox media manager.

VBoxManage.exe fails to complete the deletion of the slot vdi file.

No warning or information entry is found in the Windows event logs about VBoxManage.exe or vboxwrapper_xxxxx.exe closing unexpectedly.

Raising the priority of VBoxManage.exe to normal from idle seems to have reduced the number of broken links by 90+%.

File locked error or timing out before job cleanup?

Does vboxwrapper double check that vdi remnants are cleaned up?

(Happens on all machines and they have varying versions of VBox 5.1.26, 5.1.28 and 5.2.0)
ID: 33178 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 547
Credit: 15,357,832
RAC: 15,141
Message 33181 - Posted: 1 Dec 2017, 15:58:56 UTC - in response to Message 33178.  

Yes Marmot, I have mentioned that several times here but with no reply so I always just check my VB Manager/File/Virtual Media Manager and *remove* them myself. (for a long time now)

Maybe with more than one person says it something will be done.

Good luck
Volunteer Mad Scientist For Life
ID: 33181 · Report as offensive     Reply Quote
Profile ritterm
Avatar

Send message
Joined: 30 May 08
Posts: 88
Credit: 3,722,802
RAC: 2,015
Message 33182 - Posted: 1 Dec 2017, 19:02:08 UTC - in response to Message 33181.  

...I always just check my VB Manager/File/Virtual Media Manager and *remove* them myself...

So, do you just remove/delete the vm_image.vki files that aren't attached to anything?
ID: 33182 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 547
Credit: 15,357,832
RAC: 15,141
Message 33183 - Posted: 2 Dec 2017, 6:38:09 UTC - in response to Message 33182.  

...I always just check my VB Manager/File/Virtual Media Manager and *remove* them myself...

So, do you just remove/delete the vm_image.vki files that aren't attached to anything?


YES.......and I check them all before I start the next batch running (I know most members don't like doing that but I like to take care of problems before they happen)
ID: 33183 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 272
Credit: 12,528,659
RAC: 3,361
Message 33184 - Posted: 2 Dec 2017, 8:19:32 UTC

Have a HS2011-Server with CMS and Theory tasks for using.
This tasks become the work from Cern in exchange since weeks.
There are no .vdi to delete in Virtual Media manager.
So, it must be a other reason therefore.
Virtualbox for me is 5.1.30.
ID: 33184 · Report as offensive     Reply Quote
Profile Ray Murray
Volunteer moderator
Avatar

Send message
Joined: 29 Sep 04
Posts: 151
Credit: 5,352,024
RAC: 3,948
Message 33185 - Posted: 2 Dec 2017, 18:24:16 UTC - in response to Message 33181.  

I first saw and reported this late May / early June. Don't know if the newer wrappers at -dev fare any better? Think it's maybe a timing issue as the wrapper exits perhaps before VBox has fully finished tidying up. I have noticed that any time VBox is stopped, eg for an update to itself or Windows, any yellow triangles get cleared, otherwise they still need to be manually deleted.
ID: 33185 · Report as offensive     Reply Quote
marmot

Send message
Joined: 5 Nov 15
Posts: 90
Credit: 4,370,848
RAC: 34,240
Message 33187 - Posted: 2 Dec 2017, 23:14:29 UTC - in response to Message 33181.  

Yes Marmot, I have mentioned that several times here but with no reply so I always just check my VB Manager/File/Virtual Media Manager and *remove* them myself. (for a long time now)


When the service restarts, almost all of the broken links to vdi's get cleaned up.
So, after all VM's are stopped, try using your task/process manager and terminate the VBoxSVC.exe found under services process. Or more elegantly, use the Windows Services management console and stop and restart that service. A batch file with the appropriate net stop VBoxSVC.exe the net start can automate it. No need to restart the computer.


I think, if they would run vboxwrapper_xxxxxx at normal priority instead of idle, and when it calls vboxmanage that child process would inherit the normal priority, this issue would mostly clear up.

Since the wrapper is not using CPU slices, and it's managing the WU's in an environment where the CPU's are pegging 100% and users are trying to fit as many work units into RAM as possible, the wrapper needs to be dominant to accomplish it's management tasks.
ID: 33187 · Report as offensive     Reply Quote
marmot

Send message
Joined: 5 Nov 15
Posts: 90
Credit: 4,370,848
RAC: 34,240
Message 33411 - Posted: 16 Dec 2017, 14:56:24 UTC - in response to Message 33187.  

Adjusted the vboxwrapper and vboxmanage processes to above normal priority but the dirty environment errors persist on all machines.

Maybe a bug in Oracle's VBox that LHC has no control over.
ID: 33411 · Report as offensive     Reply Quote

Message boards : Number crunching : VM environment need to be cleaned up.


©2018 CERN