LHC@home

So I went away for a few days, no changes in the system, and when I get back I see I have a bunch of CMS errors.
This is the first of them and seems consistent with the rest, as in they all failed in about the same time and in the same way:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=407930894
As I said, I wasn't home and didn't change anything from when they were running successfully (no updates, nobody using the machine, no suspend/resume, no power cycling) so I don't know what's going on.

Full log:
<core_client_version>7.25.0</core_client_version>
<![CDATA[
<message>
process exited with code 12 (0xc, -244)</message>
<stderr_txt>
2024-03-18 06:48:55 (1463250): Detected: vboxwrapper 26206
2024-03-18 06:48:55 (1463250): Detected: BOINC client v7.25.0
2024-03-18 06:49:26 (1463250): Detected: VirtualBox VboxManage Interface (Version: 7.0.14)
2024-03-18 06:49:26 (1463250): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds)
2024-03-18 06:49:26 (1463250): Successfully copied 'init_data.xml' to the shared directory.
2024-03-18 06:49:26 (1463250): Create VM. (boinc_086a07090bec38dd, slot#11)
2024-03-18 06:49:27 (1463250): Setting Memory Size for VM. (2048MB)
2024-03-18 06:49:28 (1463250): Setting CPU Count for VM. (1)
2024-03-18 06:49:28 (1463250): Setting Chipset Options for VM.
2024-03-18 06:49:29 (1463250): Setting Graphics Controller Options for VM.
2024-03-18 06:49:30 (1463250): Setting Boot Options for VM.
2024-03-18 06:49:30 (1463250): Setting Network Configuration for NAT.
2024-03-18 06:49:31 (1463250): Enabling VM Network Access.
2024-03-18 06:49:31 (1463250): Disabling USB Support for VM.
2024-03-18 06:49:32 (1463250): Disabling COM Port Support for VM.
2024-03-18 06:49:32 (1463250): Disabling LPT Port Support for VM.
2024-03-18 06:49:33 (1463250): Disabling Audio Support for VM.
2024-03-18 06:49:34 (1463250): Disabling Clipboard Support for VM.
2024-03-18 06:49:34 (1463250): Disabling Drag and Drop Support for VM.
2024-03-18 06:49:35 (1463250): Adding storage controller(s) to VM.
2024-03-18 06:49:35 (1463250): Adding virtual disk drive to VM. (CMS_2022_09_07_prod.vdi)
2024-03-18 06:49:41 (1463250): Error in deregister parent vdi for VM: -2135228404
Command:
VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:41 (1463250): Could not create VM
2024-03-18 06:49:41 (1463250): ERROR: VM failed to start
2024-03-18 06:49:41 (1463250): Powering off VM.
2024-03-18 06:49:41 (1463250): Deregistering VM. (boinc_086a07090bec38dd, slot#11)
2024-03-18 06:49:41 (1463250): Removing network bandwidth throttle group from VM.
2024-03-18 06:49:41 (1463250): Removing VM from VirtualBox.

Hypervisor System Log:


VM Execution Log:


VM Startup Log:


VM Trace Log:

Exit Code: 0
Output:
Virtual machine 'boinc_086a07090bec38dd' is created and registered.
UUID: cf91882f-fcd4-4595-bbe9-7ef6eec900cc
Settings file: '/home/michael/media/BOINC/slots/11/boinc_086a07090bec38dd/boinc_086a07090bec38dd.vbox'

2024-03-18 06:49:27 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --description "CMS_3978621_1710704653.530945_0"
Exit Code: 0
Output:

2024-03-18 06:49:28 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --memory 2048
Exit Code: 0
Output:

2024-03-18 06:49:28 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --cpus 1
Exit Code: 0
Output:

2024-03-18 06:49:29 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --acpi on --ioapic on --rtcuseutc on
Exit Code: 0
Output:

2024-03-18 06:49:30 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --vram 16 --graphicscontroller VBoxVGA
Exit Code: 0
Output:

2024-03-18 06:49:30 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --boot1 disk --boot2 dvd --boot3 none --boot4 none
Exit Code: 0
Output:

2024-03-18 06:49:31 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --nic1 nat --natdnsproxy1 on --cableconnected1 off
Exit Code: 0
Output:

2024-03-18 06:49:31 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --cableconnected1 on
Exit Code: 0
Output:

2024-03-18 06:49:32 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --usb off
Exit Code: 0
Output:

2024-03-18 06:49:32 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --uart1 off --uart2 off
Exit Code: 0
Output:

2024-03-18 06:49:33 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --lpt1 off --lpt2 off
Exit Code: 0
Output:

2024-03-18 06:49:34 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --audio none
Exit Code: 0
Output:
Warning: --audio is deprecated and will be removed soon. Use --audio-driver instead!

2024-03-18 06:49:34 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --clipboard disabled
Exit Code: 0
Output:

2024-03-18 06:49:35 (1463250):
Command: VBoxManage -q modifyvm "boinc_086a07090bec38dd" --draganddrop disabled
Exit Code: 0
Output:

2024-03-18 06:49:35 (1463250):
Command: VBoxManage -q storagectl "boinc_086a07090bec38dd" --name "Hard Disk Controller" --add "sata" --controller "IntelAHCI" --hostiocache off --portcount 3
Exit Code: 0
Output:

2024-03-18 06:49:36 (1463250):
Command: VBoxManage -q showhdinfo "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: 0
Output:
UUID: dae25e8f-de18-4971-b11c-eca764ede402
Parent UUID: base
State: created
Type: multiattach
Location: /home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi
Storage format: VDI
Format variant: dynamic default
Capacity: 20480 MBytes
Size on disk: 3853 MBytes
Encryption: disabled
Property: AllocationBlockSize=1048576
Child UUIDs: 2ca0095e-b508-4889-ba0b-11d82d16a156

2024-03-18 06:49:36 (1463250):
Command: VBoxManage -q storageattach "boinc_086a07090bec38dd" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --mtype multiattach --medium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228409
Output:
VBoxManage: error: Cannot attach medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi': the media type 'MultiAttach' can only be attached to machines that were created with VirtualBox 4.0 or later
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component SessionMachine, interface IMachine, callee nsISupports
VBoxManage: error: Context: "AttachDevice(Bstr(pszCtl).raw(), port, device, DeviceType_HardDisk, pMedium2Mount)" at line 781 of file VBoxManageStorageController.cpp

2024-03-18 06:49:36 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:37 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:38 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:39 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:40 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:41 (1463250):
Command: VBoxManage -q closemedium "/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi"
Exit Code: -2135228404
Output:
VBoxManage: error: Cannot close medium '/home/michael/media/BOINC/projects/lhcathome.cern.ch_lhcathome/CMS_2022_09_07_prod.vdi' because it has 1 child media
VBoxManage: error: Details: code VBOX_E_OBJECT_IN_USE (0x80bb000c), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "Close()" at line 1875 of file VBoxManageDisk.cpp

2024-03-18 06:49:41 (1463250):
Command: VBoxManage -q snapshot "boinc_086a07090bec38dd" list
Exit Code: 0
Output:
This machine does not have any snapshots

2024-03-18 06:49:41 (1463250):
Command: VBoxManage -q bandwidthctl "boinc_086a07090bec38dd" remove "boinc_086a07090bec38dd_net"
Exit Code: -2135228415
Output:
VBoxManage: error: Could not find a bandwidth group named 'boinc_086a07090bec38dd_net'
VBoxManage: error: Details: code VBOX_E_OBJECT_NOT_FOUND (0x80bb0001), component BandwidthControlWrap, interface IBandwidthControl, callee nsISupports
VBoxManage: error: Context: "DeleteBandwidthGroup(name.raw())" at line 273 of file VBoxManageBandwidthControl.cpp

2024-03-18 06:49:42 (1463250):
Command: VBoxManage -q unregistervm "boinc_086a07090bec38dd" --delete
Exit Code: 0
Output:
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

06:49:46 (1463250): called boinc_finish(-2135228404)

</stderr_txt>
]]>

This is what has most likely happened:

CMS had no tasks for a while.
At restart your computer got a couple of them and started all concurrently.
This caused a race condition in Virtualbox while it attached the virtual disk.

Now the VirtualBox media registry is in an inconsistent state and needs to be cleaned up manually.


1. Ensure no vbox task is currently running
2. Not a must, but stop BOINC to avoid it starts any vbox tasks while you do the next steps
3. Use the account running BOINC to open the VirtualBox Media Manager
4. Remove the affected disk entry (here: CMS_2022_09_07_prod.vdi) and it's children; do NOT remove the parent vdi file when asked!
5. Restart BOINC
6. Start 1 (only 1!) fresh CMS task and wait until it has registered it's disk
7. Start other CMS tasks (even concurrently)

Thanks for that. I did note a single child vdi (under the CMS_2022_09_07_prod.vdi) listed in the VBox media manager and removed it. I'll remove that one in the manager as well now but not the original file. Will just have to wait until I get another CMS task as I have none in cache for now.

Apologies, I forgot to reply.

That was indeed the problem. Thanks.