LHC@home

Hi!
I'm busy on troubleshooting the last weeks with the wrapper and tasks of LHC@Home.

The new wrapper seems to work pretty well. I want to test it also with the ATLAS tasks.
The current status is, ATLAS tasks are running well. CMS and Theory tasks keep failing. Whatever I change, it makes no difference.

Because there are WORKING ATLAS tasks available. I was be able to test if the HD error was related to the wrapper or to the disk which was supplied.
It seems that the disk of CMS and THEORY are seen as disks made with older versions of VirtualBox (Before 4.0):

Last login: Wed Jan 29 22:16:07 on ttys000
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q createvm --name "test_123" --basefolder "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/test" --ostype "Linux26_64" --register
Virtual machine 'test_123' is created and registered.
UUID: dd2dea25-a9f2-41e8-87e7-c790f5dc4789
Settings file: '/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/test/test_123/test_123.vbox'
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storageattach "test_123" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --mtype multiattach --medium "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/CMS_2025_01_16_prod.vdi" 
VBoxManage: error: Could not find a controller named 'Hard Disk Controller'
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storagectl "test_123" --name "Hard Disk Controller" --add "sata" --controller "IntelAHCI" --hostiocache off --portcount 3
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storageattach "test_123" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --mtype multiattach --medium "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/CMS_2025_01_16_prod.vdi"
VBoxManage: error: Cannot change type for medium '/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/CMS_2025_01_16_prod.vdi': the media type 'MultiAttach' can only be used on media registered with a machine that was created with VirtualBox 4.0 or later
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "COMSETTER(Type)(enmMediumType)" at line 718 of file VBoxManageStorageController.cpp
VBoxManage: error: Failed to set the medium type
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storagectl "test_234" --name "Hard Disk Controller" --add "sata" --controller "IntelAHCI" --hostiocache off --portcount 3
VBoxManage: error: Too many storage controllers of this type
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component StorageControllerWrap, interface IStorageController, callee nsISupports
VBoxManage: error: Context: "AddStorageController(Bstr(pszCtl).raw(), StorageBus_SATA, ctl.asOutParam())" at line 1090 of file VBoxManageStorageController.cpp
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storagectl "test_234" --name "Hard Disk Controller" --add "sata" --controller "IntelAHCI" --hostiocache off --portcount 3
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storageattach "test_234" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --mtype multiattach --medium "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/CMS_2025_01_16_prod.vdi"
VBoxManage: error: Cannot change type for medium '/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/CMS_2025_01_16_prod.vdi': the media type 'MultiAttach' can only be used on media registered with a machine that was created with VirtualBox 4.0 or later
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MediumWrap, interface IMedium, callee nsISupports
VBoxManage: error: Context: "COMSETTER(Type)(enmMediumType)" at line 718 of file VBoxManageStorageController.cpp
VBoxManage: error: Failed to set the medium type
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storageattach "test_123" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --medium "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/ATLAS_vbox_3.01_image.vdi"
nentech@MacBook-Pro-van-Johan vboxwrapper % VBoxManage -q storageattach "test_234" --storagectl "Hard Disk Controller" --port 0 --device 0 --type hdd --medium "/Library/Application Support/BOINC Data/projects/lhcathome.cern.ch_lhcathome/ATLAS_vbox_3.01_image.vdi"

Please check the differences between the 2 projects and the different OS types. My Windows version works perfectly.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=419190518

The Hearbeat is not enabled in the ATLAS application. It is enabled in the Theory and CMS application. In the OS X version, this heartbeat is not written to the shared directory.

This is not a log created by any vboxwrapper version.

Instead it was created by a series of VBoxManage commands entered manually in a terminal.
Since that terminal environment is completely different to a BOINC task environment this command sequence is not really useful to troubleshoot the vboxwrapper issue on Apple.

It seems that the disk of CMS and THEORY are seen as disks made with older versions of VirtualBox (Before 4.0)

Nope.
It's a misleading VirtualBox message related to the MultiAttach option and vboxwrapper can deal with it.

This is not a log created by any vboxwrapper version.

Instead it was created by a series of VBoxManage commands entered manually in a terminal.
Since that terminal environment is completely different to a BOINC task environment this command sequence is not really useful to troubleshoot the vboxwrapper issue on Apple.

It is indeed not a log but a test that I get the same results via manual attach as via the wrapper. So the issue is not related to the wrapper. I can attach the ATLAS vdi without any issues. The log of failing CMS task are plenty in my account. This is the latest failing CMS task with wrapper 26206: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419192606

It seems that the disk of CMS and THEORY are seen as disks made with older versions of VirtualBox (Before 4.0)

Nope.
It's a misleading VirtualBox message related to the MultiAttach option and vboxwrapper can deal with it.

It can be misleading. But the ATLAS vdi is be able to attach via the MultiAttach while the CMS vdi can't.[/quote]

This is not a log created by any vboxwrapper version.

Instead it was created by a series of VBoxManage commands entered manually in a terminal.
Since that terminal environment is completely different to a BOINC task environment this command sequence is not really useful to troubleshoot the vboxwrapper issue on Apple.

It is indeed not a log but a test that I get the same results via manual attach as via the wrapper. So the issue is not related to the wrapper. I can attach the ATLAS vdi without any issues. The log of failing CMS task are plenty in my account. This is the latest failing CMS task with wrapper 26206: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419192606

It seems that the disk of CMS and THEORY are seen as disks made with older versions of VirtualBox (Before 4.0)

Nope.
It's a misleading VirtualBox message related to the MultiAttach option and vboxwrapper can deal with it.

It can be misleading. But the ATLAS vdi is be able to attach via the MultiAttach while the CMS vdi can't.

MultiAttach has been added to vboxwrapper in 2022.
More details here:
https://github.com/BOINC/boinc/pull/4603/files
But again, this is not the root cause for the recent issues on Apple.

Ok. What do you think were the root causes are for the recent issues on Apple?

My investigation has given the following results:
- VBoxWrapper 26206 -> ATLAS: Working
- VBoxWrapper 26210 -> ATLAS: Working
- VBoxWrapper 26206 -> CMS: Failing due to missing heartbeat
- VBoxWrapper 26208 -> CMS: Failing within 2 seconds
- VBoxWrapper 26210 -> CMS: Failing due to missing heartbeat
- VBoxWrapper 26210 (Disabled heartbeat) -> CMS: Very slow booting, no CFVMS probe, not starting work.

Task with wrapper 26206: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419186593
Task with wrapper 26210: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419189576

For the job below I added the heartbeat file manual.
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419448552


All CMS tasks are failing on Mac:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419511029

Is there anything I can do to help you guys fixing this issue?

The heartbeat file is not made on MacOS. On Windows it is present.