LHC@home

Hello.

Things were running fine under ubuntu 20.04. I recently upgraded my system to 22.04, and all hell broke loose.

I am running boinc 7.18.1 and VirtualBox 6.1.38.

I managed to get atlas and theory jobs running, however CMS jobs (the one job that I got from the system) failed with the following:

<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
2023-11-19 20:55:35 (1341412): Detected: vboxwrapper 26206
2023-11-19 20:55:35 (1341412): Detected: BOINC client v7.18.1
2023-11-19 20:55:41 (1341412): Error in guest additions for VM: -1041038848
Command:
VBoxManage -q list systemproperties
Output:
VBoxManage: error: Failed to create the VirtualBox object!
VBoxManage: error: Code NS_ERROR_SOCKET_FAIL (0xC1F30200) - IPC daemon socket error (extended info not available)
VBoxManage: error: Most likely, the VirtualBox COM server is not running or failed to start.

2023-11-19 20:55:41 (1341412): Detected: VirtualBox VboxManage Interface (Version: 6.1.38)
2023-11-19 20:55:46 (1341412): Error in host info for VM: -1041038848
Command:
VBoxManage -q list hostinfo 
Output:
VBoxManage: error: Failed to create the VirtualBox object!
VBoxManage: error: Code NS_ERROR_SOCKET_FAIL (0xC1F30200) - IPC daemon socket error (extended info not available)
VBoxManage: error: Most likely, the VirtualBox COM server is not running or failed to start.

2023-11-19 20:55:46 (1341412): WARNING: Communication with VM Hypervisor failed.
2023-11-19 20:55:46 (1341412): ERROR: VBoxManage list hostinfo failed
20:55:46 (1341412): called boinc_finish(1)

</stderr_txt>
]]>

The job in question is https://lhcathome.cern.ch/lhcathome/result.php?resultid=401935304

Funny thing is ... if I su as the boinc user

sudo -u boinc bash

and I try the VBoxManager commands that failed above, they work fine. For example:

boinc@oceanis:~/projects/lhcathome.cern.ch_lhcathome$ VBoxManage -q list systemproperties
API version:                     6_1
Minimum guest RAM size:          4 Megabytes
Maximum guest RAM size:          2097152 Megabytes
Minimum video RAM size:          0 Megabytes
Maximum video RAM size:          256 Megabytes
Maximum guest monitor count:     64
Minimum guest CPU count:         1
Maximum guest CPU count:         32
Virtual disk limit (info):       2199022206976 Bytes
Maximum Serial Port count:       4
Maximum Parallel Port count:     2
Maximum Boot Position:           4
Maximum PIIX3 Network Adapter count:   8
Maximum ICH9 Network Adapter count:   36
Maximum PIIX3 IDE Controllers:   1
Maximum ICH9 IDE Controllers:    1
Maximum IDE Port count:          2
Maximum Devices per IDE Port:    2
Maximum PIIX3 SATA Controllers:  1
Maximum ICH9 SATA Controllers:   8
Maximum SATA Port count:         30
Maximum Devices per SATA Port:   1
Maximum PIIX3 SCSI Controllers:  1
Maximum ICH9 SCSI Controllers:   8
Maximum SCSI Port count:         16
Maximum Devices per SCSI Port:   1
Maximum SAS PIIX3 Controllers:   1
Maximum SAS ICH9 Controllers:    8
Maximum SAS Port count:          255
Maximum Devices per SAS Port:    1
Maximum NVMe PIIX3 Controllers:  1
Maximum NVMe ICH9 Controllers:   8
Maximum NVMe Port count:         255
Maximum Devices per NVMe Port:   1
Maximum virtio-scsi PIIX3 Controllers:  1
Maximum virtio-scsi ICH9 Controllers:   8
Maximum virtio-scsi Port count:         256
Maximum Devices per virtio-scsi Port:   1
Maximum PIIX3 Floppy Controllers:1
Maximum ICH9 Floppy Controllers: 1
Maximum Floppy Port count:       1
Maximum Devices per Floppy Port: 2
Default machine folder:          /data/boinc/VirtualBox VMs
Raw-mode Supported:              no
Exclusive HW virtualization use: on
Default hard disk format:        VDI
VRDE auth library:               VBoxAuth
Webservice auth. library:        VBoxAuth
Remote desktop ExtPack:          Oracle VM VirtualBox Extension Pack
Log history count:               3
Default frontend:                
Default audio driver:            Pulse
Autostart database path:         
Default Guest Additions ISO:     /usr/share/virtualbox/VBoxGuestAdditions.iso
Logging Level:                   all
Proxy Mode:                      System
Proxy URL:                       
boinc@oceanis:~/projects/lhcathome.cern.ch_lhcathome$ VBoxManage -q list hostinfo
Host Information:

Host time: 2023-11-20T03:25:22.825000000Z
Processor online count: 40
Processor count: 40
Processor online core count: 20
Processor core count: 20
Processor supports HW virtualization: yes
Processor supports PAE: yes
Processor supports long mode: yes
Processor supports nested paging: yes
Processor supports unrestricted guest: yes
Processor supports nested HW virtualization: no
Processor#0 speed: 3300 MHz
Processor#0 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#1 speed: 3300 MHz
Processor#1 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#2 speed: 3300 MHz
Processor#2 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#3 speed: 3300 MHz
Processor#3 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#4 speed: 3300 MHz
Processor#4 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#5 speed: 3300 MHz
Processor#5 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#6 speed: 3300 MHz
Processor#6 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#7 speed: 3300 MHz
Processor#7 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#8 speed: 3300 MHz
Processor#8 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#9 speed: 3300 MHz
Processor#9 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#10 speed: 3300 MHz
Processor#10 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#11 speed: 3300 MHz
Processor#11 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#12 speed: 3300 MHz
Processor#12 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#13 speed: 3300 MHz
Processor#13 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#14 speed: 3300 MHz
Processor#14 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#15 speed: 3300 MHz
Processor#15 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#16 speed: 3300 MHz
Processor#16 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#17 speed: 3300 MHz
Processor#17 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#18 speed: 3300 MHz
Processor#18 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#19 speed: 3300 MHz
Processor#19 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#20 speed: 3300 MHz
Processor#20 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#21 speed: 3300 MHz
Processor#21 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#22 speed: 3300 MHz
Processor#22 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#23 speed: 3300 MHz
Processor#23 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#24 speed: 3300 MHz
Processor#24 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#25 speed: 3300 MHz
Processor#25 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#26 speed: 3300 MHz
Processor#26 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#27 speed: 3300 MHz
Processor#27 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#28 speed: 3300 MHz
Processor#28 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#29 speed: 3300 MHz
Processor#29 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#30 speed: 3300 MHz
Processor#30 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#31 speed: 3300 MHz
Processor#31 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#32 speed: 3300 MHz
Processor#32 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#33 speed: 3300 MHz
Processor#33 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#34 speed: 3300 MHz
Processor#34 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#35 speed: 3300 MHz
Processor#35 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#36 speed: 3300 MHz
Processor#36 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#37 speed: 3300 MHz
Processor#37 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#38 speed: 3300 MHz
Processor#38 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Processor#39 speed: 3300 MHz
Processor#39 description: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Memory size: 112714 MByte
Memory available: 109950 MByte
Operating system: Linux
Operating system version: 5.15.0-88-generic
boinc@oceanis:~/projects/lhcathome.cern.ch_lhcathome$ 

Virtuabox as another user and as root also work fine.

VBox extensions are also installed:

boinc@oceanis:~/projects/lhcathome.cern.ch_lhcathome$ dpkg -l | grep -i virtualbox
ii  virtualbox                                 6.1.38-dfsg-3~ubuntu1.22.04.1              amd64        x86 virtualization solution - base binaries
ii  virtualbox-dkms                            6.1.38-dfsg-3~ubuntu1.22.04.1              amd64        x86 virtualization solution - kernel module sources for dkms
ii  virtualbox-ext-pack                        6.1.38-1~ubuntu1.22.04.1                   all          extra capabilities for VirtualBox, downloader.
ii  virtualbox-guest-additions-iso             6.1.38-1~ubuntu1.22.04.1                   all          guest additions iso image for VirtualBox
ii  virtualbox-qt                              6.1.38-dfsg-3~ubuntu1.22.04.1              amd64        x86 virtualization solution - Qt based user interface
boinc@oceanis:~/projects/lhcathome.cern.ch_lhcathome$ 

Any help is appreciated. Thank you!

I managed to get atlas and theory jobs running

Your computer runs ATLAS/Theory native which don't use VirtualBox.

Your CMS log mentions this:

VBoxManage: error: Code NS_ERROR_SOCKET_FAIL (0xC1F30200) - IPC daemon socket error (extended info not available)
VBoxManage: error: Most likely, the VirtualBox COM server is not running or failed to start.

It either
- means what is written there: VirtualBox ... is not running or failed to start
or
- it is caused by Ubuntu 22.04 which is known to be more strict than Ubuntu 20.x



You need to ensure that
- your BOINC user is a member of the virtualbox group
- your BOINC service does NOT use "ProtectSystem=strict" -> See: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=6023&postid=48398

Thanks @computezrmle,

BOINC is in the vboxusers group (there is no virtualbox group), and as I said in the original post, su'ing as a boinc user lets me run VirtualBox, so that's not the issue.

Thank you for the pointer to set ProtectSystem=full. I've set that (and restarted boinc), and now I'm waiting got some jobs to see if it works.

Thanks again!

George

Yep, that works. I've gotten a few CMS tasks and they seem to be working. I haven't had any compete yet, but hopefully they will.

Thank you!

George

Ok, this is a different error for CMS tasks and I can't work it out. Nothing I've found searching online gives me a clue.
My boinc client is a local compile, I do this as the distro one is older than I want, so it's not running as a service or as a difference user.
I'm only having trouble with CMS.
Virtualbox clients run fine.
I'm a member of the VBox Users group.
Boinc is running as me.
I created a boinc group because previous errors complained I wasn't a member of the boinc group.

Here's the unit in question:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=403631861

Here's the unit log:

<core_client_version>7.25.0</core_client_version>
<![CDATA[
<message>
process exited with code 208 (0xd0, -48)</message>
<stderr_txt>
2023-12-25 17:14:16 (1205245): Detected: vboxwrapper 26206
2023-12-25 17:14:16 (1205245): Detected: BOINC client v7.25.0
2023-12-25 17:14:16 (1205245): Detected: VirtualBox VboxManage Interface (Version: 7.0.12)
2023-12-25 17:14:16 (1205245): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds)
2023-12-25 17:14:16 (1205245): Successfully copied 'init_data.xml' to the shared directory.
2023-12-25 17:14:16 (1205245): Create VM. (boinc_6452632b15b63388, slot#4)
2023-12-25 17:14:17 (1205245): Setting Memory Size for VM. (2048MB)
2023-12-25 17:14:17 (1205245): Setting CPU Count for VM. (1)
2023-12-25 17:14:17 (1205245): Setting Chipset Options for VM.
2023-12-25 17:14:17 (1205245): Setting Graphics Controller Options for VM.
2023-12-25 17:14:17 (1205245): Setting Boot Options for VM.
2023-12-25 17:14:17 (1205245): Setting Network Configuration for NAT.
2023-12-25 17:14:17 (1205245): Enabling VM Network Access.
2023-12-25 17:14:17 (1205245): Disabling USB Support for VM.
2023-12-25 17:14:18 (1205245): Disabling COM Port Support for VM.
2023-12-25 17:14:18 (1205245): Disabling LPT Port Support for VM.
2023-12-25 17:14:18 (1205245): Disabling Audio Support for VM.
2023-12-25 17:14:18 (1205245): Disabling Clipboard Support for VM.
2023-12-25 17:14:18 (1205245): Disabling Drag and Drop Support for VM.
2023-12-25 17:14:18 (1205245): Adding storage controller(s) to VM.
2023-12-25 17:14:18 (1205245): Adding virtual disk drive to VM. (CMS_2022_09_07_prod.vdi)
2023-12-25 17:14:20 (1205245): Adding VirtualBox Guest Additions to VM.
2023-12-25 17:14:20 (1205245): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB)
2023-12-25 17:14:20 (1205245): forwarding host port 49185 to guest port 80
2023-12-25 17:14:20 (1205245): Enabling remote desktop for VM.
2023-12-25 17:14:21 (1205245): Enabling shared directory for VM.
2023-12-25 17:14:21 (1205245): Starting VM using VBoxManage interface. (boinc_6452632b15b63388, slot#4)
2023-12-25 17:14:22 (1205245): Successfully started VM. (PID = '1205881')
2023-12-25 17:14:22 (1205245): Reporting VM Process ID to BOINC.
2023-12-25 17:14:22 (1205245): Guest Log: BIOS: VirtualBox 7.0.12
2023-12-25 17:14:22 (1205245): Guest Log: CPUID EDX: 0x178bfbff
2023-12-25 17:14:22 (1205245): Guest Log: BIOS: No PCI IDE controller, not probing IDE
2023-12-25 17:14:22 (1205245): Guest Log: BIOS: AHCI 0-P#0: PCHS=16383/16/63 LCHS=1024/255/63 0x0000000002800000 sectors
2023-12-25 17:14:22 (1205245): VM state change detected. (old = 'poweredoff', new = 'running')
2023-12-25 17:14:22 (1205245): Detected: Web Application Enabled (http://localhost:49185)
2023-12-25 17:14:22 (1205245): Detected: Remote Desktop Enabled (localhost:50239)
2023-12-25 17:14:22 (1205245): Preference change detected
2023-12-25 17:14:22 (1205245): Setting CPU throttle for VM. (100%)
2023-12-25 17:14:22 (1205245): Setting checkpoint interval to 600 seconds. (Higher value of (Preference: 600 seconds) or (Vbox_job.xml: 600 seconds))
2023-12-25 17:14:24 (1205245): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032
2023-12-25 17:14:24 (1205245): Guest Log: BIOS: Booting from Hard Disk...
2023-12-25 17:14:27 (1205245): Guest Log: BIOS: KBD: unsupported int 16h function 03
2023-12-25 17:14:27 (1205245): Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000
2023-12-25 17:14:55 (1205245): Guest Log: vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds
2023-12-25 17:14:55 (1205245): Guest Log: vboxguest: misc device minor 56, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
2023-12-25 17:14:56 (1205245): Guest Log: VBoxService 5.2.6 r120293 (verbosity: 0) linux.amd64 (Jan 15 2018 14:51:00) release log
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000064 main Log opened 2023-12-25T06:14:56.165527000Z
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000134 main OS Product: Linux
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000151 main OS Release: 4.14.232-19.cernvm.x86_64
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000160 main OS Version: #1 SMP Fri Apr 30 17:12:25 CEST 2021
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000168 main Executable: /usr/sbin/VBoxService
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000169 main Process ID: 2165
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.000169 main Package type: LINUX_64BITS_GENERIC
2023-12-25 17:14:56 (1205245): Guest Log: 00:00:00.002314 main 5.2.6 r120293 started. Verbose level = 0
2023-12-25 17:15:16 (1205245): Guest Log: [INFO] Mounting the shared directory
2023-12-25 17:15:16 (1205245): Guest Log: [INFO] Shared directory mounted, enabling vboxmonitor
2023-12-25 17:15:17 (1205245): Guest Log: [INFO] Sourcing essential functions from /cvmfs/grid.cern.ch
2023-12-25 17:15:17 (1205245): Guest Log: [INFO] Testing connection to cern.ch
2023-12-25 17:15:18 (1205245): Guest Log: [INFO] Testing connection to VCCS
2023-12-25 17:15:19 (1205245): Guest Log: [INFO] Testing connection to HTCondor
2023-12-25 17:15:20 (1205245): Guest Log: [INFO] Testing connection to WMAgent
2023-12-25 17:15:20 (1205245): Guest Log: [INFO] Testing connection to EOSCMS
2023-12-25 17:15:21 (1205245): Guest Log: [INFO] Testing connection to CMS-Factory
2023-12-25 17:15:22 (1205245): Guest Log: [INFO] Testing connection to CMS-Frontier
2023-12-25 17:15:23 (1205245): Guest Log: [INFO] Testing connection to Frontier
2023-12-25 17:15:25 (1205245): Guest Log: [INFO] Got a proxy from the local BOINC client
2023-12-25 17:15:25 (1205245): Guest Log: [INFO] Will use it for CVMFS and Frontier
2023-12-25 17:15:26 (1205245): Guest Log: [INFO] Reloading and probing the CVMFS configuration
2023-12-25 17:15:32 (1205245): Guest Log: [INFO] Probing /cvmfs/grid.cern.ch... OK
2023-12-25 17:15:35 (1205245): Guest Log: [INFO] Probing /cvmfs/cms.cern.ch... OK
2023-12-25 17:15:36 (1205245): Guest Log: [INFO] Probing /cvmfs/singularity.opensciencegrid.org... OK
2023-12-25 17:15:37 (1205245): Guest Log: [INFO] Probing /cvmfs/oasis.opensciencegrid.org... OK
2023-12-25 17:15:39 (1205245): Guest Log: [INFO] Probing /cvmfs/cms-ib.cern.ch... OK
2023-12-25 17:15:39 (1205245): Guest Log: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY
2023-12-25 17:15:39 (1205245): Guest Log: [INFO] 2.7.2.0 http://s1swinburne-cvmfs.openhtc.io:8080 http://192.168.1.253:3128
2023-12-25 17:15:39 (1205245): Guest Log: [INFO] Environment HTTP proxy: http://192.168.1.253:3128
2023-12-25 17:15:39 (1205245): Guest Log: [INFO] Reading volunteer information
2023-12-25 17:15:47 (1205245): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2023-12-25 17:15:49 (1205245): Guest Log: [INFO] CMS application starting. Check log files.
2023-12-25 17:15:50 (1205245): Guest Log: [INFO] Requesting an idtoken from LHC@home
2023-12-25 17:36:08 (1205245): Guest Log: [ERROR] glidein exited with return value 1.
2023-12-25 17:36:08 (1205245): Guest Log: [DEBUG] Volunteer: Dark Angel (268818)
2023-12-25 17:36:08 (1205245): Guest Log: [INFO] Shutting Down.
2023-12-25 17:36:38 (1205245): VM Completion File Detected.
2023-12-25 17:36:38 (1205245): VM Completion Message: glidein exited with return value 1.
.
2023-12-25 17:36:38 (1205245): Powering off VM.
2023-12-25 17:36:38 (1205245): Successfully stopped VM.
2023-12-25 17:36:38 (1205245): Deregistering VM. (boinc_6452632b15b63388, slot#4)
2023-12-25 17:36:39 (1205245): Removing network bandwidth throttle group from VM.
2023-12-25 17:36:39 (1205245): Removing VM from VirtualBox.
17:36:44 (1205245): called boinc_finish(208)

</stderr_txt>
]]>

2023-12-25 17:15:50 (1205245): Guest Log: [INFO] Requesting an idtoken from LHC@home
2023-12-25 17:36:08 (1205245): Guest Log: [ERROR] glidein exited with return value 1.
This need support, but the next two weeks....

Ok, I've noticed two things, neither of which might be relevant or they could be ... I don't know.

I ran out of Atlas tasks. Every CMS task that failed did so while I also had Atlas tasks (three at a time) running.
I checked the permissions on the Slots folder, and issued "chmod +wrx -R slots" to address a suspected inconsistency.
The next CMS task I allowed to start has been running for over ten hours now so I started eleven more to push the issue and those additional tasks have all been running for about an hour without issue.

There are also two long running Theory Native tasks currently running and the rest of my 24 threads are taken up with Asteroids tasks (nine) and one thread kept free for system stuff.