Message boards : CMS Application : VM Hypervisor failed to enter an online state in a timely fashion
Message board moderation

To post messages, you must log in.

AuthorMessage
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52700 - Posted: 26 Nov 2025, 19:03:35 UTC

For the first time ever I finally got a CMS WU to DL and start but it fails repeatedly saying:
Postponed: VM Hypervisor failed to enter an online state in a timely fashion. (4 CPUs)
It's the only thing running on that computer. I've rebooted the computer several times to get it to restart but it halts at 0.552%.
Anyone know how to fix it?
Linux Mint 21.3
VirtualBox 6.1.50
BOINC 8.2.8
i9-10980XE
4 x 16 GB RAM
https://lhcathome.cern.ch/lhcathome/result.php?resultid=430688722
ID: 52700 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52701 - Posted: 26 Nov 2025, 19:09:32 UTC

Here's how I got CMS to run:
If needed, enable virtualization in the BIOS. (All my CPUs have virtualization. Most have an LHC message saying virtualbox not installed when it already is installed.)
sudo apt-get install virtualbox -y

wget https://cvmrepo.web.cern.ch/cvmrepo/apt/cvmfs-release-latest_all.deb
sudo dpkg -i cvmfs-release-latest_all.deb
rm -f cvmfs-release-latest_all.deb
sudo apt update && sudo apt upgrade -y

https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-libs_2.13.3+ubuntu22.04_amd64.deb
https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-fuse3_2.13.3+ubuntu22.04_amd64.deb
https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs_2.13.3+ubuntu22.04_amd64.deb
https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-dev_2.13.3+ubuntu22.04_amd64.deb

sudo apt-get install autofs -y ; sudo systemctl enable autofs ; sudo systemctl status autofs

If CVMFS is installed for the very first time run once after the packets are installed:
sudo cvmfs_config setup

To be used without a local proxy:
sudo xed /etc/cvmfs/default.local

CVMFS_HTTP_PROXY="auto;DIRECT"
# set quota limit between 4000 and 10000
CVMFS_QUOTA_LIMIT=6000
# to be used for ATLAS native and Theory native
CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice"
# enable Cloudflare's CDN
# a MUST if you don't use a local proxy
CVMFS_USE_CDN=yes
# remove CVMFS_SERVER_URL from this file since it will be set automatically

sudo xed /etc/cvmfs/domain.d/cern.ch.local
# usually this is the only option that should be set in this file
CVMFS_CONFIG_REPO_REQUIRED=yes

sudo xed /etc/cvmfs/config.d/cvmfs-config.cern.ch.local
# usually this is the only option that should be set in this file
CVMFS_CONFIG_REPO_REQUIRED=no

cvmfs_config probe

https://cernvm-forum.cern.ch/t/warning-for-cvmrepo-during-sudo-apt-get-update/362/2
Warning for cvmrepo during sudo apt-get update
As a workaround, could you try changing your /etc/apt/sources.list.d/cernvm.list file to read:
sudo xed /etc/apt/sources.list.d/cernvm.list
deb [arch=amd64] http://cvmrepo.s3.cern.ch/cvmrepo/apt jammy-prod main
# deb http://cvmrepo.s3.cern.ch/cvmrepo/apt jammy-testing main

sudo reboot

To check BIOS setting from cli:
sudo dmidecode | more

Check client_state.xml that <p_vm_extensions_disabled>0</p_vm_extensions_disabled> is zero and not one.
ID: 52701 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52702 - Posted: 26 Nov 2025, 21:44:56 UTC - in response to Message 52701.  

Looks like you mix a few things:

1. ATLAS native is currently the only app here that requires a local CVMFS client.
None of the other apps require it.
All the stuff related to CVMFS is not required for CMS.


2.1 To run vbox apps your computer needs a CPU that supports hardware virtualization AND it must be enabled in the BIOS.
2.2 VirtualBox must be installed
2.3 The user running BOINC must be a member of the group vboxusers
2.4 <p_vm_extensions_disabled> in client_state.xml must not be set to 1

Also good to know:
If you set your prefs to allow ATLAS native and CMS/Theory on a computer you will also get ATLAS vbox work.
ID: 52702 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52703 - Posted: 26 Nov 2025, 21:52:20 UTC
Last modified: 26 Nov 2025, 21:55:32 UTC

How can I strip all the superfluous stuff?

I'm trying it on a computer where I only installed VirtualBox. It says it's not installed:
Computer: Local machine
23262	LHC@home	Nov 26, 2025, 01:52:59 PM	Message from server: VirtualBox is not installed
How can I get VB recognized? TIA
ID: 52703 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52704 - Posted: 26 Nov 2025, 22:07:56 UTC - in response to Message 52703.  

You got VirtualBox working here:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10970631

BTW
VirtualBox 6.1.50 is not wrong, but consider to upgrade to 7.2.2 which is more recent and does not have the issues 7.2.4 has.
If your Linux vendor does not provide it you can download it here:
https://www.virtualbox.org/wiki/Download_Old_Builds
ID: 52704 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52705 - Posted: 27 Nov 2025, 15:43:39 UTC
Last modified: 27 Nov 2025, 15:48:13 UTC

I installed VB 7.2.2 on two Linux Mint 21.3 computers and launched it. Was warned there was no folder /home/aurum/VirtualBox VMs so I created it and the warning went away.
I joined the vboxusers group and confirmed it by opening Users and Groups. Rebooted and still LHC does not see VB.
I found this and tried it but LHC still doesn't see VB.
BOINC complains that VirtualBox is not installed, what to do? https://lhcathome.web.cern.ch/boinc-complains-virtualbox-not-installed-what-do
This command boinc -dir ${HOME}/boinc does not complete and return my command prompt. Been stuck here for several hours:
27-Nov-2025 05:02:46 [---] Running CPU benchmarks
27-Nov-2025 05:02:46 [---] Suspending computation - CPU benchmarks in progress
27-Nov-2025 05:03:17 [---] Benchmark results:
27-Nov-2025 05:03:17 [---] Number of CPUs: 36
27-Nov-2025 05:03:17 [---] 2905 floating point MIPS (Whetstone) per CPU
27-Nov-2025 05:03:17 [---] 5688 integer MIPS (Dhrystone) per CPU
27-Nov-2025 05:03:18 [---] Suspending GPU computation - computer is in use
27-Nov-2025 05:05:42 [---] Resuming GPU computation
On the second computer I tried closing the terminal window and rebooting. BOINC works but LHC still doesn't see my VB.
This makes a copy of a number of files from my Elevated Privileges BOINC folder including my password file gui_rpc_auth.cfg with a different password.
The stdoutgpudetect.txt copy says "27-Nov-2025 04:58:55 [---] cc_config.xml not found - using defaults" but I do have a cc_config.xml still in my real BOINC folder.
I can't see anything in these files that solves this problem and won't be trying that again. Hopefully I can delete this useless folder without breaking BOINC.

As for Rig-05 I don't know how LHC sees my VB but BOINC 8.2.8 is broken and dev says I must've left something on it from a previous install. I'm going to wipe its drive and do a fresh LM 21.3 build.
Surely some Linux Mint user has been able to run CMS WUs.
ID: 52705 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52708 - Posted: 28 Nov 2025, 8:26:41 UTC - in response to Message 52705.  

BOINC can either be started
1. as a user app
2. as a service (recommended under Linux)

In case of (1.) it runs under the user's original account (in your case may be 'aurum').
In case of (2.) it runs under the username 'boinc' (which is created by the installer).


If you want to run vbox VMs via BOINC
install VirtualBox and verify the user running BOINC is a member of the group 'vboxusers'.
In case of (1.) that would be 'aurum'.
In case of (2.) that would be 'boinc'.

To verify the group membership run:
id -Gn boinc |grep vboxusers
# optional, if 'id' is not installed
grep vboxusers /etc/group

If this is missing, add 'boinc' to 'vboxusers':
sudo useradd -G vboxusers boinc

=> at least boinc service must be restarted once after this
=> after a fresh VirtualBox installation a reboot is recommended


You can verify if VirtualBox is running (post the output here):
lsmod |grep vbox
VBoxManage --version
VBoxManage list systemproperties
VBoxManage list hostinfo


If boinc is running as a service do NOT start it again e.g. via 'boinc -dir ${HOME}/boinc'.
Instead, use this to check it:
systemctl status boinc-client
ID: 52708 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52709 - Posted: 28 Nov 2025, 13:27:40 UTC

Here's a real mystery. I rebuilt Rig-05. I installed VB 7.2.2 and joined vboxusers before installing BOINC 8.2.8. Started off that LHC recognized VB and sent 5 CMS WUs that completed successfully:
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10982337
While they were running the VirtualBox in not installed appeared again and Rig-05 details no longer showed virtualization details.
Appears the problem lies with the LHC server.
ID: 52709 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52711 - Posted: 28 Nov 2025, 16:03:47 UTC - in response to Message 52709.  
Last modified: 28 Nov 2025, 16:04:19 UTC

This is typical for multiple local BOINC instances using the same host ID.
One of them is aware of VirtualBox and gets vbox tasks.
At least 1 other instance is not aware and reports VirtualBox is not installed.

Things like that usually happen when you copy BOINC's data directory to various locations and run 1 instance as service and another instance from the command line.
ID: 52711 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52714 - Posted: 30 Nov 2025, 13:47:50 UTC - in response to Message 52711.  
Last modified: 30 Nov 2025, 14:04:26 UTC

This is typical for multiple local BOINC instances using the same host ID.
One of them is aware of VirtualBox and gets vbox tasks.
At least 1 other instance is not aware and reports VirtualBox is not installed.

Things like that usually happen when you copy BOINC's data directory to various locations and run 1 instance as service and another instance from the command line.
I've never done that.
The problem lies entirely with the LHC server. Will anyone attempt to fix it?
Besides every computer I have now has VirtualBox and has a different name. Exactly where do I find what you call a "host ID"? My computers all have a different "Computer ID" and "Name".
https://lhcathome.cern.ch/lhcathome/hosts_user.php?sort=id&rev=0&show_all=0&userid=571928
I explained clearly how I rebuilt Rig-05 from scratch and it did not involve copying anything from another computer on my LAN.
I'll now detach every computer but Rig-05 from LHC and see if it revives.
ID: 52714 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52717 - Posted: 30 Nov 2025, 17:47:26 UTC - in response to Message 52714.  

The problem lies entirely with the LHC server.

Do you have any evidence, e.g. a log or anything like that?
If yes, then present it.
If not, this is nothing but a weak claim.

Will anyone attempt to fix it?

No, unless you present any evidence.

At the main BOINC forum you claimed BOINC 8.2.8 is broken although you did a 'clean install'.
https://boinc.berkeley.edu//forum_thread.php?id=15633&postid=117554
The log snippets you posted there clearly show that you also installed an outdated script from here that must not be used any more on recent Linux systems.
That script was never part of the original BOINC installer but you seem to have ignored that information.

I explained clearly how I rebuilt Rig-05 from scratch

No, you didn't.
Instead, you missed important steps, e.g. how the outdated script made it to your computer.
It requires root credentials to install the script, so please don't blame the BOINC client or the project server as both don't run as root on your computer.
ID: 52717 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52723 - Posted: 4 Dec 2025, 12:25:33 UTC

Is anyone successfully running CMS WUs on Linux Mint 21.3 Virginia Ubuntu 22.04 jammy???
If so how did you do it?
ID: 52723 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52724 - Posted: 4 Dec 2025, 12:39:53 UTC - in response to Message 52717.  
Last modified: 4 Dec 2025, 12:42:08 UTC

I explained clearly how I rebuilt Rig-05 from scratch.
No, you didn't.
Instead, you missed important steps, e.g. how the outdated script made it to your computer.
It requires root credentials to install the script, so please don't blame the BOINC client or the project server as both don't run as root on your computer.

I don't know how it got there. Probably when LM is installed from an ico it has 4 options and I picked the wrong leaving old programs there. This time i picked erase drive and install LM 21.3 as stated below. Then I installed Virtualbox 7.2.2 as you recommended. Then I joined vboxusers. Then LHC recognized virtualization and DLed 5 WUs that completed successfully. Before they completed the CMS server claimed I did not have VB installed and sent no further units.
Then you made the specious claim that I copied something to Rig-05. I have never installed anything by coping it from another computer on my LAN. I install from repositories.
I detached all computers but Rig-05 from LHC so your CMS server can't get confused but it still does.
Tell me what this log is and I'll post it.
From my vantage CMS is incapable of recognizing properly configured Linux Mint computers with Virtualbox 7.2.2 installed and both boinc and aurum are members of the vboxusers group.
BTW, I turned off Rig-05 and am now attempting to use Rig-10 to run CMS WUs but the CMS server is incapable of recognizing that has Virtualbox 7.2.2 installed.
ID: 52724 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52725 - Posted: 5 Dec 2025, 9:19:47 UTC - in response to Message 52724.  
Last modified: 5 Dec 2025, 9:37:52 UTC

I detached all computers but Rig-05 from LHC so your CMS server can't get confused but it still does.

You still ignore this fact:
The project server does not change anything unless your client initiates a scheduler request.
So, if you really detached all but 1 computers, why do 6 of them show a 'last contact' timestamp from early this morning?
Answer: Because all those clients sent a scheduler request to the server!


I recently asked you to run some test commands and post the output, but you never did that.
So, do it now:
lsmod |grep vbox

id -Gn boinc |grep vboxusers
# optional, if 'id' is not installed
grep vboxusers /etc/group

which VBoxManage
# if the last command reports a path to VBoxManage, run the next command
# this is what the BOINC client runs to test if VirtualBox is installed
$(which VBoxManage 2>/dev/null) --version

if grep -cE 'boinc |boinc$' ~/.bash_history; then grep -E 'boinc |boinc$' ~/.bash_history; fi

ps -AF |grep '/[b]oinc '

systemctl --no-pager status boinc-client

sudo find / -name "client_state.xml" |xargs -I {} sh -c "stat {}; sed -ne '/<host_info>/,/<\/host_info>/p' -e '/<master_url>.*lhcathome\.cern\.ch/,/<\/scheduler_url>/p' {}; echo"



Edit
Run the following command on each of your computers and check if you find any of the host IDs more than once - either on the same or across all computers.
sudo find / -name "client_state.xml" |xargs -I {} sh -c "stat {}; sed -ne '/<master_url>.*lhcathome\.cern\.ch/,/<\/scheduler_url>/ {/<hostid>/p}' 
{}; echo"
ID: 52725 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52752 - Posted: 15 Dec 2025, 16:51:18 UTC - in response to Message 52725.  
Last modified: 15 Dec 2025, 17:18:30 UTC

Haven't checked in lately but I've started chipping away at your list now. At this moment I have 7 computers running BOINC. Rig-50 has just successfully completed 8 WUs and has started the next round. I just rebuilt Rig-50 yesterday by installing Linux Mint 21.3 from an ico that erased the drive. Then I installed Virtualbox 7.2.2 and joined vboxusers. Then I installed BOINC 8.2.8 and attached to minecrafthome (GPU WU) and LHC. My LHC preferences were set for 8 CMS WUs with 8 CPUs and it sent me 8 with 4 CPUs. Other computers that failed or never got CMS WUs also had other projects attached and were also running Einstein CPU WUs.

Edit
Run the following command on each of your computers and check if you find any of the host IDs more than once - either on the same or across all computers.
sudo find / -name "client_state.xml" |xargs -I {} sh -c "stat {}; sed -ne '/<master_url>.*lhcathome\.cern\.ch/,/<\/scheduler_url>/ {/<hostid>/p}' 
{}; echo"
aurum@Rig-44:~$ sudo find / -name "client_state.xml" |xargs -I {} sh -c "stat {}; sed -ne '/<master_url>.*lhcathome\.cern\.ch/,/<\/scheduler_url>/ {/<hostid>/p}'
{}; echo"
find: ‘/proc/55080/task/55080/net’: Invalid argument
find: ‘/proc/55080/net’: Invalid argument
File: /var/lib/boinc/client_state.xml
Size: 442581 Blocks: 872 IO Block: 4096 regular file
Device: 10303h/66307d Inode: 12192706 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 128/ boinc) Gid: ( 136/ boinc)
Access: 2025-12-15 07:05:39.075657323 -0800
Modify: 2025-12-15 07:05:17.689359458 -0800
Change: 2025-12-15 07:05:17.692359500 -0800
Birth: 2025-12-15 07:05:17.683359374 -0800
sh: 2: /var/lib/boinc/client_state.xml: Permission denied

find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

Removed the carriage return:
aurum@Rig-44:~$ sudo find / -name "client_state.xml" |xargs -I {} sh -c "stat {}; sed -ne '/<master_url>.*lhcathome\.cern\.ch/,/<\/scheduler_url>/ {/<hostid>/p}' \ {}; echo"
find: ‘/proc/55080/task/55080/net’: Invalid argument
find: ‘/proc/55080/net’: Invalid argument
File: /var/lib/boinc/client_state.xml
Size: 442571 Blocks: 872 IO Block: 4096 regular file
Device: 10303h/66307d Inode: 12192680 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 128/ boinc) Gid: ( 136/ boinc)
Access: 2025-12-15 08:28:08.962421125 -0800
Modify: 2025-12-15 08:27:33.415013308 -0800
Change: 2025-12-15 08:27:33.419013354 -0800
Birth: 2025-12-15 08:27:33.408013228 -0800
sed: can't read /var/lib/boinc/client_state.xml: No such file or directory

find: ‘/run/user/1000/doc’: Permission denied
find: ‘/run/user/1000/gvfs’: Permission denied

Since permission was denied I searched manually on each computer.
LHC@home project hostid from client_state.xml
Rig-02: <hostid>10933324</hostid> only appears once on this computer and no other computer
Rig-03: <hostid>10988454</hostid> only appears once on this computer and no other computer
Rig-36: <hostid>10933327</hostid> only appears once on this computer and no other computer
Rig-44: <hostid>10938639</hostid> only appears once on this computer and no other computer
Rig-50: <hostid>10993869</hostid> only appears once on this computer and no other computer

/run/user/1000/doc and /run/user/1000/gvfs are binaries so I can't open them.
ID: 52752 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52753 - Posted: 15 Dec 2025, 18:50:18 UTC - in response to Message 52752.  

nally it runs fine on this computer:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10993869
And it wasn't a project server issue, was it?


It still doesn't run on this computer:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10982337
Hints:
The computer uses a more recent Linux kernel which requires a more recent VirtualBox version (currently 6.1.50).
Hence the typical errors:
[pre]2025-12-14 07:04:54 (31374): VM state change detected. (old = 'running', new = 'gurumeditation')
.
.
.
00:00:03.198128 emR3Debug: rc=VERR_VMM_SET_JMP_ABORTED_RESUME[/pre]
The internet gives lots of advice how to handle this.
I'm sure you will find them yourself.
ID: 52753 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52754 - Posted: 15 Dec 2025, 21:47:33 UTC - in response to Message 52753.  
Last modified: 15 Dec 2025, 21:50:12 UTC

So, finally it runs fine on this computer:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10993869
And it wasn't a project server issue, was it?
Jury is still out on that. May have something to do with not sharing RAM when another project is running. CMS is by far the most difficult project to get running I've ever tried and I've been doing BOINC and distributed computing for over two decades.
It still doesn't run on this computer:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10982337
Hints:
The computer uses a more recent Linux kernel which requires a more recent VirtualBox version (currently 6.1.50).
Hence the typical errors:
2025-12-14 07:04:54 (31374): VM state change detected. (old = 'running', new = 'gurumeditation')
00:00:03.198128 emR3Debug: rc=VERR_VMM_SET_JMP_ABORTED_RESUME

The internet gives lots of advice how to handle this.
I'm sure you will find them yourself.
After VB 7.2.2 failed repeatedly I thought I'd try again using the version installed by the UC Berkeley Linux repository and that failed as well.
I've been trying to post the other stuff you asked for but it's many times. I'll try until it gets there. Maybe there's a character in the output that's forbidden by LHC forum.
Can't create post. Failed to add post to DB.
ID: 52754 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52755 - Posted: 15 Dec 2025, 22:06:35 UTC

Rig-50 that successfully completed 8 CMS WUs today:
aurum@Rig-50:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 32768 0
vboxdrv 618496 7 vboxnetadp,vboxnetflt
aurum@Rig-50:~$ id -Gn boinc |grep vboxusers
aurum@Rig-50:~$ grep vboxusers /etc/group
vboxusers:x:138:aurum
aurum@Rig-50:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-50:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
aurum@Rig-50:~$ if grep -cE 'boinc |boinc$' ~/.bash_history; then grep -E 'boinc |boinc$' ~/.bash_history; fi
0
aurum@Rig-50:~$ ps -AF |grep '/[b]oinc '
aurum@Rig-50:~$ systemctl --no-pager status boinc-client
boinc-client.service - Berkeley Open Infrastructure Network Computing Client
Loaded: loaded (/lib/systemd/system/boinc-client.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2025-12-14 17:19:11 PST; 15h ago
Docs: man:boinc(1)
Main PID: 2177 (boinc)
Tasks: 169 (limit: 76635)
Memory: 4.2G
CPU: 2w 4d 7h 49min 2.297s
CGroup: /system.slice/boinc-client.service
2177 /usr/local/bin/boinc
4465 ../../projects/lhcathome.cern.ch_lhcathome/vboxwrapper_26210_x86_64-pc-linux-gnu --memory_size_mb 4584 --nthreads 4
4703 /usr/lib/virtualbox/VBoxSVC --auto-shutdown --inherit-startup-pipe 10
8273 /usr/lib/virtualbox/VBoxHeadless --comment boinc_042b01e6db197702 --startvm 00b03812-f2df-43e6-9812-f4946cda2df1 --vrde config
20002 ../../projects/lhcathome.cern.ch_lhcathome/vboxwrapper_26210_x86_64-pc-linux-gnu --memory_size_mb 4584 --nthreads 4
20488 /usr/lib/virtualbox/VBoxHeadless --comment boinc_5b5972108e1b7c98 --startvm 7d7b44e2-a168-4eac-b36a-45ce72235b4d --vrde config
20534 ../../projects/lhcathome.cern.ch_lhcathome/vboxwrapper_26210_x86_64-pc-linux-gnu --memory_size_mb 4584 --nthreads 4
20535 ../../projects/lhcathome.cern.ch_lhcathome/vboxwrapper_26210_x86_64-pc-linux-gnu --memory_size_mb 4584 --nthreads 4
20541 ../../projects/lhcathome.cern.ch_lhcathome/vboxwrapper_26210_x86_64-pc-linux-gnu --memory_size_mb 4584 --nthreads 4
21689 /usr/lib/virtualbox/VBoxHeadless --comment boinc_4f5ed10ff4ca83dc --startvm 14b1e628-8ea0-4639-9dfe-81b86218650f --vrde config
21878 /usr/lib/virtualbox/VBoxHeadless --comment boinc_478a18c3fcf35642 --startvm 393eb137-8b91-4475-a048-c4890f21cdb7 --vrde config
22062 /usr/lib/virtualbox/VBoxHeadless --comment boinc_676d81daaf1177fd --startvm c98f9eee-4234-4576-b747-779623beb07e --vrde config
22224 ../../projects/wuprop.boinc-af.org/data_collect_v4_432_x86_64-pc-linux-gnu__nci -n 60 -c 60 -g 2
22526 ../../projects/minecraftathome.com_minecrafthome/repeatdecor_1.00_x86_64-pc-linux-gnu-cuda.bin -s 7503313638548045824 -e 7503322434641068032

Dec 15 08:53:44 Rig-50 boinc[2177]: 15-Dec-2025 08:53:44 [minecrafthome] Scheduler request completed: got 1 new tasks
Dec 15 08:53:44 Rig-50 boinc[2177]: 15-Dec-2025 08:53:44 [minecrafthome] Project requested delay of 7 seconds
Dec 15 08:53:54 Rig-50 boinc[2177]: 15-Dec-2025 08:53:54 [minecrafthome] Sending scheduler request: To fetch work.
Dec 15 08:53:54 Rig-50 boinc[2177]: 15-Dec-2025 08:53:54 [minecrafthome] Requesting new tasks for NVIDIA GPU
Dec 15 08:53:55 Rig-50 boinc[2177]: 15-Dec-2025 08:53:55 [minecrafthome] Scheduler request completed: got 0 new tasks
Dec 15 08:53:55 Rig-50 boinc[2177]: 15-Dec-2025 08:53:55 [minecrafthome] Project requested delay of 7 seconds
Dec 15 09:04:52 Rig-50 boinc[2177]: 15-Dec-2025 09:04:52 [minecrafthome] Sending scheduler request: To fetch work.
Dec 15 09:04:52 Rig-50 boinc[2177]: 15-Dec-2025 09:04:52 [minecrafthome] Requesting new tasks for NVIDIA GPU
Dec 15 09:04:53 Rig-50 boinc[2177]: 15-Dec-2025 09:04:53 [minecrafthome] Scheduler request completed: got 0 new tasks
Dec 15 09:04:53 Rig-50 boinc[2177]: 15-Dec-2025 09:04:53 [minecrafthome] Project requested delay of 7 seconds
ID: 52755 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 138
Credit: 57,147,810
RAC: 16,840
Message 52757 - Posted: 16 Dec 2025, 15:39:25 UTC
Last modified: 16 Dec 2025, 15:48:32 UTC

From where I sit the CMS server is incapable of recognizing a properly installed VirtualBox 7.2.2 so the problem lies with the CMS server. Most compelling is the fact that one computer was running 8 WUs and before they finished the CMS server changed its mind and decided VB 7.2.2 was not installed any more. As a result none of these 6 computers are sent any CMS WUs.
Please post instructions for how to run CMS WUs as I'd like to support your project. I'm surprised there's not already a pinned post with working instructions.
aurum@Rig-02:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-02:~$ id -Gn boinc |grep vboxusers
boinc video render docker vboxusers
aurum@Rig-02:~$ grep vboxusers /etc/group
vboxusers:x:132:aurum,boinc
aurum@Rig-02:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-02:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Thu 11 Dec 2025 06:09:43 PM PST

aurum@Rig-03:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-03:~$ id -Gn boinc |grep vboxusers
aurum@Rig-03:~$ grep vboxusers /etc/group
vboxusers:x:142:aurum
aurum@Rig-03:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-03:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Thu 11 Dec 2025 05:32:18 PM PST

aurum@Rig-23:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-23:~$ id -Gn boinc |grep vboxusers
aurum@Rig-23:~$ grep vboxusers /etc/group
vboxusers:x:142:aurum
aurum@Rig-23:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-23:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Tue 16 Dec 2025 07:10:56 AM PST

aurum@Rig-36:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-36:~$ id -Gn boinc |grep vboxusers
aurum@Rig-36:~$ grep vboxusers /etc/group
vboxusers:x:136:aurum
aurum@Rig-36:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-36:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Thu 11 Dec 2025 05:54:40 PM PST

aurum@Rig-44:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-44:~$ id -Gn boinc |grep vboxusers
boinc video render docker vboxusers
aurum@Rig-44:~$ grep vboxusers /etc/group
vboxusers:x:143:aurum,boinc
aurum@Rig-44:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-44:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Thu 11 Dec 2025 04:39:13 PM PST

aurum@Rig-49:~$ lsmod |grep vbox
vboxnetadp 28672 0
vboxnetflt 36864 0
vboxdrv 696320 2 vboxnetadp,vboxnetflt
aurum@Rig-49:~$ id -Gn boinc |grep vboxusers
aurum@Rig-49:~$ grep vboxusers /etc/group
vboxusers:x:132:aurum
aurum@Rig-49:~$ which VBoxManage
/usr/bin/VBoxManage
aurum@Rig-49:~$ $(which VBoxManage 2>/dev/null) --version
7.2.2r170484
LHC@home: Notice from server
VirtualBox is not installed
Mon 15 Dec 2025 09:42:30 PM PST
ID: 52757 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2724
Credit: 299,002,782
RAC: 86,570
Message 52758 - Posted: 16 Dec 2025, 17:15:04 UTC - in response to Message 52757.  

aurum@Rig-02:~$ id -Gn boinc |grep vboxusers
boinc video render docker vboxusers

Remove docker/podman and all related packages.
Then reboot.

Recent BOINC clients complain about missing docker/podman.
Ignore this!
ID: 52758 · Report as offensive     Reply Quote

Message boards : CMS Application : VM Hypervisor failed to enter an online state in a timely fashion


©2026 CERN