Message boards :
CMS Application :
VM Hypervisor failed to enter an online state in a timely fashion
Message board moderation
| Author | Message |
|---|---|
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
For the first time ever I finally got a CMS WU to DL and start but it fails repeatedly saying: Postponed: VM Hypervisor failed to enter an online state in a timely fashion. (4 CPUs)It's the only thing running on that computer. I've rebooted the computer several times to get it to restart but it halts at 0.552%. Anyone know how to fix it? Linux Mint 21.3 VirtualBox 6.1.50 BOINC 8.2.8 i9-10980XE 4 x 16 GB RAM https://lhcathome.cern.ch/lhcathome/result.php?resultid=430688722 |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
Here's how I got CMS to run: If needed, enable virtualization in the BIOS. (All my CPUs have virtualization. Most have an LHC message saying virtualbox not installed when it already is installed.) sudo apt-get install virtualbox -y wget https://cvmrepo.web.cern.ch/cvmrepo/apt/cvmfs-release-latest_all.deb sudo dpkg -i cvmfs-release-latest_all.deb rm -f cvmfs-release-latest_all.deb sudo apt update && sudo apt upgrade -y https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-libs_2.13.3+ubuntu22.04_amd64.deb https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-fuse3_2.13.3+ubuntu22.04_amd64.deb https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs_2.13.3+ubuntu22.04_amd64.deb https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.13.3/cvmfs-dev_2.13.3+ubuntu22.04_amd64.deb sudo apt-get install autofs -y ; sudo systemctl enable autofs ; sudo systemctl status autofs If CVMFS is installed for the very first time run once after the packets are installed: sudo cvmfs_config setup To be used without a local proxy: sudo xed /etc/cvmfs/default.local CVMFS_HTTP_PROXY="auto;DIRECT" # set quota limit between 4000 and 10000 CVMFS_QUOTA_LIMIT=6000 # to be used for ATLAS native and Theory native CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice" # enable Cloudflare's CDN # a MUST if you don't use a local proxy CVMFS_USE_CDN=yes # remove CVMFS_SERVER_URL from this file since it will be set automatically sudo xed /etc/cvmfs/domain.d/cern.ch.local # usually this is the only option that should be set in this file CVMFS_CONFIG_REPO_REQUIRED=yes sudo xed /etc/cvmfs/config.d/cvmfs-config.cern.ch.local # usually this is the only option that should be set in this file CVMFS_CONFIG_REPO_REQUIRED=no cvmfs_config probe https://cernvm-forum.cern.ch/t/warning-for-cvmrepo-during-sudo-apt-get-update/362/2 Warning for cvmrepo during sudo apt-get update As a workaround, could you try changing your /etc/apt/sources.list.d/cernvm.list file to read: sudo xed /etc/apt/sources.list.d/cernvm.list deb [arch=amd64] http://cvmrepo.s3.cern.ch/cvmrepo/apt jammy-prod main # deb http://cvmrepo.s3.cern.ch/cvmrepo/apt jammy-testing main sudo reboot To check BIOS setting from cli: sudo dmidecode | more Check client_state.xml that <p_vm_extensions_disabled>0</p_vm_extensions_disabled> is zero and not one. |
|
Send message Joined: 15 Jun 08 Posts: 2710 Credit: 292,058,192 RAC: 145,516 |
Looks like you mix a few things: 1. ATLAS native is currently the only app here that requires a local CVMFS client. None of the other apps require it. All the stuff related to CVMFS is not required for CMS. 2.1 To run vbox apps your computer needs a CPU that supports hardware virtualization AND it must be enabled in the BIOS. 2.2 VirtualBox must be installed 2.3 The user running BOINC must be a member of the group vboxusers 2.4 <p_vm_extensions_disabled> in client_state.xml must not be set to 1 Also good to know: If you set your prefs to allow ATLAS native and CMS/Theory on a computer you will also get ATLAS vbox work. |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
How can I strip all the superfluous stuff? I'm trying it on a computer where I only installed VirtualBox. It says it's not installed: Computer: Local machine 23262 LHC@home Nov 26, 2025, 01:52:59 PM Message from server: VirtualBox is not installedHow can I get VB recognized? TIA |
|
Send message Joined: 15 Jun 08 Posts: 2710 Credit: 292,058,192 RAC: 145,516 |
You got VirtualBox working here: https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10970631 BTW VirtualBox 6.1.50 is not wrong, but consider to upgrade to 7.2.2 which is more recent and does not have the issues 7.2.4 has. If your Linux vendor does not provide it you can download it here: https://www.virtualbox.org/wiki/Download_Old_Builds |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
I installed VB 7.2.2 on two Linux Mint 21.3 computers and launched it. Was warned there was no folder /home/aurum/VirtualBox VMs so I created it and the warning went away. I joined the vboxusers group and confirmed it by opening Users and Groups. Rebooted and still LHC does not see VB. I found this and tried it but LHC still doesn't see VB. BOINC complains that VirtualBox is not installed, what to do? https://lhcathome.web.cern.ch/boinc-complains-virtualbox-not-installed-what-do This command boinc -dir ${HOME}/boinc does not complete and return my command prompt. Been stuck here for several hours: 27-Nov-2025 05:02:46 [---] Running CPU benchmarks 27-Nov-2025 05:02:46 [---] Suspending computation - CPU benchmarks in progress 27-Nov-2025 05:03:17 [---] Benchmark results: 27-Nov-2025 05:03:17 [---] Number of CPUs: 36 27-Nov-2025 05:03:17 [---] 2905 floating point MIPS (Whetstone) per CPU 27-Nov-2025 05:03:17 [---] 5688 integer MIPS (Dhrystone) per CPU 27-Nov-2025 05:03:18 [---] Suspending GPU computation - computer is in use 27-Nov-2025 05:05:42 [---] Resuming GPU computationOn the second computer I tried closing the terminal window and rebooting. BOINC works but LHC still doesn't see my VB. This makes a copy of a number of files from my Elevated Privileges BOINC folder including my password file gui_rpc_auth.cfg with a different password. The stdoutgpudetect.txt copy says "27-Nov-2025 04:58:55 [---] cc_config.xml not found - using defaults" but I do have a cc_config.xml still in my real BOINC folder. I can't see anything in these files that solves this problem and won't be trying that again. Hopefully I can delete this useless folder without breaking BOINC. As for Rig-05 I don't know how LHC sees my VB but BOINC 8.2.8 is broken and dev says I must've left something on it from a previous install. I'm going to wipe its drive and do a fresh LM 21.3 build. Surely some Linux Mint user has been able to run CMS WUs. |
|
Send message Joined: 15 Jun 08 Posts: 2710 Credit: 292,058,192 RAC: 145,516 |
BOINC can either be started 1. as a user app 2. as a service (recommended under Linux) In case of (1.) it runs under the user's original account (in your case may be 'aurum'). In case of (2.) it runs under the username 'boinc' (which is created by the installer). If you want to run vbox VMs via BOINC install VirtualBox and verify the user running BOINC is a member of the group 'vboxusers'. In case of (1.) that would be 'aurum'. In case of (2.) that would be 'boinc'. To verify the group membership run: id -Gn boinc |grep vboxusers # optional, if 'id' is not installed grep vboxusers /etc/group If this is missing, add 'boinc' to 'vboxusers': sudo useradd -G vboxusers boinc => at least boinc service must be restarted once after this => after a fresh VirtualBox installation a reboot is recommended You can verify if VirtualBox is running (post the output here): lsmod |grep vbox VBoxManage --version VBoxManage list systemproperties VBoxManage list hostinfo If boinc is running as a service do NOT start it again e.g. via 'boinc -dir ${HOME}/boinc'. Instead, use this to check it: systemctl status boinc-client |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
Here's a real mystery. I rebuilt Rig-05. I installed VB 7.2.2 and joined vboxusers before installing BOINC 8.2.8. Started off that LHC recognized VB and sent 5 CMS WUs that completed successfully: https://lhcathome.cern.ch/lhcathome/results.php?hostid=10982337 While they were running the VirtualBox in not installed appeared again and Rig-05 details no longer showed virtualization details. Appears the problem lies with the LHC server.
|
|
Send message Joined: 15 Jun 08 Posts: 2710 Credit: 292,058,192 RAC: 145,516 |
This is typical for multiple local BOINC instances using the same host ID. One of them is aware of VirtualBox and gets vbox tasks. At least 1 other instance is not aware and reports VirtualBox is not installed. Things like that usually happen when you copy BOINC's data directory to various locations and run 1 instance as service and another instance from the command line. |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
This is typical for multiple local BOINC instances using the same host ID.I've never done that. The problem lies entirely with the LHC server. Will anyone attempt to fix it? Besides every computer I have now has VirtualBox and has a different name. Exactly where do I find what you call a "host ID"? My computers all have a different "Computer ID" and "Name". https://lhcathome.cern.ch/lhcathome/hosts_user.php?sort=id&rev=0&show_all=0&userid=571928 I explained clearly how I rebuilt Rig-05 from scratch and it did not involve copying anything from another computer on my LAN. I'll now detach every computer but Rig-05 from LHC and see if it revives. |
|
Send message Joined: 15 Jun 08 Posts: 2710 Credit: 292,058,192 RAC: 145,516 |
The problem lies entirely with the LHC server. Do you have any evidence, e.g. a log or anything like that? If yes, then present it. If not, this is nothing but a weak claim. Will anyone attempt to fix it? No, unless you present any evidence. At the main BOINC forum you claimed BOINC 8.2.8 is broken although you did a 'clean install'. https://boinc.berkeley.edu//forum_thread.php?id=15633&postid=117554 The log snippets you posted there clearly show that you also installed an outdated script from here that must not be used any more on recent Linux systems. That script was never part of the original BOINC installer but you seem to have ignored that information. I explained clearly how I rebuilt Rig-05 from scratch No, you didn't. Instead, you missed important steps, e.g. how the outdated script made it to your computer. It requires root credentials to install the script, so please don't blame the BOINC client or the project server as both don't run as root on your computer. |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
Is anyone successfully running CMS WUs on Linux Mint 21.3 Virginia Ubuntu 22.04 jammy??? If so how did you do it? |
|
Send message Joined: 12 Jun 18 Posts: 134 Credit: 55,928,848 RAC: 2,655 |
I explained clearly how I rebuilt Rig-05 from scratch. I don't know how it got there. Probably when LM is installed from an ico it has 4 options and I picked the wrong leaving old programs there. This time i picked erase drive and install LM 21.3 as stated below. Then I installed Virtualbox 7.2.2 as you recommended. Then I joined vboxusers. Then LHC recognized virtualization and DLed 5 WUs that completed successfully. Before they completed the CMS server claimed I did not have VB installed and sent no further units. Then you made the specious claim that I copied something to Rig-05. I have never installed anything by coping it from another computer on my LAN. I install from repositories. I detached all computers but Rig-05 from LHC so your CMS server can't get confused but it still does. Tell me what this log is and I'll post it. From my vantage CMS is incapable of recognizing properly configured Linux Mint computers with Virtualbox 7.2.2 installed and both boinc and aurum are members of the vboxusers group. BTW, I turned off Rig-05 and am now attempting to use Rig-10 to run CMS WUs but the CMS server is incapable of recognizing that has Virtualbox 7.2.2 installed.
|
©2025 CERN