LHC@home

There seems to be a credential problem, like Theroy had a while back.

e.g.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=307695455
https://lhcathome.cern.ch/lhcathome/result.php?resultid=305661561

Agreed. Every CMS task I get is failing. (That said, it's stating that those tasks don't exist when I navigate to those URLs you provided). I might just disable CMS for now so I don't junk up the pipeline with errored-out tasks.

I have a lot of CMS task failures from my windows box. Looks like the VM has issues getting X509 certs from LHC.
https://lhcathome.cern.ch/lhcathome/result.php?resultid=313565843

<core_client_version>7.16.11</core_client_version>
<![CDATA[
<message>
The filename or extension is too long.
(0xce) - exit code 206 (0xce)</message>
<stderr_txt>
2021-04-20 03:45:06 (1464): Detected: vboxwrapper 26197
2021-04-20 03:45:06 (1464): Detected: BOINC client v7.7
2021-04-20 03:45:06 (1464): Detected: VirtualBox VboxManage Interface (Version: 6.1.16)
2021-04-20 03:45:06 (1464): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds)
2021-04-20 03:45:06 (1464): Successfully copied 'init_data.xml' to the shared directory.
2021-04-20 03:45:08 (1464): Create VM. (boinc_80a422322ddcce90, slot#11)
2021-04-20 03:45:08 (1464): Setting Memory Size for VM. (2048MB)
2021-04-20 03:45:09 (1464): Setting CPU Count for VM. (1)
2021-04-20 03:45:09 (1464): Setting Chipset Options for VM.
2021-04-20 03:45:09 (1464): Setting Boot Options for VM.
2021-04-20 03:45:09 (1464): Setting Network Configuration for NAT.
2021-04-20 03:45:10 (1464): Enabling VM Network Access.
2021-04-20 03:45:10 (1464): Disabling USB Support for VM.
2021-04-20 03:45:10 (1464): Disabling COM Port Support for VM.
2021-04-20 03:45:10 (1464): Disabling LPT Port Support for VM.
2021-04-20 03:45:11 (1464): Disabling Audio Support for VM.
2021-04-20 03:45:11 (1464): Disabling Clipboard Support for VM.
2021-04-20 03:45:11 (1464): Disabling Drag and Drop Support for VM.
2021-04-20 03:45:12 (1464): Adding storage controller(s) to VM.
2021-04-20 03:45:12 (1464): Adding virtual disk drive to VM. (vm_image.vdi)
2021-04-20 03:45:12 (1464): Adding VirtualBox Guest Additions to VM.
2021-04-20 03:45:12 (1464): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB)
2021-04-20 03:45:13 (1464): forwarding host port 58261 to guest port 80
2021-04-20 03:45:13 (1464): Enabling remote desktop for VM.
2021-04-20 03:45:13 (1464): Enabling shared directory for VM.
2021-04-20 03:45:14 (1464): Starting VM using VBoxManage interface. (boinc_80a422322ddcce90, slot#11)
2021-04-20 03:45:18 (1464): Successfully started VM. (PID = '7968')
2021-04-20 03:45:18 (1464): Reporting VM Process ID to BOINC.
2021-04-20 03:45:18 (1464): Guest Log: BIOS: VirtualBox 6.1.16

2021-04-20 03:45:18 (1464): Guest Log: CPUID EDX: 0x178bfbff

2021-04-20 03:45:18 (1464): Guest Log: BIOS: ata0-0: PCHS=16383/16/63 LCHS=1024/255/63

2021-04-20 03:45:18 (1464): VM state change detected. (old = 'PoweredOff', new = 'Running')
2021-04-20 03:45:18 (1464): Detected: Web Application Enabled (http://localhost:58261)
2021-04-20 03:45:18 (1464): Detected: Remote Desktop Enabled (localhost:58262)
2021-04-20 03:45:18 (1464): Preference change detected
2021-04-20 03:45:18 (1464): Setting CPU throttle for VM. (100%)
2021-04-20 03:45:18 (1464): Setting checkpoint interval to 600 seconds. (Higher value of (Preference: 600 seconds) or (Vbox_job.xml: 600 seconds))
2021-04-20 03:45:20 (1464): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032
2021-04-20 03:45:20 (1464): Guest Log: BIOS: Booting from Hard Disk...
2021-04-20 03:45:22 (1464): Guest Log: BIOS: KBD: unsupported int 16h function 03
2021-04-20 03:45:22 (1464): Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000
2021-04-20 03:45:36 (1464): Guest Log: vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds
2021-04-20 03:45:36 (1464): Guest Log: vboxguest: misc device minor 56, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
2021-04-20 03:45:56 (1464): Guest Log: VBoxService 5.2.6 r120293 (verbosity: 0) linux.amd64 (Jan 15 2018 14:51:00) release log
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000074 main Log opened 2021-04-20T07:45:56.310756000Z
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000244 main OS Product: Linux
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000275 main OS Release: 4.14.157-17.cernvm.x86_64
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000297 main OS Version: #1 SMP Wed Dec 4 17:26:45 CET 2019
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000319 main Executable: /usr/share/vboxguest52/usr/sbin/VBoxService
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000319 main Process ID: 2976
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000320 main Package type: LINUX_64BITS_GENERIC
2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000886 main 5.2.6 r120293 started. Verbose level = 0
2021-04-20 03:47:51 (1464): Guest Log: [INFO] Mounting the shared directory
2021-04-20 03:47:51 (1464): Guest Log: [INFO] Shared directory mounted, enabling vboxmonitor
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing network connection to cern.ch on port 80
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to cern.ch 80 port [tcp/http] succeeded!
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] 0
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing VCCS connection to vccs.cern.ch on port 443
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to vccs.cern.ch 443 port [tcp/https] succeeded!
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] 0
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing connection to Condor server on port 9618
2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to vocms0840.cern.ch 9618 port [tcp/condor] succeeded!
2021-04-20 03:47:52 (1464): Guest Log: [DEBUG] 0
2021-04-20 03:50:39 (1464): Guest Log: [DEBUG] Probing CVMFS ...
2021-04-20 03:50:40 (1464): Guest Log: Probing /cvmfs/grid.cern.ch... OK
2021-04-20 03:50:45 (1464): Guest Log: VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2021-04-20 03:50:45 (1464): Guest Log: 2.4.4.0 3760 4 27856 11625 2 1 1234377 4096000 2 65024 0 3 100 0 0 http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/grid.cern.ch http://128.142.161.84:3126 0
2021-04-20 03:54:16 (1464): Guest Log: [INFO] Reading volunteer information
2021-04-20 03:54:16 (1464): Guest Log: [INFO] Volunteer: scotth (787857)
2021-04-20 03:54:16 (1464): Guest Log: [INFO] VMID: 49b2fac1-df25-48d2-a4ee-4612ca6a31f8
2021-04-20 03:54:16 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:54:16 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:54:47 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:54:47 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:55:17 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:55:18 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:55:48 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:55:48 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:56:19 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:56:19 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:56:49 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2021-04-20 03:56:49 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2021-04-20 03:57:20 (1464): Guest Log: [DEBUG]
2021-04-20 03:57:20 (1464): Guest Log: curl: (60) Peer certificate cannot be authenticated with known CA certificates
2021-04-20 03:57:20 (1464): Guest Log: More details here: http://curl.haxx.se/docs/sslcerts.html
2021-04-20 03:57:20 (1464): Guest Log: curl performs SSL certificate verification by default, using a "bundle"
2021-04-20 03:57:20 (1464): Guest Log: of Certificate Authority (CA) public keys (CA certs). If the default
2021-04-20 03:57:20 (1464): Guest Log: bundle file isn't adequate, you can specify an alternate file
2021-04-20 03:57:20 (1464): Guest Log: using the --cacert option.
2021-04-20 03:57:20 (1464): Guest Log: If this HTTPS server uses a certificate signed by a CA represented in
2021-04-20 03:57:20 (1464): Guest Log: the bundle, the certificate verification probably failed due to a
2021-04-20 03:57:20 (1464): Guest Log: problem with the certificate (it might be expired, or the name might
2021-04-20 03:57:20 (1464): Guest Log: not match the domain name in the URL).
2021-04-20 03:57:20 (1464): Guest Log: If you'd like to turn off curl's verification of the certificate, use
2021-04-20 03:57:20 (1464): Guest Log: the -k (or --insecure) option.
2021-04-20 03:57:20 (1464): Guest Log: [DEBUG]
2021-04-20 03:57:20 (1464): Guest Log: ERROR: Couldn't find a valid proxy.
2021-04-20 03:57:20 (1464): Guest Log: globus_sysconfig: File has zero length: File: /tmp/x509up_u0
2021-04-20 03:57:20 (1464): Guest Log: Use -debug for further information.
2021-04-20 03:57:20 (1464): Guest Log: [ERROR] Could not get an x509 credential
2021-04-20 03:57:20 (1464): Guest Log: [ERROR] The x509 proxy creation failed.
2021-04-20 03:57:20 (1464): Guest Log: [INFO] Shutting Down.
2021-04-20 03:57:20 (1464): VM Completion File Detected.
2021-04-20 03:57:20 (1464): VM Completion Message: The x509 proxy creation failed.
.
2021-04-20 03:57:20 (1464): Powering off VM.
2021-04-20 04:02:21 (1464): VM did not power off when requested.
2021-04-20 04:02:21 (1464): VM was successfully terminated.
2021-04-20 04:02:21 (1464): Deregistering VM. (boinc_80a422322ddcce90, slot#11)
2021-04-20 04:02:21 (1464): Removing network bandwidth throttle group from VM.
2021-04-20 04:02:21 (1464): Removing VM from VirtualBox.
04:02:27 (1464): called boinc_finish(206)

</stderr_txt>
]]>

---

I didn't get anymore for a while so I think it's some transient error

I removed my one Windows 10 host from getting CMS Sim (vbox64) workunits for a while, thinking that the issue is transient. The errors are back again in new workunits. Below are a few of them.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=316190883
https://lhcathome.cern.ch/lhcathome/result.php?resultid=316187982
https://lhcathome.cern.ch/lhcathome/result.php?resultid=316180651

This host is completing some of the CMS Simulation workunits, but the majority are failing from the certificate issue.

---