Posts by skydivingnerd

21) Questions and Answers : Windows : Windows vbox64 CMS Simulation tasks failing - VM unable to validate X509 credential from LHC@home (Message 44940) Posted 12 May 2021 by skydivingnerd Post: I have a Win10 machine with BOINC client 7.16.11 and Vbox 6.1.22 installed. All the CMS Simulation tasks on my host are failing when the VM attemtps to validate the x509 certificate with LHC@home. I installed the CERN Root and Grid CA certificates, https://cafiles.cern.ch/cafiles/, on my local host, seeing if that corrected the issue of validation. It did not. Failed jobs examples: https://lhcathome.cern.ch/lhcathome/result.php?resultid=316190883 https://lhcathome.cern.ch/lhcathome/result.php?resultid=316187982 https://lhcathome.cern.ch/lhcathome/result.php?resultid=316180651 I've verified the local windows FW as well as my pfSense FW, including Snort, is passing traffic as it should. I ran a packet capture while the VM was attempting to reach out for the validation and see that the VM is communicating with LHC servers (vccs.cern.ch @ 137.138.120.99). The VM does not recognize the CERN server side CA. The stream exits with a TLSv1.2 Fatal error: Unknown CA The relevant packet is #10 No. Time Source Destination Protocol Length Info 1 0.000000 192.168.150.30 137.138.120.99 TCP 66 55514 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1 Frame 1: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 0, Len: 0 No. Time Source Destination Protocol Length Info 2 0.108285 137.138.120.99 192.168.150.30 TCP 66 443 → 55514 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128 Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 0, Ack: 1, Len: 0 No. Time Source Destination Protocol Length Info 3 0.108513 192.168.150.30 137.138.120.99 TCP 60 55514 → 443 [ACK] Seq=1 Ack=1 Win=262656 Len=0 Frame 3: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 1, Ack: 1, Len: 0 No. Time Source Destination Protocol Length Info 4 0.186955 192.168.150.30 137.138.120.99 TLSv1.2 224 Client Hello Frame 4: 224 bytes on wire (1792 bits), 224 bytes captured (1792 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 1, Ack: 1, Len: 170 Secure Sockets Layer No. Time Source Destination Protocol Length Info 5 0.297779 137.138.120.99 192.168.150.30 TCP 54 443 → 55514 [ACK] Seq=1 Ack=171 Win=30336 Len=0 Frame 5: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 1, Ack: 171, Len: 0 No. Time Source Destination Protocol Length Info 6 0.306888 137.138.120.99 192.168.150.30 TLSv1.2 1514 Server Hello Frame 6: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 1, Ack: 171, Len: 1460 Secure Sockets Layer No. Time Source Destination Protocol Length Info 7 0.306897 137.138.120.99 192.168.150.30 TLSv1.2 1514 Certificate [TCP segment of a reassembled PDU] Frame 7: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 1461, Ack: 171, Len: 1460 [2 Reassembled TCP Segments (2315 bytes): #6(1366), #7(949)] Secure Sockets Layer No. Time Source Destination Protocol Length Info 8 0.306905 137.138.120.99 192.168.150.30 TLSv1.2 146 Server Key Exchange, Server Hello Done Frame 8: 146 bytes on wire (1168 bits), 146 bytes captured (1168 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 2921, Ack: 171, Len: 92 [2 Reassembled TCP Segments (594 bytes): #7(511), #8(83)] Secure Sockets Layer Secure Sockets Layer No. Time Source Destination Protocol Length Info 9 0.307078 192.168.150.30 137.138.120.99 TCP 60 55514 → 443 [ACK] Seq=171 Ack=3013 Win=262656 Len=0 Frame 9: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 171, Ack: 3013, Len: 0 No. Time Source Destination Protocol Length Info 10 0.308588 192.168.150.30 137.138.120.99 TLSv1.2 61 Alert (Level: Fatal, Description: Unknown CA) Frame 10: 61 bytes on wire (488 bits), 61 bytes captured (488 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 171, Ack: 3013, Len: 7 Secure Sockets Layer No. Time Source Destination Protocol Length Info 11 0.308688 192.168.150.30 137.138.120.99 TCP 60 55514 → 443 [FIN, ACK] Seq=178 Ack=3013 Win=262656 Len=0 Frame 11: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 178, Ack: 3013, Len: 0 No. Time Source Destination Protocol Length Info 12 0.418915 137.138.120.99 192.168.150.30 TCP 54 443 → 55514 [FIN, ACK] Seq=3013 Ack=179 Win=30336 Len=0 Frame 12: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) Ethernet II, Src: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10), Dst: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09) Internet Protocol Version 4, Src: 137.138.120.99, Dst: 192.168.150.30 Transmission Control Protocol, Src Port: 443, Dst Port: 55514, Seq: 3013, Ack: 179, Len: 0 No. Time Source Destination Protocol Length Info 13 0.419178 192.168.150.30 137.138.120.99 TCP 60 55514 → 443 [ACK] Seq=179 Ack=3014 Win=262656 Len=0 Frame 13: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) Ethernet II, Src: AsustekC_ee:47:09 (3c:7c:3f:ee:47:09), Dst: IntelCor_6b:d4:10 (00:1b:21:6b:d4:10) Internet Protocol Version 4, Src: 192.168.150.30, Dst: 137.138.120.99 Transmission Control Protocol, Src Port: 55514, Dst Port: 443, Seq: 179, Ack: 3014, Len: 0 I believe this is an issue with the VM itself not having the correct host certificate. Can an admin check into this? R/S Scott
22) Message boards : CMS Application : Could not get X509 credentials (Message 44913) Posted 8 May 2021 by skydivingnerd Post: I removed my one Windows 10 host from getting CMS Sim (vbox64) workunits for a while, thinking that the issue is transient. The errors are back again in new workunits. Below are a few of them. https://lhcathome.cern.ch/lhcathome/result.php?resultid=316190883 https://lhcathome.cern.ch/lhcathome/result.php?resultid=316187982 https://lhcathome.cern.ch/lhcathome/result.php?resultid=316180651 This host is completing some of the CMS Simulation workunits, but the majority are failing from the certificate issue.
23) Message boards : CMS Application : Could not get X509 credentials (Message 44770) Posted 20 Apr 2021 by skydivingnerd Post: I have a lot of CMS task failures from my windows box. Looks like the VM has issues getting X509 certs from LHC. https://lhcathome.cern.ch/lhcathome/result.php?resultid=313565843 <core_client_version>7.16.11</core_client_version> <![CDATA[ <message> The filename or extension is too long. (0xce) - exit code 206 (0xce)</message> <stderr_txt> 2021-04-20 03:45:06 (1464): Detected: vboxwrapper 26197 2021-04-20 03:45:06 (1464): Detected: BOINC client v7.7 2021-04-20 03:45:06 (1464): Detected: VirtualBox VboxManage Interface (Version: 6.1.16) 2021-04-20 03:45:06 (1464): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds) 2021-04-20 03:45:06 (1464): Successfully copied 'init_data.xml' to the shared directory. 2021-04-20 03:45:08 (1464): Create VM. (boinc_80a422322ddcce90, slot#11) 2021-04-20 03:45:08 (1464): Setting Memory Size for VM. (2048MB) 2021-04-20 03:45:09 (1464): Setting CPU Count for VM. (1) 2021-04-20 03:45:09 (1464): Setting Chipset Options for VM. 2021-04-20 03:45:09 (1464): Setting Boot Options for VM. 2021-04-20 03:45:09 (1464): Setting Network Configuration for NAT. 2021-04-20 03:45:10 (1464): Enabling VM Network Access. 2021-04-20 03:45:10 (1464): Disabling USB Support for VM. 2021-04-20 03:45:10 (1464): Disabling COM Port Support for VM. 2021-04-20 03:45:10 (1464): Disabling LPT Port Support for VM. 2021-04-20 03:45:11 (1464): Disabling Audio Support for VM. 2021-04-20 03:45:11 (1464): Disabling Clipboard Support for VM. 2021-04-20 03:45:11 (1464): Disabling Drag and Drop Support for VM. 2021-04-20 03:45:12 (1464): Adding storage controller(s) to VM. 2021-04-20 03:45:12 (1464): Adding virtual disk drive to VM. (vm_image.vdi) 2021-04-20 03:45:12 (1464): Adding VirtualBox Guest Additions to VM. 2021-04-20 03:45:12 (1464): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB) 2021-04-20 03:45:13 (1464): forwarding host port 58261 to guest port 80 2021-04-20 03:45:13 (1464): Enabling remote desktop for VM. 2021-04-20 03:45:13 (1464): Enabling shared directory for VM. 2021-04-20 03:45:14 (1464): Starting VM using VBoxManage interface. (boinc_80a422322ddcce90, slot#11) 2021-04-20 03:45:18 (1464): Successfully started VM. (PID = '7968') 2021-04-20 03:45:18 (1464): Reporting VM Process ID to BOINC. 2021-04-20 03:45:18 (1464): Guest Log: BIOS: VirtualBox 6.1.16 2021-04-20 03:45:18 (1464): Guest Log: CPUID EDX: 0x178bfbff 2021-04-20 03:45:18 (1464): Guest Log: BIOS: ata0-0: PCHS=16383/16/63 LCHS=1024/255/63 2021-04-20 03:45:18 (1464): VM state change detected. (old = 'PoweredOff', new = 'Running') 2021-04-20 03:45:18 (1464): Detected: Web Application Enabled (http://localhost:58261) 2021-04-20 03:45:18 (1464): Detected: Remote Desktop Enabled (localhost:58262) 2021-04-20 03:45:18 (1464): Preference change detected 2021-04-20 03:45:18 (1464): Setting CPU throttle for VM. (100%) 2021-04-20 03:45:18 (1464): Setting checkpoint interval to 600 seconds. (Higher value of (Preference: 600 seconds) or (Vbox_job.xml: 600 seconds)) 2021-04-20 03:45:20 (1464): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032 2021-04-20 03:45:20 (1464): Guest Log: BIOS: Booting from Hard Disk... 2021-04-20 03:45:22 (1464): Guest Log: BIOS: KBD: unsupported int 16h function 03 2021-04-20 03:45:22 (1464): Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000 2021-04-20 03:45:36 (1464): Guest Log: vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds 2021-04-20 03:45:36 (1464): Guest Log: vboxguest: misc device minor 56, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000) 2021-04-20 03:45:56 (1464): Guest Log: VBoxService 5.2.6 r120293 (verbosity: 0) linux.amd64 (Jan 15 2018 14:51:00) release log 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000074 main Log opened 2021-04-20T07:45:56.310756000Z 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000244 main OS Product: Linux 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000275 main OS Release: 4.14.157-17.cernvm.x86_64 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000297 main OS Version: #1 SMP Wed Dec 4 17:26:45 CET 2019 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000319 main Executable: /usr/share/vboxguest52/usr/sbin/VBoxService 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000319 main Process ID: 2976 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000320 main Package type: LINUX_64BITS_GENERIC 2021-04-20 03:45:56 (1464): Guest Log: 00:00:00.000886 main 5.2.6 r120293 started. Verbose level = 0 2021-04-20 03:47:51 (1464): Guest Log: [INFO] Mounting the shared directory 2021-04-20 03:47:51 (1464): Guest Log: [INFO] Shared directory mounted, enabling vboxmonitor 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing network connection to cern.ch on port 80 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to cern.ch 80 port [tcp/http] succeeded! 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] 0 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing VCCS connection to vccs.cern.ch on port 443 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to vccs.cern.ch 443 port [tcp/https] succeeded! 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] 0 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Testing connection to Condor server on port 9618 2021-04-20 03:47:51 (1464): Guest Log: [DEBUG] Connection to vocms0840.cern.ch 9618 port [tcp/condor] succeeded! 2021-04-20 03:47:52 (1464): Guest Log: [DEBUG] 0 2021-04-20 03:50:39 (1464): Guest Log: [DEBUG] Probing CVMFS ... 2021-04-20 03:50:40 (1464): Guest Log: Probing /cvmfs/grid.cern.ch... OK 2021-04-20 03:50:45 (1464): Guest Log: VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE 2021-04-20 03:50:45 (1464): Guest Log: 2.4.4.0 3760 4 27856 11625 2 1 1234377 4096000 2 65024 0 3 100 0 0 http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/grid.cern.ch http://128.142.161.84:3126 0 2021-04-20 03:54:16 (1464): Guest Log: [INFO] Reading volunteer information 2021-04-20 03:54:16 (1464): Guest Log: [INFO] Volunteer: scotth (787857) 2021-04-20 03:54:16 (1464): Guest Log: [INFO] VMID: 49b2fac1-df25-48d2-a4ee-4612ca6a31f8 2021-04-20 03:54:16 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:54:16 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:54:47 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:54:47 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:55:17 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:55:18 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:55:48 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:55:48 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:56:19 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:56:19 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:56:49 (1464): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2021-04-20 03:56:49 (1464): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev 2021-04-20 03:57:20 (1464): Guest Log: [DEBUG] 2021-04-20 03:57:20 (1464): Guest Log: curl: (60) Peer certificate cannot be authenticated with known CA certificates 2021-04-20 03:57:20 (1464): Guest Log: More details here: http://curl.haxx.se/docs/sslcerts.html 2021-04-20 03:57:20 (1464): Guest Log: curl performs SSL certificate verification by default, using a "bundle" 2021-04-20 03:57:20 (1464): Guest Log: of Certificate Authority (CA) public keys (CA certs). If the default 2021-04-20 03:57:20 (1464): Guest Log: bundle file isn't adequate, you can specify an alternate file 2021-04-20 03:57:20 (1464): Guest Log: using the --cacert option. 2021-04-20 03:57:20 (1464): Guest Log: If this HTTPS server uses a certificate signed by a CA represented in 2021-04-20 03:57:20 (1464): Guest Log: the bundle, the certificate verification probably failed due to a 2021-04-20 03:57:20 (1464): Guest Log: problem with the certificate (it might be expired, or the name might 2021-04-20 03:57:20 (1464): Guest Log: not match the domain name in the URL). 2021-04-20 03:57:20 (1464): Guest Log: If you'd like to turn off curl's verification of the certificate, use 2021-04-20 03:57:20 (1464): Guest Log: the -k (or --insecure) option. 2021-04-20 03:57:20 (1464): Guest Log: [DEBUG] 2021-04-20 03:57:20 (1464): Guest Log: ERROR: Couldn't find a valid proxy. 2021-04-20 03:57:20 (1464): Guest Log: globus_sysconfig: File has zero length: File: /tmp/x509up_u0 2021-04-20 03:57:20 (1464): Guest Log: Use -debug for further information. 2021-04-20 03:57:20 (1464): Guest Log: [ERROR] Could not get an x509 credential 2021-04-20 03:57:20 (1464): Guest Log: [ERROR] The x509 proxy creation failed. 2021-04-20 03:57:20 (1464): Guest Log: [INFO] Shutting Down. 2021-04-20 03:57:20 (1464): VM Completion File Detected. 2021-04-20 03:57:20 (1464): VM Completion Message: The x509 proxy creation failed. . 2021-04-20 03:57:20 (1464): Powering off VM. 2021-04-20 04:02:21 (1464): VM did not power off when requested. 2021-04-20 04:02:21 (1464): VM was successfully terminated. 2021-04-20 04:02:21 (1464): Deregistering VM. (boinc_80a422322ddcce90, slot#11) 2021-04-20 04:02:21 (1464): Removing network bandwidth throttle group from VM. 2021-04-20 04:02:21 (1464): Removing VM from VirtualBox. 04:02:27 (1464): called boinc_finish(206) </stderr_txt> ]]>

Previous 20

LHC@home