LHC@home

For both 32bit and 64bit Theory VM's, I can't get X509 credentials, so all tasks error out.

Seems to be a major problem across more than one project.
I have the exact same problem with LHCb tasks!

Regards, djoser.

---

Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us

So this is the issue that gets 'unable to mount root volume' and tosses the VM into diagnostic console across all my tasks since last night?

Posted a diagnostic message in the number crunching thread.

The problem should now be fixed.

Still getting unable to mount root device errors.

Do I need to abort all the WU's from this morning and fresh d/l?

The problem should now be fixed.

For Win64 it is fixed, but on my Win10 32-bit the x509 proxy creation is still failing after testing with 2 tasks sequential.
The VM request 5 times the credentials from LHC@home and vLHC@home-dev.
That's the machine that got yesterday the new Theory32 vdi-file.

Is requesting from vLHC@home-dev the problem here?

The Win64 is requesting only from LHC@home.

https://lhcathome.cern.ch/lhcathome/results.php?hostid=10416365&offset=0&show_names=0&state=6&appid=13

In -dev is no Application shown for Theory(x86-32bit).

In -dev is no Application shown for Theory(x86-32bit).

Requesting credentials has nothing to do with what BOINC-applications are running on a server.

I find vLHC@home-dev suspect, cause there was a vLHCathome in the past, but no vLHCathome-dev.

I opened up the failing WU VM's on Saturday and saw some with a failed X509 cert error after a run of the diagnostics console.
The failed-to-mount-root WU's mentioned diagnostics console, but I didn't wait around for them to continue the diagnostics console (it didn't start within 10 minutes) and just powered down those VM's.

Mounting the root volume from the local VDI shouldn't in anyway depend on the X509 certificate, right? The problems with my WU's were a coincidence to the server side issues? If so, then this machine might need reconfiguration.

I opened up the failing WU VM's on Saturday and saw some with a failed X509 cert error after a run of the diagnostics console.

I had the same error shown for several CMS tasks during last Saturday night:

2017-10-07 23:42:59 (8056): Guest Log: [ERROR] Could not get an x509 credential
2017-10-07 23:42:59 (8056): Guest Log: [ERROR] The x509 proxy creation failed.
2017-10-07 23:42:59 (8056): Guest Log: [INFO] Shutting Down.

All these WUs got shut down after some 6 minutes.

Yesterday on Sunday this was fixed from Laurence! See other thread.

Got this on 3 old host added.

Exit status	206 (0x000000CE) EXIT_INIT_FAILURE

<message>
The filename or extension is too long.
 (0xce) - exit code 206 (0xce)</message>

But log show more why:

[019-06-23 00:33:02 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:33:03 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:33:34 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:33:35 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:34:06 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:34:06 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:34:37 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:34:38 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:35:09 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:35:10 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:35:41 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:35:42 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:36:14 (5080): Guest Log: [DEBUG] 

2019-06-23 00:36:14 (5080): Guest Log: curl: (60) Peer certificate cannot be authenticated with known CA certificates

2019-06-23 00:36:14 (5080): Guest Log: More details here: http://curl.haxx.se/docs/sslcerts.html

2019-06-23 00:36:14 (5080): Guest Log: curl performs SSL certificate verification by default, using a "bundle"

2019-06-23 00:36:14 (5080): Guest Log:  of Certificate Authority (CA) public keys (CA certs). If the default

2019-06-23 00:36:14 (5080): Guest Log:  bundle file isn't adequate, you can specify an alternate file

2019-06-23 00:36:14 (5080): Guest Log:  using the --cacert option.

2019-06-23 00:36:14 (5080): Guest Log: If this HTTPS server uses a certificate signed by a CA represented in

2019-06-23 00:36:14 (5080): Guest Log:  the bundle, the certificate verification probably failed due to a

2019-06-23 00:36:14 (5080): Guest Log:  problem with the certificate (it might be expired, or the name might

2019-06-23 00:36:14 (5080): Guest Log:  not match the domain name in the URL).

2019-06-23 00:36:14 (5080): Guest Log: If you'd like to turn off curl's verification of the certificate, use

2019-06-23 00:36:14 (5080): Guest Log:  the -k (or --insecure) option.

2019-06-23 00:36:14 (5080): Guest Log: [DEBUG] 

2019-06-23 00:36:14 (5080): Guest Log: ERROR: Couldn't find a valid proxy.

2019-06-23 00:36:14 (5080): Guest Log:        globus_sysconfig: File has zero length: File: /tmp/x509up_u0

2019-06-23 00:36:14 (5080): Guest Log: Use -debug for further information.

2019-06-23 00:36:15 (5080): Guest Log: [ERROR] Could not get an x509 credential

2019-06-23 00:36:15 (5080): Guest Log: [ERROR] The x509 proxy creation failed.

2019-06-23 00:36:15 (5080): Guest Log: [INFO] Shutting Down.

Not sure if cert renew in setup files or depended on what server hand out. I let host retry later but if not changed and it would use old certs more host would suffer from it.
These host (win 10 x64) have direct connection to server with no proxy/vpn attached.

Several of these.......on a weekend of course.

I had the same problem 2 days ago:

2019-06-22 12:57:04 (5416): Guest Log: [ERROR] Could not get an x509 credential

for more details, see here:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=233901549

here the next one:

https://lhcathome.cern.ch/lhcathome/result.php?resultid=234039580

what's happening there?

I had 4 more in a row, yesterday evening, between 22:29 hrs and 23:01 hrs UTC.

same thing, just a few minutes ago, here: https://lhcathome.cern.ch/lhcathome/result.php?resultid=234039580
and here: https://lhcathome.cern.ch/lhcathome/result.php?resultid=234513174

is there no-one who can get this problem fixed?

This does not seem to be a general problem. I have looked at a few things on the server. Please let me know if these problems are still occurring.

Please let me know if these problems are still occurring.

there was another one short time ago:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=234571121

Edit: Now I am seeing this problem on another of my machines, too. So my inital suspicion that something might be wrong with computer ID: 10555784 is definitely wrong.

Here the failed task from computer ID: 10450564: https://lhcathome.cern.ch/lhcathome/results.php?hostid=10450564

Please let me know if these problems are still occurring.

there was another one short time ago:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=234571121

Edit: Now I am seeing this problem on another of my machines, too. So my inital suspicion that something might be wrong with computer ID: 10555784 is definitely wrong.

Or there is something wrong with computer ID: 10555784 and it has now spread to the other machine.