Message boards : Theory Application : Could not get X509 credentials
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 750
Credit: 6,028,092
RAC: 700
Message 32670 - Posted: 7 Oct 2017, 8:36:46 UTC

For both 32bit and 64bit Theory VM's, I can't get X509 credentials, so all tasks error out.
ID: 32670 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 83
Credit: 7,154,202
RAC: 3,692
Message 32672 - Posted: 7 Oct 2017, 12:16:41 UTC

Seems to be a major problem across more than one project.
I have the exact same problem with LHCb tasks!

Regards, djoser.
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! www.gridcoin.us
ID: 32672 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 127
Credit: 6,204,427
RAC: 197
Message 32677 - Posted: 7 Oct 2017, 15:46:38 UTC - in response to Message 32670.  
Last modified: 7 Oct 2017, 15:46:54 UTC

So this is the issue that gets 'unable to mount root volume' and tosses the VM into diagnostic console across all my tasks since last night?

Posted a diagnostic message in the number crunching thread.
ID: 32677 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 303
Credit: 231,154
RAC: 329
Message 32680 - Posted: 7 Oct 2017, 21:13:01 UTC - in response to Message 32670.  

The problem should now be fixed.
ID: 32680 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 127
Credit: 6,204,427
RAC: 197
Message 32683 - Posted: 8 Oct 2017, 3:32:06 UTC - in response to Message 32680.  

Still getting unable to mount root device errors.

Do I need to abort all the WU's from this morning and fresh d/l?
ID: 32683 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 750
Credit: 6,028,092
RAC: 700
Message 32689 - Posted: 8 Oct 2017, 7:58:25 UTC - in response to Message 32680.  
Last modified: 8 Oct 2017, 7:59:49 UTC

The problem should now be fixed.

For Win64 it is fixed, but on my Win10 32-bit the x509 proxy creation is still failing after testing with 2 tasks sequential.
The VM request 5 times the credentials from LHC@home and vLHC@home-dev.
That's the machine that got yesterday the new Theory32 vdi-file.

Is requesting from vLHC@home-dev the problem here?

The Win64 is requesting only from LHC@home.

https://lhcathome.cern.ch/lhcathome/results.php?hostid=10416365&offset=0&show_names=0&state=6&appid=13
ID: 32689 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,571,125
RAC: 39,305
Message 32690 - Posted: 8 Oct 2017, 8:59:52 UTC

In -dev is no Application shown for Theory(x86-32bit).
ID: 32690 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 750
Credit: 6,028,092
RAC: 700
Message 32691 - Posted: 8 Oct 2017, 9:24:56 UTC - in response to Message 32690.  

In -dev is no Application shown for Theory(x86-32bit).

Requesting credentials has nothing to do with what BOINC-applications are running on a server.

I find vLHC@home-dev suspect, cause there was a vLHCathome in the past, but no vLHCathome-dev.
ID: 32691 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 127
Credit: 6,204,427
RAC: 197
Message 32707 - Posted: 9 Oct 2017, 5:19:57 UTC - in response to Message 32691.  

I opened up the failing WU VM's on Saturday and saw some with a failed X509 cert error after a run of the diagnostics console.
The failed-to-mount-root WU's mentioned diagnostics console, but I didn't wait around for them to continue the diagnostics console (it didn't start within 10 minutes) and just powered down those VM's.

Mounting the root volume from the local VDI shouldn't in anyway depend on the X509 certificate, right? The problems with my WU's were a coincidence to the server side issues? If so, then this machine might need reconfiguration.
ID: 32707 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 32712 - Posted: 9 Oct 2017, 9:20:32 UTC - in response to Message 32707.  

I opened up the failing WU VM's on Saturday and saw some with a failed X509 cert error after a run of the diagnostics console.

I had the same error shown for several CMS tasks during last Saturday night:

2017-10-07 23:42:59 (8056): Guest Log: [ERROR] Could not get an x509 credential
2017-10-07 23:42:59 (8056): Guest Log: [ERROR] The x509 proxy creation failed.
2017-10-07 23:42:59 (8056): Guest Log: [INFO] Shutting Down.

All these WUs got shut down after some 6 minutes.
ID: 32712 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,571,125
RAC: 39,305
Message 32713 - Posted: 9 Oct 2017, 9:29:33 UTC

Yesterday on Sunday this was fixed from Laurence! See other thread.
ID: 32713 · Report as offensive     Reply Quote
Gunde

Send message
Joined: 9 Jan 15
Posts: 37
Credit: 274,095,524
RAC: 432,294
Message 39169 - Posted: 22 Jun 2019, 23:41:18 UTC
Last modified: 22 Jun 2019, 23:48:22 UTC

Got this on 3 old host added.
Exit status	206 (0x000000CE) EXIT_INIT_FAILURE

<message>
The filename or extension is too long.
 (0xce) - exit code 206 (0xce)</message>


But log show more why:

[019-06-23 00:33:02 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:33:03 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:33:34 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:33:35 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:34:06 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:34:06 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:34:37 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:34:38 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:35:09 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:35:10 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:35:41 (5080): Guest Log: [INFO] Requesting an X509 credential from LHC@home

2019-06-23 00:35:42 (5080): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev

2019-06-23 00:36:14 (5080): Guest Log: [DEBUG] 

2019-06-23 00:36:14 (5080): Guest Log: curl: (60) Peer certificate cannot be authenticated with known CA certificates

2019-06-23 00:36:14 (5080): Guest Log: More details here: http://curl.haxx.se/docs/sslcerts.html

2019-06-23 00:36:14 (5080): Guest Log: curl performs SSL certificate verification by default, using a "bundle"

2019-06-23 00:36:14 (5080): Guest Log:  of Certificate Authority (CA) public keys (CA certs). If the default

2019-06-23 00:36:14 (5080): Guest Log:  bundle file isn't adequate, you can specify an alternate file

2019-06-23 00:36:14 (5080): Guest Log:  using the --cacert option.

2019-06-23 00:36:14 (5080): Guest Log: If this HTTPS server uses a certificate signed by a CA represented in

2019-06-23 00:36:14 (5080): Guest Log:  the bundle, the certificate verification probably failed due to a

2019-06-23 00:36:14 (5080): Guest Log:  problem with the certificate (it might be expired, or the name might

2019-06-23 00:36:14 (5080): Guest Log:  not match the domain name in the URL).

2019-06-23 00:36:14 (5080): Guest Log: If you'd like to turn off curl's verification of the certificate, use

2019-06-23 00:36:14 (5080): Guest Log:  the -k (or --insecure) option.

2019-06-23 00:36:14 (5080): Guest Log: [DEBUG] 

2019-06-23 00:36:14 (5080): Guest Log: ERROR: Couldn't find a valid proxy.

2019-06-23 00:36:14 (5080): Guest Log:        globus_sysconfig: File has zero length: File: /tmp/x509up_u0

2019-06-23 00:36:14 (5080): Guest Log: Use -debug for further information.

2019-06-23 00:36:15 (5080): Guest Log: [ERROR] Could not get an x509 credential

2019-06-23 00:36:15 (5080): Guest Log: [ERROR] The x509 proxy creation failed.

2019-06-23 00:36:15 (5080): Guest Log: [INFO] Shutting Down.


Not sure if cert renew in setup files or depended on what server hand out. I let host retry later but if not changed and it would use old certs more host would suffer from it.
These host (win 10 x64) have direct connection to server with no proxy/vpn attached.
ID: 39169 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 852
Credit: 37,602,121
RAC: 25,344
Message 39174 - Posted: 23 Jun 2019, 20:54:34 UTC
Last modified: 23 Jun 2019, 21:01:51 UTC



Several of these.......on a weekend of course.

ID: 39174 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 39175 - Posted: 24 Jun 2019, 5:30:09 UTC

I had the same problem 2 days ago:

2019-06-22 12:57:04 (5416): Guest Log: [ERROR] Could not get an x509 credential

for more details, see here:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=233901549
ID: 39175 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 39187 - Posted: 24 Jun 2019, 17:26:21 UTC

ID: 39187 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 39190 - Posted: 26 Jun 2019, 11:42:39 UTC - in response to Message 39187.  

I had 4 more in a row, yesterday evening, between 22:29 hrs and 23:01 hrs UTC.
ID: 39190 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 39197 - Posted: 26 Jun 2019, 19:45:11 UTC

same thing, just a few minutes ago, here: https://lhcathome.cern.ch/lhcathome/result.php?resultid=234039580
and here: https://lhcathome.cern.ch/lhcathome/result.php?resultid=234513174

is there no-one who can get this problem fixed?
ID: 39197 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 303
Credit: 231,154
RAC: 329
Message 39200 - Posted: 27 Jun 2019, 8:37:03 UTC - in response to Message 39197.  

This does not seem to be a general problem. I have looked at a few things on the server. Please let me know if these problems are still occurring.
ID: 39200 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,213
RAC: 28,095
Message 39201 - Posted: 27 Jun 2019, 12:21:51 UTC - in response to Message 39200.  
Last modified: 27 Jun 2019, 13:16:31 UTC

Please let me know if these problems are still occurring.
there was another one short time ago:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=234571121

Edit: Now I am seeing this problem on another of my machines, too. So my inital suspicion that something might be wrong with computer ID: 10555784 is definitely wrong.

Here the failed task from computer ID: 10450564: https://lhcathome.cern.ch/lhcathome/results.php?hostid=10450564
ID: 39201 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,245,672
RAC: 10,071
Message 39203 - Posted: 27 Jun 2019, 13:32:04 UTC - in response to Message 39201.  

Please let me know if these problems are still occurring.
there was another one short time ago:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=234571121

Edit: Now I am seeing this problem on another of my machines, too. So my inital suspicion that something might be wrong with computer ID: 10555784 is definitely wrong.


Or there is something wrong with computer ID: 10555784 and it has now spread to the other machine.
ID: 39203 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Theory Application : Could not get X509 credentials


©2019 CERN