Message boards : CMS Application : CMS tasks failing
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3

AuthorMessage
M0CZY

Send message
Joined: 27 Apr 24
Posts: 21
Credit: 1,253,898
RAC: 1,544
Message 52597 - Posted: 26 Oct 2025, 14:11:02 UTC - in response to Message 52588.  

you should be aware though that they are of no use for the science.

I know that. It's their problem, not mine.
ID: 52597 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,064,045
RAC: 145,481
Message 52611 - Posted: 2 Nov 2025, 7:23:01 UTC

CMS seem to work on some hosts but mine get this error:

2025-11-02 08:12:45 (1029094): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-02 08:12:46 (1029094): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-02 08:13:17 (1029094): Guest Log: [DEBUG] % Total % Received % Xferd Average Speed Time Time Time Current
2025-11-02 08:13:17 (1029094): Guest Log: Dload Upload Total Spent Left Speed
2025-11-02 08:13:17 (1029094): Guest Log: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2025-11-02 08:13:17 (1029094): Guest Log: 100 54 0 54 0 0 70 0 --:--:-- --:--:-- --:--:-- 70
2025-11-02 08:13:17 (1029094): Guest Log: 100 54 0 54 0 0 66 0 --:--:-- --:--:-- --:--:-- 66
2025-11-02 08:13:17 (1029094): Guest Log: [DEBUG]
2025-11-02 08:13:17 (1029094): Guest Log: ERROR: Couldn't read proxy from: /tmp/x509up_u0
2025-11-02 08:13:17 (1029094): Guest Log: globus_credential: Error reading proxy credential
2025-11-02 08:13:17 (1029094): Guest Log: globus_credential: Error reading proxy credential: Couldn't read PEM from bio
2025-11-02 08:13:17 (1029094): Guest Log: OpenSSL Error: pem_lib.c:707: in library: PEM routines, function PEM_read_bio: no start line
2025-11-02 08:13:17 (1029094): Guest Log: Use -debug for further information.
2025-11-02 08:13:17 (1029094): Guest Log: [ERROR] Could not get an x509 credential
2025-11-02 08:13:17 (1029094): Guest Log: [ERROR] The x509 proxy creation failed.
ID: 52611 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1923
Credit: 149,493,174
RAC: 143,917
Message 52612 - Posted: 2 Nov 2025, 12:04:33 UTC - in response to Message 52611.  

CMS seem to work on some hosts but mine get this error:
...
Guest Log: [ERROR] Could not get an x509 credential
Guest Log: [ERROR] The x509 proxy creation failed.
same problem here :-(
ID: 52612 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,927,016
RAC: 1,807
Message 52613 - Posted: 2 Nov 2025, 16:44:48 UTC - in response to Message 52612.  

CMS seem to work on some hosts but mine get this error:
...
Guest Log: [ERROR] Could not get an x509 credential
Guest Log: [ERROR] The x509 proxy creation failed.
same problem here :-(
On the development system CMS is running OK:

00:01:21.379155 VMMDev: Guest Log: [INFO] Reading volunteer information
00:01:26.181010 VMMDev: Guest Log: [INFO] Requesting an X509 credential from LHC@home
00:01:26.975471 VMMDev: Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
00:01:28.156063 VMMDev: Guest Log: [INFO] Requesting an idtoken from LHC@home
00:01:28.755910 VMMDev: Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
00:01:29.491935 VMMDev: Guest Log: [INFO] CMS application starting. Check log files.
ID: 52613 · Report as offensive     Reply Quote
Pascal

Send message
Joined: 13 May 20
Posts: 52
Credit: 2,795,997
RAC: 3,164
Message 52614 - Posted: 2 Nov 2025, 19:27:56 UTC - in response to Message 52613.  

bonsoir, je ne sais pas si cela pourra aider mais il y a quelques jours ,toutes mes taches Cms partaient en erreur.J'ai réduit la limite d'unités téléchargées a 8 au lieu de"pas de limite".
J'ai demandé a Claude ai qui m'as fait faire une manipulation dans le terminal-sudo usermod -aG vboxusers $USER- et depuis tout fonctionne correctement.
Je suis sous linux mint 22.2 et virtualbox 7.24 et j'ai retiré kvm intel du noyau dans -sudo nano /etc/modprobe.d/blacklist-kvm.conf
blacklist kvm_intel
blacklist kvm_amd
blacklist kvm"

good evening, I don’t know if it will help but a few days ago, all my tasks Cms was leaving in error. I have reduced the limit of downloaded units to 8 instead of 'no limit'.
I asked Claude AI who made me do a manipulation in the terminal-sudo usermod -aG vboxusers $USER- and since then everything works correctly.
I am on linux mint 22.2 and virtualbox 7.24 and I removed kvm intel from the kernel in -sudo nano /etc/modprobe. d/blacklist-kvm.conf
kvm_intel blacklist
blacklist kvm_amd
blacklist kvm"
ID: 52614 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1119
Credit: 10,397,050
RAC: 19,398
Message 52620 - Posted: 3 Nov 2025, 15:45:56 UTC

Hello everyone. Thanks for your patience these last weeks. We were finally able to get our "supply chain" sorted out last Friday and get jobs flowing again. I refrained from posting a celebration, because Hallowe'en... Sure enough we had a little hiccup that I was able to get fixed on Saturday, but things went bad again this morning with failures in getting certificate proxies -- funnily enough it didn't affect my running machine, I must have got my last task just before the problem arose. The failure was in a CA server run by CERN IT, who were able to fix it soon after we raised a ticket.
So, fingers crossed, we are now back in action again and you can resume getting new tasks if you have been holding off.
ID: 52620 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,064,045
RAC: 145,481
Message 52623 - Posted: 4 Nov 2025, 7:40:10 UTC

Hint for volunteers using a local firewall:
CMS now requires TCP port 9620 to be open for outgoing connections to HTCondor CCB.
ID: 52623 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,064,045
RAC: 145,481
Message 52625 - Posted: 4 Nov 2025, 13:42:32 UTC - in response to Message 52623.  

Hint for volunteers using a local firewall:
CMS now requires TCP port 9620 to be open for outgoing connections to HTCondor CCB.

This afternoon a test for port 9620 has been added to the CMS bootstrap script.
Tasks passing the test report something like this to stderr.txt:
2025-11-04 10:29:28 (113263): Guest Log: [INFO] Testing connection to HTCondor-Collector
2025-11-04 10:29:29 (113263): Guest Log: [INFO] Testing connection to HTCondor-CCB
ID: 52625 · Report as offensive     Reply Quote
treaclepumpkin

Send message
Joined: 24 Jan 06
Posts: 7
Credit: 9,452,791
RAC: 10,393
Message 52650 - Posted: 11 Nov 2025, 7:45:34 UTC

Looks like the x509 errors are back? This is on a Windows VM.
2025-11-11 04:43:39 (6964): Guest Log: [INFO] Testing connection to http://cms-frontier.openhtc.io:8080/FrontierProd/Frontier/
2025-11-11 04:43:40 (6964): Guest Log: [INFO] Got a proxy from the local BOINC client
2025-11-11 04:43:40 (6964): Guest Log: [INFO] Will use it for CVMFS and Frontier
2025-11-11 04:43:41 (6964): Guest Log: [INFO] Reloading and probing the CVMFS configuration
2025-11-11 04:43:48 (6964): Guest Log: [INFO] Probing /cvmfs/grid.cern.ch... OK
2025-11-11 04:43:51 (6964): Guest Log: [INFO] Probing /cvmfs/singularity.opensciencegrid.org... OK
2025-11-11 04:43:51 (6964): Guest Log: [INFO] Probing /cvmfs/cms-ib.cern.ch... OK
2025-11-11 04:43:52 (6964): Guest Log: [INFO] Probing /cvmfs/oasis.opensciencegrid.org... OK
2025-11-11 04:44:13 (6964): Guest Log: [INFO] Probing /cvmfs/cms.cern.ch... OK
2025-11-11 04:44:14 (6964): Guest Log: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY
2025-11-11 04:44:14 (6964): Guest Log: [INFO] 2.7.2.0 http://s1ral-cvmfs.openhtc.io http://192.168.1.15:3128
2025-11-11 04:44:14 (6964): Guest Log: [INFO] Environment HTTP proxy: http://treacle:3128
2025-11-11 04:44:14 (6964): Guest Log: [INFO] Reading volunteer information
2025-11-11 04:44:16 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:44:17 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:44:47 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:44:47 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:45:17 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:45:18 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:45:48 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:45:48 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:46:18 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:46:18 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:46:49 (6964): Guest Log: [INFO] Requesting an X509 credential from LHC@home
2025-11-11 04:46:49 (6964): Guest Log: [INFO] Requesting an X509 credential from vLHC@home-dev
2025-11-11 04:47:19 (6964): Guest Log: [DEBUG] % Total % Received % Xferd Average Speed Time Time Time Current
2025-11-11 04:47:19 (6964): Guest Log: Dload Upload Total Spent Left Speed
2025-11-11 04:47:19 (6964): Guest Log: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2025-11-11 04:47:19 (6964): Guest Log: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
2025-11-11 04:47:19 (6964): Guest Log: curl: (56) Received HTTP code 403 from proxy after CONNECT
2025-11-11 04:47:20 (6964): Guest Log: [DEBUG]
2025-11-11 04:47:20 (6964): Guest Log: ERROR: Couldn't find a valid proxy.
2025-11-11 04:47:20 (6964): Guest Log: globus_sysconfig: File has zero length: File: /tmp/x509up_u0
2025-11-11 04:47:20 (6964): Guest Log: Use -debug for further information.
2025-11-11 04:47:20 (6964): Guest Log: [ERROR] Could not get an x509 credential
2025-11-11 04:47:20 (6964): Guest Log: [ERROR] The x509 proxy creation failed.

Getting quite a few of them again over the last few days. FWIW: The connection check to HTCondor CCB is passing.
Theory tasks seem to be working fine and even had one CMS task get to completion, amongst the failures.
ID: 52650 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1119
Credit: 10,397,050
RAC: 19,398
Message 52651 - Posted: 11 Nov 2025, 9:48:24 UTC - in response to Message 52650.  

Thanks for the report. I've pinged Laurence to alert him.
ID: 52651 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1119
Credit: 10,397,050
RAC: 19,398
Message 52675 - Posted: 18 Nov 2025, 14:39:26 UTC

We've had a lot of jobs fail this morning. From the logs it seems to be a network problem.
I suspect it's the Cloudflare outage.
ID: 52675 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1119
Credit: 10,397,050
RAC: 19,398
Message 52676 - Posted: 18 Nov 2025, 15:01:08 UTC - in response to Message 52675.  
Last modified: 18 Nov 2025, 15:42:06 UTC

We've had a lot of jobs fail this morning. From the logs it seems to be a network problem.
I suspect it's the Cloudflare outage.

We seem to be recovering now, according to the Running Jobs graph.
ID: 52676 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3

Message boards : CMS Application : CMS tasks failing


©2025 CERN