Counting up the job failures in the current workflow, the 5 users with highest counts (anonymised due to GDPR) are:

      3 614757
     13 xxxxxx
     56 yyyyy
    109 zzzzzz
    426 nnnnnn

---

Got a bunch that failed again the past couple of hours, switched over to ATLAS for the time being

Actually, it seems to be in perfect correlation with the number of jobs constantly dropping since 10:24 UTC

---

I just noticed that since about noontime of today most of the tasks failed after various periods of time (between a few minutes and half an hour).
In many cases, stderr says that filename or extension is too long:

Der Dateiname oder die Erweiterung ist zu lang.
(0xce) - exit code 206 (0xce)</message>

what's going on?

In reply to Erich56's message of 28 Jan 2026:

I just noticed that since about noontime of today most of the tasks failed after various periods of time (between a few minutes and half an hour).
In many cases, stderr says that filename or extension is too long:

Der Dateiname oder die Erweiterung ist zu lang.
(0xce) - exit code 206 (0xce)

what's going on?

Yes, I've noticed the jobs falling over the last hours. Not sure what's going on yet, that's not a common error message I'm tempted to believe it's coming from the operating system, but more digging needed.

---

In reply to Garrulus glandarius's message of 28 Jan 2026:

Got a bunch that failed again the past couple of hours, switched over to ATLAS for the time being

Actually, it seems to be in perfect correlation with the number of jobs constantly dropping since 10:24 UTC

Yes, that does seem to be the case.

---

In reply to Erich56's message of 28 Jan 2026:

I just noticed that since about noontime of today most of the tasks failed after various periods of time (between a few minutes and half an hour).
In many cases, stderr says that filename or extension is too long:

Der Dateiname oder die Erweiterung ist zu lang.
(0xce) - exit code 206 (0xce)

what's going on?

Some of your failures are network related; could be a problem at cern:

2026-01-28 15:27:42 (24776): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2026-01-28 15:27:42 (24776): Guest Log: Ncat: Connection to 137.138.53.124 failed: Connection timed out.
2026-01-28 15:27:42 (24776): Guest Log: Ncat: Trying next address...
2026-01-28 15:27:42 (24776): Guest Log: Ncat: Network is unreachable.

[TinyPC:words] > nslookup 137.138.53.124
Server: UnKnown
Address: 10.174.98.157

Name: vocms0204.cern.ch
Address: 137.138.53.124

---

Same here

---

Volunteer Mad Scientist For Life

unbelievable are you trying to promote linux again?

It looks like the dip might be bottoming out. Digits cruciate!

---

In reply to ivan's message of 28 Jan 2026:

It looks like the dip might be bottoming out. Digits cruciate!

I hope so Ivan.......fingers crossed

In reply to Magic Quantum Mechanic's message of 28 Jan 2026:

In reply to ivan's message of 28 Jan 2026:

It looks like the dip might be bottoming out. Digits cruciate!

I hope so Ivan.......fingers crossed

I wish I'd said that!

---

I figured 2 fingers crossed might work so I tried a new CMS and since the ones crashing would do it in 25 minutes or less it looks like this one is running and just passed the first hour so if this ends up Valid I will start up more on here and if they keep running reload the other hosts.

I did have to get in the VB Manager and remove all those crashed ones

We do have a fairly good rebound, but it might be stalling. Federica has called us in for a meeting tomorrow; let's see if we can come up with some ideas by then. I' m preoccupied by having just four weeks to reclaim my Uni and CERN computing credentials, tho'but...

---

Sounds good and have a fine day.......it is a holiday here on the 29th

Things seem to be going smoothly now. My main LHC rig is still busy with ATLAS, but an old i7 has been running a new-ish CMS task for over 13 hours with apparently normal progress.

---

same here (with the exception of only 1 task this late morning) :-)