From time to time I notice a CMS VM that prints the error "... singularity.conf not found".
Those VMs switch to idle a while later.

I suspect that at least one of the mirrors where the software packets are downloaded from is not up to date.

Got a bit more of this error message.
It looks like:
"sed: can't read .... singularity.conf"
It appears short before the condor ping.

Sorry, the setup continues too fast to get the rest (yet).

There's a big spike in job failures for the last hour or two; haven't caught the problem yet.

---

There's a big spike in job failures for the last hour or two; haven't caught the problem yet.

Exit code 99109, that means Uncaught exception in WMAgent step executor

Somebody at CERN may check the DNS nameservers that resolve gitlab.cern.ch.
I already asked Laurence.

Exit code 99109, that means Uncaught exception in WMAgent step executor

didn't the WMAgent play a big role in some major disturbtions about a year ago (or longer back)?

Exit code 99109, that means Uncaught exception in WMAgent step executor

didn't the WMAgent play a big role in some major disturbtions about a year ago (or longer back)?

There were some instabilities at one point, but I think we've caught most of the bugs by now. Since it's the thing that creates and manages all the jobs and steps of the work-flow (WM stands for work-flow management), it's not surprising that its name comes up from time to time.

---

WM stands for work-flow management

thanks, Ivan, for this information; I was wondering anyway all time long what WM(Agent) exactly means :-)