From the jobs graphs, things came good again about 2000 UTC last night.

---

[ERROR] Could not connect to vocms0830.cern.ch on port 9618

seems like there is some issues with the servers.

The servers do that here as soon as I am asleep
Over 100 of these https://lhcathome.cern.ch/lhcathome/result.php?resultid=432978091 (-dev too of course)

In reply to Toby Broom's message of 22 Feb 2026:

[ERROR] Could not connect to vocms0830.cern.ch on port 9618

seems like there is some issues with the servers.

Yes, there are jobs in the queues, but I'm getting that error on all my tasks at present.

---

All CMS tasks failing within 4 minutes


<core_client_version>8.2.4</core_client_version>
<![CDATA[
<message>
The filename or extension is too long.
(0xce) - exit code 206 (0xce)</message>
<stderr_txt>

In reply to alf's message of 23 Feb 2026:

All CMS tasks failing within 4 minutes


<core_client_version>8.2.4</core_client_version>
<![CDATA[
<message>
The filename or extension is too long.
(0xce) - exit code 206 (0xce)</message>
<stderr_txt>

the old, frequently happening and well known error :-(

A closer look at the stderrs of the failed tasks reveales that there are numerous connection timeouts to all kinds of targets.
And at the end, it says "VM Completion Message: Could not connect to all required network services"
So, there seems to exist a major network problem at CERN

In reply to Erich56's message of 23 Feb 2026:

A closer look at the stderrs of the failed tasks reveales that there are numerous connection timeouts to all kinds of targets.
And at the end, it says "VM Completion Message: Could not connect to all required network services"
So, there seems to exist a major network problem at CERN

Yes, there is a mishmash of error messages, but the one of real importance seems to be the failure to connect to vocms0830. I can recreate the error with

nc -z -vvv vocms0830.cern.ch 9618

. However, this works inside CERN (from lxplus), so there does appear to be a network component to the problem. I've put a request out to the internal Mattermost discussion boards, hopefully to the appropriate channel!

---

In reply to ivan's message of 23 Feb 2026:

Yes, there is a mishmash of error messages, but the one of real importance seems to be the failure to connect to vocms0830. I can recreate the error with

nc -z -vvv vocms0830.cern.ch 9618

. However, this works inside CERN (from lxplus), so there does appear to be a network component to the problem. I've put a request out to the internal Mattermost discussion boards, hopefully to the appropriate channel!

Interesting...

Is this related to puppet not running on vocms0830 and firewall exceptions being removed?

---

Interesting...

Is this related to puppet not running on vocms0830 and firewall exceptions being removed?

Yes, this is related. Puppet is not running, because the host certificate puppet uses is expired. If puppet is not running for a couple of days, the host will be removed from landbsets, hence removing all firewall rules. Usually, the certificate on the host should have been renewed automatically. I am currently checking, why it didn't do that. Also we should have gotten an (probably multiple) emails on that.

---

just ran a single test an still not working

No, I'm waiting to hear back how they plan to fix it -- manually reinstate the certificate and firewall rules, or wait until they've worked out why the autorenew didn't work.

---

In reply to ivan's message of 23 Feb 2026:

No, I'm waiting to hear back how they plan to fix it -- manually reinstate the certificate and firewall rules, or wait until they've worked out why the autorenew didn't work.

whatever will happen - I'm afraid it will take a while until CMS is working again :-(

Good morning everyone, after calculating the tasks, I restarted the project and I can confirm that the CMS tasks are still failing.

For now, I've removed it from the options for requesting tasks.

February 24, 2026

Best regards

---

In reply to Erich56's message of 24 Feb 2026:

In reply to ivan's message of 23 Feb 2026:

No, I'm waiting to hear back how they plan to fix it -- manually reinstate the certificate and firewall rules, or wait until they've worked out why the autorenew didn't work.

whatever will happen - I'm afraid it will take a while until CMS is working again :-(

Unfortunately, you may be right. There has been some dififculty in reinstating the collector's certificate.

---

We've now been told that the cleanest solution is to rebuild the host from scratch, so that is being worked on.

---

That's a strange error:

The filename wildcard characters * or ? were entered incorrectly or too many were specified.
(0xd0) - exit code 208 (0xd0)</message>

In reply to [VENETO] boboviz's message of 25 Feb 2026:

That's a strange error:

The filename wildcard characters * or ? were entered incorrectly or too many were specified.
(0xd0) - exit code 208 (0xd0)</message>

That's a wrong interpretation of the error code by Windows like your errors:

Nome del file o estensione troppo lunga.
(0xce) - exit code 206 (0xce)</message>

and

I caratteri jolly per i nomi di file, * o ?, sono stati immessi incorrettamente oppure ne sono stati specificati troppi.
(0xd0) - exit code 208 (0xd0)</message>

In reply to Crystal Pellet's message of 25 Feb 2026:

I caratteri jolly per i nomi di file, * o ?, sono stati immessi incorrettamente oppure ne sono stati specificati troppi.
(0xd0) - exit code 208 (0xd0)</message>

So i have to wait a new cms app version?

In reply to ivan's message of 24 Feb 2026:

We've now been told that the cleanest solution is to rebuild the host from scratch, so that is being worked on.

Ivan, any idea how long this might take? a couple of days? 1 week or 2? or even longer?