Advanced search

Message boards : CMS Application : CMS jobs fail because it can't find vocms0267.cern.ch
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
seanr22a

Send message
Joined: 29 Nov 18
Posts: 41
Credit: 2,644,024
RAC: 38
Message 51613 - Posted: 28 Feb 2025, 19:47:02 UTC
Last modified: 28 Feb 2025, 19:47:59 UTC

https://lhcathome.cern.ch/lhcathome/result.php?resultid=419896635

2025-02-28 20:37:00 (114248): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-02-28 20:37:00 (114248): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-02-28 20:37:00 (114248): Guest Log: run 2
2025-02-28 20:37:00 (114248): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-02-28 20:37:00 (114248): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-02-28 20:37:00 (114248): Guest Log: run 3
2025-02-28 20:37:00 (114248): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-02-28 20:37:00 (114248): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-02-28 20:37:00 (114248): Guest Log: [ERROR] Could not connect to vocms0267.cern.ch on port 4080


root@pm108:~# nslookup
> server 8.8.8.8
Default server: 8.8.8.8
Address: 8.8.8.8#53
> cern.ch
Server: 8.8.8.8
Address: 8.8.8.8#53

Non-authoritative answer:
Name: cern.ch
Address: 188.184.77.250
Name: cern.ch
Address: 2001:1458:d00:3c::100:2f9
>
> vocms0267.cern.ch
Server: 8.8.8.8
Address: 8.8.8.8#53

** server can't find vocms0267.cern.ch: NXDOMAIN
>
ID: 51613 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2679
Credit: 286,806,599
RAC: 72,396
Message 51614 - Posted: 28 Feb 2025, 20:03:21 UTC - in response to Message 51613.  

Did you allow all required ports in your firewall?
These must be allowed for CMS:
80
8000
8080
443
9618
4080
1094

In addition vocms0267.cern.ch doesn't exist any more and should automatically be redirected to vocms267.cern.ch.
ID: 51614 · Report as offensive     Reply Quote
seanr22a

Send message
Joined: 29 Nov 18
Posts: 41
Credit: 2,644,024
RAC: 38
Message 51615 - Posted: 28 Feb 2025, 20:19:20 UTC - in response to Message 51614.  
Last modified: 28 Feb 2025, 20:52:08 UTC

if the server can't look up the ip address from the most common DNS server 8.8.8.8 it really doesn't matter which port the application is trying to use. Anyway I have no blocks on any outgoing port. I've tried several other DNS servers but they all say the same.

This is servers I have in my home in Sweden, just to test I tried from a server I have in my home in Thailand and the result is the same vocms0267.cern.ch does not exist.

vocms267.cern.ch works perfect from both sites so it seems that the DNS redirect vocms0267.cern.ch to vocms267.cern.ch isn't working?

[Edit]
I did an ugly fix so the CMS app look-up the correct ip local for vocms0267.cern.ch. Now it seems to be working, it has been running more than 10 minutes. Before it crashed in less than a minute.
At least it will be working until you change the ip address :)
ID: 51615 · Report as offensive     Reply Quote
alf

Send message
Joined: 5 Apr 20
Posts: 3
Credit: 42,594,890
RAC: 27,626
Message 51617 - Posted: 1 Mar 2025, 0:44:07 UTC

Everything was crunching flawlessly 24/7 for days on Atlas and CMS until we ran out of work yesterday.
All these new CMS tasks are aborting around the 2 min mark on all my computers.
I REALLY doubt all of a sudden volunteers need to go fidget with ports now.
I'll just go back on other projects til somebody figures this out like back in december with the empty 30min CMS tasks.


2025-02-28 17:25:45 (10704): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-02-28 17:25:45 (10704): Guest Log: [ERROR] Could not connect to vocms0267.cern.ch on port 4080
ID: 51617 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1234
Credit: 79,650,549
RAC: 96,588
Message 51618 - Posted: 1 Mar 2025, 1:28:53 UTC - in response to Message 51617.  

https://lhcathome.cern.ch/lhcathome/results.php?hostid=9930008&offset=0&show_names=0&state=6&appid=
ID: 51618 · Report as offensive     Reply Quote
seanr22a

Send message
Joined: 29 Nov 18
Posts: 41
Credit: 2,644,024
RAC: 38
Message 51619 - Posted: 1 Mar 2025, 3:46:25 UTC - in response to Message 51614.  


In addition vocms0267.cern.ch doesn't exist any more and should automatically be redirected to vocms267.cern.ch.


It seems like this is the problem. The redirect is not working.

I've slowly ramped up how many CMS jobs my server accept and is now running 4 jobs without problems. So my temporary ugly fix seems to solve the problem for now but it would be better if it was fixed on the cern side.
ID: 51619 · Report as offensive     Reply Quote
PaoloNasca

Send message
Joined: 11 Jul 19
Posts: 11
Credit: 1,806,754
RAC: 27
Message 51620 - Posted: 2 Mar 2025, 8:49:22 UTC
Last modified: 2 Mar 2025, 8:54:35 UTC

I’ve just upgraded VirtualBOX from 7.0.6 to 7.1.6 to revolve the 'Error while computing' after the powering on of the CMS VM.

@computezrmle
Today the issue is inside the CMS Virtual Machine regarding WMAgent, missing trusded CA Certificate and the “written-wrong” vocms0267.cern.ch server.

2025-03-02 04:03:51 (24108): Guest Log: [INFO] Testing connection to WMAgent
2025-03-02 04:03:52 (24108): Guest Log: [DEBUG] Status run 1 of up to 3: 1
2025-03-02 04:03:58 (24108): Guest Log: [DEBUG] Status run 2 of up to 3: 1
2025-03-02 04:04:10 (24108): Guest Log: [DEBUG] Status run 3 of up to 3: 1
2025-03-02 04:04:10 (24108): Guest Log: [DEBUG] run 1
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Connection refused.
2025-03-02 04:04:10 (24108): Guest Log: run 2
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Connection refused.
2025-03-02 04:04:10 (24108): Guest Log: run 3
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-03-02 04:04:10 (24108): Guest Log: NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
2025-03-02 04:04:10 (24108): Guest Log: NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:02001002:system library:fopen:No such file or directory
2025-03-02 04:04:10 (24108): Guest Log: libnsock nsi_new2(): nsi_new (IOD #1)
2025-03-02 04:04:10 (24108): Guest Log: libnsock nsock_connect_tcp(): TCP connection requested to 127.0.0.2:4080 (IOD #1) EID 8
2025-03-02 04:04:10 (24108): Guest Log: libnsock nsock_trace_handler_callback(): Callback: CONNECT ERROR [Connection refused (111)] for EID 8 [127.0.0.2:4080]
2025-03-02 04:04:10 (24108): Guest Log: Ncat: Connection refused.
2025-03-02 04:04:11 (24108): Guest Log: [ERROR] Could not connect to vocms0267.cern.ch on port 4080

https://lhcathome.cern.ch/lhcathome/result.php?resultid=419942573
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10865276
ID: 51620 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2679
Credit: 286,806,599
RAC: 72,396
Message 51621 - Posted: 2 Mar 2025, 9:10:31 UTC - in response to Message 51620.  

The cert issue is a red herring and can be ignored.
The real issue is a failing remapping of vocms0267.cern.ch (wrong) to vocms267.cern.ch (right).

If you use a local DNS service you may try to fix it until it is solved at CERN.
Otherwise you will have to wait until they fixed it there.
ID: 51621 · Report as offensive     Reply Quote
seanr22a

Send message
Joined: 29 Nov 18
Posts: 41
Credit: 2,644,024
RAC: 38
Message 51622 - Posted: 2 Mar 2025, 10:53:04 UTC - in response to Message 51620.  

If you are using pihole or similar or have a local DNS server it's easy to do a temporary fix.

You just have to add a local DNS entry (I did it in my firewall) if your router/firewall doesn't have that possibility maybe you have Pihole - click local DNS records (not CNAME), add vocms0267.cern.ch and ip 188.185.64.105 click save. From now on when CMS ask for vocms0267.cern.ch your Pihole will return the correct ip. When Cern had time to fix the issue just delete the DNS record.
ID: 51622 · Report as offensive     Reply Quote
PaoloNasca

Send message
Joined: 11 Jul 19
Posts: 11
Credit: 1,806,754
RAC: 27
Message 51623 - Posted: 3 Mar 2025, 5:59:53 UTC

It' works.
It was easy to add the suggested DNS entry in my home router.
Thanks
ID: 51623 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1908
Credit: 144,550,824
RAC: 76,453
Message 51625 - Posted: 3 Mar 2025, 7:48:27 UTC - in response to Message 51623.  

It' works.
It was easy to add the suggested DNS entry in my home router.
Thanks
you lucky one; my router unfortunately did NOT offer me this possibility :-(
So let's wait and see when the problem will be fixed server-side.
ID: 51625 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1110
Credit: 9,381,594
RAC: 5,065
Message 51632 - Posted: 3 Mar 2025, 13:46:46 UTC

Sorry I hadn't picked up on this earlier, I'm having a complicated private life at the moment. I thought the lack of jobs was due to other problems (e.g. my main machine seems to be having VirtualBox problems at the moment, and I hadn't looked beyond that over the weekend.
Laurence has fixed the naming bug and we seem to be getting more jobs running again now.
Again, apologies for not having looked into this earlier.
ID: 51632 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2679
Credit: 286,806,599
RAC: 72,396
Message 51633 - Posted: 3 Mar 2025, 16:52:27 UTC

The vocms267 issue is solved since around 13:12 UTC.
No need to modify a local DNS service any more.
ID: 51633 · Report as offensive     Reply Quote
alf

Send message
Joined: 5 Apr 20
Posts: 3
Credit: 42,594,890
RAC: 27,626
Message 51634 - Posted: 3 Mar 2025, 18:55:56 UTC

I confirm cms tasks no longer failing on my machines. Thanks to the cern team for the fix. Let's smash!
ID: 51634 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1234
Credit: 79,650,549
RAC: 96,588
Message 51635 - Posted: 3 Mar 2025, 21:50:10 UTC

Thanks Ivan
ID: 51635 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 118
Credit: 28,990,976
RAC: 45,755
Message 52096 - Posted: 18 Aug 2025, 23:39:04 UTC

THis appears to be happening again.
I just started doing CMS tasks again and am getting repeated failures due to these errors:
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Could not resolve hostname "cern.ch": Name or service not known. QUITTING.
2025-08-19 09:03:09 (7426): Guest Log: run 2
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Could not resolve hostname "cern.ch": Name or service not known. QUITTING.
2025-08-19 09:03:09 (7426): Guest Log: run 3
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:09 (7426): Guest Log: Ncat: Could not resolve hostname "cern.ch": Name or service not known. QUITTING.
2025-08-19 09:03:14 (7426): Guest Log: [ERROR] Could not connect to cern.ch on port 80
2025-08-19 09:03:20 (7426): Guest Log: [INFO] Testing connection to VCCS
2025-08-19 09:03:24 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:03:34 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:03:51 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:03:56 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Could not resolve hostname "vccs.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:03:56 (7426): Guest Log: run 2
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Could not resolve hostname "vccs.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:03:56 (7426): Guest Log: run 3
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:03:56 (7426): Guest Log: Ncat: Could not resolve hostname "vccs.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:04:02 (7426): Guest Log: [ERROR] Could not connect to vccs.cern.ch on port 443
2025-08-19 09:04:10 (7426): Guest Log: [INFO] Testing connection to HTCondor
2025-08-19 09:04:15 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:04:24 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:04:36 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:04:42 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0840.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:04:42 (7426): Guest Log: run 2
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0840.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:04:42 (7426): Guest Log: run 3
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:04:42 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0840.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:04:46 (7426): Guest Log: [ERROR] Could not connect to vocms0840.cern.ch on port 9618
2025-08-19 09:04:50 (7426): Guest Log: [INFO] Testing connection to WMAgent
2025-08-19 09:04:56 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:05:11 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:05:25 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:05:30 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:05:30 (7426): Guest Log: run 2
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:05:30 (7426): Guest Log: run 3
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:05:30 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0267.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:05:33 (7426): Guest Log: [ERROR] Could not connect to vocms0267.cern.ch on port 4080
2025-08-19 09:05:38 (7426): Guest Log: [INFO] Testing connection to EOSCMS
2025-08-19 09:05:42 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:05:52 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:06:07 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:06:11 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Could not resolve hostname "eoscms-ns-ip563.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:11 (7426): Guest Log: run 2
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Could not resolve hostname "eoscms-ns-ip563.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:11 (7426): Guest Log: run 3
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:11 (7426): Guest Log: Ncat: Could not resolve hostname "eoscms-ns-ip563.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:12 (7426): Guest Log: [ERROR] Could not connect to eoscms-ns-ip563.cern.ch on port 1094
2025-08-19 09:06:19 (7426): Guest Log: [INFO] Testing connection to CMS-Factory
2025-08-19 09:06:23 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:06:33 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:06:48 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:06:52 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0205.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:52 (7426): Guest Log: run 2
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0205.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:52 (7426): Guest Log: run 3
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:06:52 (7426): Guest Log: Ncat: Could not resolve hostname "vocms0205.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:06:57 (7426): Guest Log: [ERROR] Could not connect to vocms0205.cern.ch on port 80
2025-08-19 09:07:08 (7426): Guest Log: [INFO] Testing connection to CMS-Frontier
2025-08-19 09:07:12 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:07:24 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:07:37 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:07:40 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Could not resolve hostname "cmsfrontier.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:07:40 (7426): Guest Log: run 2
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Could not resolve hostname "cmsfrontier.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:07:40 (7426): Guest Log: run 3
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:07:40 (7426): Guest Log: Ncat: Could not resolve hostname "cmsfrontier.cern.ch": Name or service not known. QUITTING.
2025-08-19 09:07:46 (7426): Guest Log: [ERROR] Could not connect to cmsfrontier.cern.ch on port 8000
2025-08-19 09:07:50 (7426): Guest Log: [INFO] Testing connection to Frontier
2025-08-19 09:07:54 (7426): Guest Log: [DEBUG] Status run 1 of up to 3: 2
2025-08-19 09:08:04 (7426): Guest Log: [DEBUG] Status run 2 of up to 3: 2
2025-08-19 09:08:20 (7426): Guest Log: [DEBUG] Status run 3 of up to 3: 2
2025-08-19 09:08:25 (7426): Guest Log: [DEBUG] run 1
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Could not resolve hostname "cms-frontier.openhtc.io": Name or service not known. QUITTING.
2025-08-19 09:08:25 (7426): Guest Log: run 2
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Could not resolve hostname "cms-frontier.openhtc.io": Name or service not known. QUITTING.
2025-08-19 09:08:25 (7426): Guest Log: run 3
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2025-08-19 09:08:25 (7426): Guest Log: Ncat: Could not resolve hostname "cms-frontier.openhtc.io": Name or service not known. QUITTING.
2025-08-19 09:08:29 (7426): Guest Log: [ERROR] Could not connect to cms-frontier.openhtc.io on port 8080
2025-08-19 09:08:36 (7426): Guest Log: [DEBUG] Check your firewall and your network load
2025-08-19 09:08:40 (7426): Guest Log: [ERROR] Could not connect to all required network services
2025-08-19 09:08:42 (7426): Guest Log: [DEBUG] Volunteer: Dark Angel (268818)
2025-08-19 09:08:46 (7426): Guest Log: [INFO] Shutting Down.
2025-08-19 09:09:17 (7426): VM Completion File Detected.
2025-08-19 09:09:17 (7426): VM Completion Message: Could not connect to all required network services

I have an Atlas task that appears to be running fine now it's gotten some data via CVMFS (the config probe is successful from this machine, I checked) so I don't know what's going on.
ID: 52096 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1110
Credit: 9,381,594
RAC: 5,065
Message 52100 - Posted: 20 Aug 2025, 13:28:06 UTC - in response to Message 52096.  

Hmm, you are picking up outdated addresses from somewhere. vocms0267.cern.ch definitely doesn't exist any more.
It doesn't exist in any recent files on cvmfs:
[lxplus998:mancl] > find /cvmfs/grid.cern.ch/vc -type f -exec grep --color 'vocms0267' {} \; -ls
    nc -z -v -w 30 vocms0267.cern.ch 4080 >/tmp/stdout 2>/tmp/stderr
    50529     18 -rwxrwxr-x   1 cvmfs    cvmfs       18304 Mar  5  2021 /cvmfs/grid.cern.ch/vc/sbin/bootstrap
bs_test_basic_network_connection vocms0267.cern.ch 4080 WMAgent
    65691     10 -rwxr-xr-x   1 cvmfs    cvmfs        9904 Jun 23  2022 /cvmfs/grid.cern.ch/vc/sbin/bootstrap-cms
bs_test_basic_network_connection vocms0267.cern.ch 4080 WMAgent
    65698     10 -rwxr-xr-x   1 cvmfs    cvmfs       10009 Jul 27  2022 /cvmfs/grid.cern.ch/vc/sbin/bootstrap-idtoken.org
bs_test_basic_network_connection vocms0267.cern.ch 4080 WMAgent
    65699     11 -rwxr-xr-x   1 cvmfs    cvmfs       10257 Aug 11  2022 /cvmfs/grid.cern.ch/vc/sbin/bootstrap-idtoken
MATCH_SESSION = submit-side@matchsession/vccondorce02.cern.ch, submit-side@matchsession/vocms0267.cern.ch, submit-side@matchsession/188.184.94.254, submit-side@matchsession/137.138.52.94
      404      3 -rw-rw-r--   1 cvmfs    cvmfs        2691 Oct  2  2018 /cvmfs/grid.cern.ch/vc/etc/condor/config.d/10_security.config
MATCH_SESSION = submit-side@matchsession/vccondorce02.cern.ch, submit-side@matchsession/vocms0267.cern.ch, submit-side@matchsession/188.184.94.254, submit-side@matchsession/137.138.52.94
    65815      3 -rw-r--r--   1 cvmfs    cvmfs        2691 Aug 26  2022 /cvmfs/grid.cern.ch/vc/vm-qa/etc/condor/config.d/10_security.config
    nc -z -v -w 30 vocms0267.cern.ch 4080 >/tmp/stdout 2>/tmp/stderr
    65795     15 -rwxr-xr-x   1 cvmfs    cvmfs       14339 Aug 26  2022 /cvmfs/grid.cern.ch/vc/vm-qa/sbin/bootstrap
bs_test_basic_network_connection vocms0267.cern.ch 4080 WMAgent
    66080     10 -rwxr-xr-x   1 cvmfs    cvmfs       10117 Sep 30  2022 /cvmfs/grid.cern.ch/vc/vm-qa/sbin/bootstrap-cms
MATCH_SESSION = submit-side@matchsession/vccondorce02.cern.ch, submit-side@matchsession/vocms0267.cern.ch, submit-side@matchsession/188.184.94.254, submit-side@matchsession/137.138.52.94
    66288      3 -rw-r--r--   1 cvmfs    cvmfs        2691 Nov 17  2022 /cvmfs/grid.cern.ch/vc/vm-master/etc/condor/config.d/10_security.config
    nc -z -v -w 30 vocms0267.cern.ch 4080 >/tmp/stdout 2>/tmp/stderr
    66280     15 -rwxr-xr-x   1 cvmfs    cvmfs       14339 Nov 17  2022 /cvmfs/grid.cern.ch/vc/vm-master/sbin/bootstrap
bs_test_basic_network_connection vocms0267.cern.ch 4080 WMAgent
    66360     10 -rwxr-xr-x   1 cvmfs    cvmfs       10117 Nov 17  2022 /cvmfs/grid.cern.ch/vc/vm-master/sbin/bootstrap-cms
W

Its replacement vocms267 is mentioned in two files:
[lxplus998:mancl] > find /cvmfs/grid.cern.ch/vc -type f -exec grep --color 'vocms267' {} \; -ls
bs_test_basic_network_connection vocms267.cern.ch 4080 WMAgent
    92842      7 -rwxr-xr-x   1 cvmfs    cvmfs        7001 Apr  4 09:33 /cvmfs/grid.cern.ch/vc/vm-qa/sbin/bootstrap-idtoken
bs_test_basic_network_connection vocms267.cern.ch 4080 WMAgent
    92850      7 -rwxr-xr-x   1 cvmfs    cvmfs        7001 Apr  4 09:34 /cvmfs/grid.cern.ch/vc/vm-master/sbin/bootstrap-idtoken

I'll alert Laurence to this thread.
ID: 52100 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1110
Credit: 9,381,594
RAC: 5,065
Message 52101 - Posted: 20 Aug 2025, 15:34:23 UTC - in response to Message 52100.  

OK, this doesn't make sense to us. Laurence suggests a project reset.
ID: 52101 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2679
Credit: 286,806,599
RAC: 72,396
Message 52102 - Posted: 20 Aug 2025, 15:56:42 UTC - in response to Message 52101.  

OK, this doesn't make sense to us. Laurence suggests a project reset.

I'd suggest to also restart the local Squid with an empty cache.
ID: 52102 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1234
Credit: 79,650,549
RAC: 96,588
Message 52138 - Posted: 29 Aug 2025, 8:56:32 UTC

just started getting this on all of my CMS running hosts

ID: 52138 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : CMS Application : CMS jobs fail because it can't find vocms0267.cern.ch


©2025 CERN