Message boards : CMS Application : CMS computation error
Message board moderation

To post messages, you must log in.

AuthorMessage
hadron

Send message
Joined: 4 Sep 22
Posts: 15
Credit: 1,772,522
RAC: 16,598
Message 47410 - Posted: 23 Oct 2022, 3:42:25 UTC

Every CMS task I received since 22 Oct 2022, 20:21:30 UTC (10 of them until I stopped asking for them) has ended in a computation error after only about 2 min 30 sec. Their stderr logs all show this:

2022-10-22 21:03:43 (1968): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2022-10-22 21:03:43 (1968): Guest Log: Ncat: Connection timed out.
2022-10-22 21:03:43 (1968): Guest Log: run 2
2022-10-22 21:03:43 (1968): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2022-10-22 21:03:43 (1968): Guest Log: Ncat: Connection timed out.
2022-10-22 21:03:43 (1968): Guest Log: run 3
2022-10-22 21:03:43 (1968): Guest Log: Ncat: Version 7.50 ( https://nmap.org/ncat )
2022-10-22 21:03:43 (1968): Guest Log: NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
2022-10-22 21:03:43 (1968): Guest Log: NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:02001002:system library:fopen:No such file or directory
2022-10-22 21:03:43 (1968): Guest Log: libnsock nsi_new2(): nsi_new (IOD #1)
2022-10-22 21:03:43 (1968): Guest Log: libnsock nsock_connect_tcp(): TCP connection requested to 137.138.156.85:9618 (IOD #1) EID 8
2022-10-22 21:03:43 (1968): Guest Log: libnsock nsock_trace_handler_callback(): Callback: CONNECT TIMEOUT for EID 8 [137.138.156.85:9618]
2022-10-22 21:03:43 (1968): Guest Log: Ncat: Connection timed out.
2022-10-22 21:03:43 (1968): Guest Log: [ERROR] Could not connect to vocms0840.cern.ch on port 9618
2022-10-22 21:03:43 (1968): Guest Log: [INFO] Testing connection to WMAgent
2022-10-22 21:03:44 (1968): Guest Log: [INFO] Testing connection to EOSCMS
2022-10-22 21:03:44 (1968): Guest Log: [INFO] Testing connection to CMS-Frontier
2022-10-22 21:03:45 (1968): Guest Log: [INFO] Testing connection to Frontier
2022-10-22 21:03:45 (1968): Guest Log: [DEBUG] Check your firewall and your network load
2022-10-22 21:03:45 (1968): Guest Log: [ERROR] Could not connect to all required network services
2022-10-22 21:03:45 (1968): Guest Log: [DEBUG] Volunteer: hadron (806228)
2022-10-22 21:03:45 (1968): Guest Log: [INFO] Shutting Down.
2022-10-22 21:04:14 (1968): VM Completion File Detected.
2022-10-22 21:04:14 (1968): VM Completion Message: Could not connect to all required network services

I am quite sure there are no network or firewall issues on my end, since ATLAS and Theory tasks do not have any problems. It may be some obscure setting I need to make that I never found documented anywhere. However, since this has never occurred previously, I'm inclined to believe it is an error that has unexpectedly occurred at vocms0840.cern.ch
ID: 47410 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1028
Credit: 48,282,708
RAC: 5,534
Message 47411 - Posted: 23 Oct 2022, 3:54:22 UTC - in response to Message 47410.  
Last modified: 23 Oct 2022, 4:36:53 UTC

it isn't on your end and is happening here and at -dev
Suspend your CMS tasks.

VM Completion Message: Could not connect to all required network services

ID: 47411 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,344,222
RAC: 114,049
Message 47413 - Posted: 23 Oct 2022, 9:44:24 UTC

Looks like server side issues are solved and fresh CMS tasks are working fine again.
ID: 47413 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1028
Credit: 48,282,708
RAC: 5,534
Message 47414 - Posted: 23 Oct 2022, 10:06:26 UTC - in response to Message 47413.  

no kidding
ID: 47414 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 918
Credit: 6,039,734
RAC: 1,405
Message 47415 - Posted: 23 Oct 2022, 14:59:47 UTC - in response to Message 47413.  

Looks like server side issues are solved and fresh CMS tasks are working fine again.

Yes, I don't know what the problem was (I was asleep!) but someone seems to have fixed it. The VM with connection problems - vocms0840 - is our HTCondor server.
ID: 47415 · Report as offensive     Reply Quote

Message boards : CMS Application : CMS computation error


©2023 CERN