Message boards : News : CERN network problem
Message board moderation

To post messages, you must log in.

AuthorMessage
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 990
Credit: 6,264,307
RAC: 191
Message 34873 - Posted: 5 Apr 2018, 9:44:31 UTC

There was a major network problem at CERN this morning. It has apparently been resolved but not yet understood, according to the above link.
ID: 34873 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2375
Credit: 221,697,108
RAC: 142,935
Message 34874 - Posted: 5 Apr 2018, 10:00:46 UTC - in response to Message 34873.  

CMS requests lots of data from cmsfrontier.cern.ch, usually via TCP port 8000.
Since the outage at CERN this morning my running VMs partly generate requests to cmsfrontier.cern.ch TCP port 80 instead.

A check shows that port 8000 responds as expected but not port 80.

nc -v -z -w 5 cmsfrontier.cern.ch 8000
Connection to cmsfrontier.cern.ch 8000 port [tcp/irdmi] succeeded!

nc -v -z -w 5 cmsfrontier.cern.ch 80
nc: connect to cmsfrontier.cern.ch port 80 (tcp) timed out: Operation now in progress

It should also be checked why the VMs generate wrong requests.
ID: 34874 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 990
Credit: 6,264,307
RAC: 191
Message 34877 - Posted: 5 Apr 2018, 10:35:33 UTC - in response to Message 34874.  

That's working for me now:
[eesridr@pion:BOINC] > nc -v -z -w 5 cmsfrontier.cern.ch 8000
Connection to cmsfrontier.cern.ch 8000 port [tcp/irdmi] succeeded!
[eesridr@pion:BOINC] > nc -v -z -w 5 cmsfrontier.cern.ch 80
Connection to cmsfrontier.cern.ch 80 port [tcp/http] succeeded!

Thanks for the report, I guess it's up to CERN to look at the VM operation.
BTW, the outage affected our WMAgent too, but that was quickly rectified.
ID: 34877 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2375
Credit: 221,697,108
RAC: 142,935
Message 34878 - Posted: 5 Apr 2018, 12:59:20 UTC - in response to Message 34877.  

Although my project tasks are back to normal operation I wonder if there is an access restriction to port 80 for connections coming from outside.

I suspect this because the test is still not working:
nc -v -z -w 5 cmsfrontier.cern.ch 80
nc: connect to cmsfrontier.cern.ch port 80 (tcp) timed out: Operation now in progress


I crosschecked my firewall to see if there is a restriction, but all of my relevant local hosts are allowed to send/receive packets to/from CERN port 80.
ID: 34878 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 990
Credit: 6,264,307
RAC: 191
Message 34879 - Posted: 5 Apr 2018, 13:29:48 UTC - in response to Message 34878.  

My connection was from London, so not inside the CERN network.
ID: 34879 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2375
Credit: 221,697,108
RAC: 142,935
Message 35095 - Posted: 26 Apr 2018, 7:28:52 UTC

About half a hour ago my running VMs started to pull lots of CVMFS data from cvmfs02.grid.sinica.edu.tw.
According to my geolocation this is very unlikely as that server is on position 5 of the CVMFS configuration.
I wonder if this situation points out a major network problem affecting a couple of CVMFS locations.
ID: 35095 · Report as offensive     Reply Quote
m

Send message
Joined: 6 Sep 08
Posts: 115
Credit: 10,921,565
RAC: 5,347
Message 35112 - Posted: 29 Apr 2018, 11:29:02 UTC - in response to Message 35095.  
Last modified: 29 Apr 2018, 11:53:52 UTC

Mine are still trying... and not getting very far.

ACCESSED SITE....................CONNECT........BYTES......TIME.......USERS

cvmfs02.grid.sinica.edu.tw.......64....................0...............0:06:01....10
ID: 35112 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2375
Credit: 221,697,108
RAC: 142,935
Message 35116 - Posted: 29 Apr 2018, 19:23:38 UTC - in response to Message 35112.  

Mine are still trying... and not getting very far.

ACCESSED SITE....................CONNECT........BYTES......TIME.......USERS

cvmfs02.grid.sinica.edu.tw.......64....................0...............0:06:01....10

Ah, well, the sinica port problem.

According to the CVMFS documentation, stratum 1 systems should listen to both TCP ports, 80 and 8000.
http://cvmfs.readthedocs.io/en/stable/cpt-replica.html

cvmfs02.grid.sinica.edu.tw is one of those stratum 1 systems, but

nc -w 5 -z -v cvmfs02.grid.sinica.edu.tw 80 -> test to port 80 failes
nc -w 5 -z -v cvmfs02.grid.sinica.edu.tw 8000 -> test to port 8000 succeeds

This is not critical for normal operation as the other stratum 1 servers will respond, but if cvmfs02.grid.sinica.edu.tw is the only system left online for failover, it will become a problem.

I pointed that out several times on the old lhc@home MB but it seems that the comments never reached the responsible admins.
ID: 35116 · Report as offensive     Reply Quote

Message boards : News : CERN network problem


©2024 CERN