Message boards :
News :
CERN network problem
Message board moderation
| Author | Message |
|---|---|
 Send message Joined: 29 Aug 05 Posts: 925 Credit: 6,097,504 RAC: 1,015  |
There was a major network problem at CERN this morning. It has apparently been resolved but not yet understood, according to the above link.  |
 Send message Joined: 15 Jun 08 Posts: 2139 Credit: 174,899,083 RAC: 105,682   |
CMS requests lots of data from cmsfrontier.cern.ch, usually via TCP port 8000. Since the outage at CERN this morning my running VMs partly generate requests to cmsfrontier.cern.ch TCP port 80 instead. A check shows that port 8000 responds as expected but not port 80. nc -v -z -w 5 cmsfrontier.cern.ch 8000 Connection to cmsfrontier.cern.ch 8000 port [tcp/irdmi] succeeded! nc -v -z -w 5 cmsfrontier.cern.ch 80 nc: connect to cmsfrontier.cern.ch port 80 (tcp) timed out: Operation now in progress It should also be checked why the VMs generate wrong requests. |
|
Send message Joined: 29 Aug 05 Posts: 925 Credit: 6,097,504 RAC: 1,015 |
That's working for me now: [eesridr@pion:BOINC] > nc -v -z -w 5 cmsfrontier.cern.ch 8000 Connection to cmsfrontier.cern.ch 8000 port [tcp/irdmi] succeeded! [eesridr@pion:BOINC] > nc -v -z -w 5 cmsfrontier.cern.ch 80 Connection to cmsfrontier.cern.ch 80 port [tcp/http] succeeded! Thanks for the report, I guess it's up to CERN to look at the VM operation. BTW, the outage affected our WMAgent too, but that was quickly rectified. |
|
Send message Joined: 15 Jun 08 Posts: 2139 Credit: 174,899,083 RAC: 105,682 |
Although my project tasks are back to normal operation I wonder if there is an access restriction to port 80 for connections coming from outside. I suspect this because the test is still not working: nc -v -z -w 5 cmsfrontier.cern.ch 80 nc: connect to cmsfrontier.cern.ch port 80 (tcp) timed out: Operation now in progress I crosschecked my firewall to see if there is a restriction, but all of my relevant local hosts are allowed to send/receive packets to/from CERN port 80. |
|
Send message Joined: 29 Aug 05 Posts: 925 Credit: 6,097,504 RAC: 1,015 |
My connection was from London, so not inside the CERN network. |
|
Send message Joined: 15 Jun 08 Posts: 2139 Credit: 174,899,083 RAC: 105,682 |
About half a hour ago my running VMs started to pull lots of CVMFS data from cvmfs02.grid.sinica.edu.tw. According to my geolocation this is very unlikely as that server is on position 5 of the CVMFS configuration. I wonder if this situation points out a major network problem affecting a couple of CVMFS locations. |
|
Send message Joined: 6 Sep 08 Posts: 112 Credit: 9,247,894 RAC: 2,565  
|
Mine are still trying... and not getting very far. ACCESSED SITE....................CONNECT........BYTES......TIME.......USERS cvmfs02.grid.sinica.edu.tw.......64....................0...............0:06:01....10 |
|
Send message Joined: 15 Jun 08 Posts: 2139 Credit: 174,899,083 RAC: 105,682 |
Mine are still trying... and not getting very far. Ah, well, the sinica port problem. According to the CVMFS documentation, stratum 1 systems should listen to both TCP ports, 80 and 8000. http://cvmfs.readthedocs.io/en/stable/cpt-replica.html cvmfs02.grid.sinica.edu.tw is one of those stratum 1 systems, but nc -w 5 -z -v cvmfs02.grid.sinica.edu.tw 80 -> test to port 80 failes nc -w 5 -z -v cvmfs02.grid.sinica.edu.tw 8000 -> test to port 8000 succeeds This is not critical for normal operation as the other stratum 1 servers will respond, but if cvmfs02.grid.sinica.edu.tw is the only system left online for failover, it will become a problem. I pointed that out several times on the old lhc@home MB but it seems that the comments never reached the responsible admins. |
©2023 CERN