Message boards : ATLAS application : lcgft-atlas.gridpp.rl.ac.uk down
Message board moderation

To post messages, you must log in.

AuthorMessage
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1351
Credit: 67,802,233
RAC: 96,721
Message 40901 - Posted: 11 Dec 2019, 15:46:23 UTC

My ATLAS tasks switched to the fallback server ccfrontier.in2p3.fr.
Looks like lcgft-atlas.gridpp.rl.ac.uk is down.
ID: 40901 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 311
Credit: 9,931,593
RAC: 8,063
Message 40902 - Posted: 11 Dec 2019, 16:10:35 UTC - in response to Message 40901.  

You are too fast :) This server has been permanently shut down today, and earlier I changed things so that new WU do not use it.

The existing WU should fallover to one of the other servers so tasks should not fail.
ID: 40902 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1351
Credit: 67,802,233
RAC: 96,721
Message 40903 - Posted: 11 Dec 2019, 16:27:57 UTC - in response to Message 40902.  

Permanently?
That means, I can remove that server from my proxy configuration?

Is ccfrontier.in2p3.fr the new preferred server or still a fallback?
Or does ATLAS now get a Cloudflare proxy like CMS (cms-frontier.openhtc.io)?
ID: 40903 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 311
Credit: 9,931,593
RAC: 8,063
Message 40915 - Posted: 12 Dec 2019, 11:03:36 UTC - in response to Message 40903.  

Yes, permanently. ccfrontier.in2p3.fr and atlasfrontier-ai.cern.ch are the only two Frontier servers used by ATLAS now.

Where do you have this server in your configuration? I don't think it's possible to set Frontier servers yourself, and only the squid proxies are configured by the ATLAS bootstrap scripts.

Frontier access is configured through an environment variable which is set automatically by the task, or in the case where someone has a proxy configured, it is set by the bootstrap script to use that proxy, eg (for the case without own proxy)

FRONTIER_SERVER=(serverurl=http://atlasfrontier-ai.cern.ch:8000/atlr)(serverurl=http://atlasfrontier2-ai.cern.ch:8000/atlr)(serverurl=http://atlasfrontier1-ai.cern.ch:8000/atlr)(serverurl=http://ccfrontier.in2p3.fr:23128/ccin2p3-AtlasFrontier)(serverurl=http://ccfrontier01.in2p3.fr:23128/ccin2p3-AtlasFrontier)(serverurl=http://ccfrontier05.in2p3.fr:23128/ccin2p3-AtlasFrontier)(proxyurl=http://db-atlas-squid.ndgf.org:3128)(proxyurl=http://atlas-db-squid.grid.uio.no:3128)(proxyurl=http://dcache.ijs.si:3128)(proxyurl=http://atlasbpfrontier.cern.ch:3127)(proxyurl=http://atlasbpfrontier.fnal.gov:3127)


serverurl defines the Frontier servers themselves and proxyurl the squid proxies.

I agree it would be a good idea to use a Cloudflare proxy for ATLAS volunteers without their own squid. I wonder if the existing CVMFS proxies could be used for Frontier.
ID: 40915 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1351
Credit: 67,802,233
RAC: 96,721
Message 40925 - Posted: 12 Dec 2019, 15:27:18 UTC - in response to Message 40915.  

Where do you have this server in your configuration? I don't think it's possible to set Frontier servers yourself, and only the squid proxies are configured by the ATLAS bootstrap scripts.

I configured it as parent in my squid configuration.
Nothing to worry about. It just creates a couple of statistics data.


What I see from the currently running tasks is that the order of the frontier servers is different from your example.
Order matters as the first server in the list will be asked first and all others will be there for failover.
If you need a special order you may investigate why this is different, otherwise just ignore it.

Your example:
FRONTIER_SERVER=(serverurl=http://atlasfrontier-ai.cern.ch:8000/atlr)(serverurl=http://atlasfrontier2-ai.cern.ch:8000/atlr)(serverurl=http://atlasfrontier1-ai.cern.ch:8000/atlr)(serverurl=http://ccfrontier.in2p3.fr:23128/ccin2p3-AtlasFrontier)(serverurl=http://ccfrontier01.in2p3.fr:23128/ccin2p3-AtlasFrontier)(serverurl=http://ccfrontier05.in2p3.fr:23128/ccin2p3-AtlasFrontier)(proxyurl=http://db-atlas-squid.ndgf.org:3128)(proxyurl=http://atlas-db-squid.grid.uio.no:3128)(proxyurl=http://dcache.ijs.si:3128)(proxyurl=http://atlasbpfrontier.cern.ch:3127)(proxyurl=http://atlasbpfrontier.fnal.gov:3127)

My local setup.sh.local (configured by the task):
export FRONTIER_SERVER="(serverurl=http://ccfrontier.in2p3.fr:23128/ccin2p3-AtlasFrontier)(serverurl=http://atlasfrontier-ai.cern.ch:8000/atlr)(proxyurl=http://</censored>:3128)"



I agree it would be a good idea to use a Cloudflare proxy for ATLAS volunteers without their own squid. I wonder if the existing CVMFS proxies could be used for Frontier.

CMS is using CVMFS via Cloudflare but also Frontier via cms-frontier.openhtc.io:8080.
For details you may contact Laurence and/or Dave Dykstra.
ID: 40925 · Report as offensive     Reply Quote
Gunde

Send message
Joined: 9 Jan 15
Posts: 65
Credit: 318,776,653
RAC: 118,195
Message 41084 - Posted: 26 Dec 2019, 19:57:07 UTC

http://cvmrepo.web.cern.ch/ is down
it would timeout and try mirror but repeat with same ipv6 address.

Could anyone report to CERN IT team?
ID: 41084 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 881
Credit: 32,655,678
RAC: 45,820
Message 41085 - Posted: 26 Dec 2019, 20:28:17 UTC - in response to Message 41084.  

You can send a PM to Nils Høimyr.
Maybe he reach someone tomorrow in Cern-IT.
ID: 41085 · Report as offensive     Reply Quote
Gunde

Send message
Joined: 9 Jan 15
Posts: 65
Credit: 318,776,653
RAC: 118,195
Message 41086 - Posted: 26 Dec 2019, 21:53:26 UTC

Sent a pm to him about this issue.

In middle of retry i got message to try yum-config-manager --disable cernvm or subscription-manager repos --disable=cernvm but could not continue after using them. I will try another day.
ID: 41086 · Report as offensive     Reply Quote

Message boards : ATLAS application : lcgft-atlas.gridpp.rl.ac.uk down


©2020 CERN