Message boards :
ATLAS application :
Probing /cvmfs/atlas.cern.ch... Failed!
Message board moderation
Author | Message |
---|---|
Send message Joined: 9 Jul 18 Posts: 11 Credit: 604,030 RAC: 0 |
Atlas WU failed after about 10 minutes with the following error. <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> process exited with code 195 (0xc3, -61)</message> <stderr_txt> 22:30:59 (268507): wrapper (7.7.26015): starting 22:30:59 (268507): wrapper: running run_atlas (--nthreads 6) [2023-04-13 22:30:59] Arguments: --nthreads 6 [2023-04-13 22:30:59] Threads: 6 [2023-04-13 22:30:59] Checking for CVMFS [2023-04-13 22:31:01] Probing /cvmfs/atlas.cern.ch... Failed! [2023-04-13 22:31:01] Probing /cvmfs/atlas-condb.cern.ch... Failed! [2023-04-13 22:31:01] cvmfs_config probe atlas.cern.ch atlas-condb.cern.ch failed, aborting the job 22:41:01 (268507): run_atlas exited; CPU time 0.076063 22:41:01 (268507): app exit status: 0x1 22:41:01 (268507): called boinc_finish(195) </stderr_txt> ]]> I did run sudo cvmfs_config chksetup before starting with a return of "OK" so I thought I was good. Any idea what I'm missing? Running Linux Ubuntu 22.04.2 LTS [5.19.0-32-generic|libc 2.35] with VB 7.0.6-155176 |
Send message Joined: 21 Feb 11 Posts: 59 Credit: 543,728 RAC: 19 |
Does cvmfs_config probe work? |
Send message Joined: 15 Jun 08 Posts: 2425 Credit: 227,502,920 RAC: 129,877 |
Is it your first try to use CVMFS? If so, did you follow the advices given here? https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594 https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5595 Run "[sudo] cvmfs_config probe" before you try another ATLAS task. As long as the probes fail your ATLAS tasks will all fail. If you can't get it running, post the output of: [sudo] cvmfs_config showconfig -s atlas.cern.ch systemctl status autofs.service BTW: Your computer details page shows you are running BOINC version 7.18.1. This is an Android only version and needs to be replaced. See: https://boinc.berkeley.edu/forum_thread.php?id=14650 |
Send message Joined: 9 Jul 18 Posts: 11 Credit: 604,030 RAC: 0 |
No, same results when I run it. |
Send message Joined: 21 Feb 11 Posts: 59 Credit: 543,728 RAC: 19 |
Did you run "cvmfs_config setup"? |
Send message Joined: 9 Jul 18 Posts: 11 Credit: 604,030 RAC: 0 |
$ sudo cvmfs_config showconfig -s atlas.cern.ch systemctl status autofs.service CVMFS_REPOSITORY_NAME=atlas.cern.ch CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf CVMFS_CACHE_DIR=/var/lib/cvmfs/shared CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_FALLBACK_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_HIDE_MAGIC_XATTRS=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf CVMFS_HTTP_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf CVMFS_MAGIC_XATTRS_VISIBILITY=rootonly # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf CVMFS_SEND_INFO_HEADER=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_SERVER_URL='http://cvmfs-stratum-one.cern.ch:8000/cvmfs/atlas.cern.ch;http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1bnl.opensciencegrid.org:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch;http://cvmfsrep.grid.sinica.edu.tw:8000/cvmfs/atlas.cern.ch;http://cvmfs-stratum-one.ihep.ac.cn:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1.hpc.swin.edu.au:8000/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf |
Send message Joined: 21 Feb 11 Posts: 59 Credit: 543,728 RAC: 19 |
You are supposed to separate commands with && like this: sudo cvmfs_config showconfig -s atlas.cern.ch && systemctl status autofs.service What is in /etc/cvmfs/default.local ? If it is empty run this: sudo wget https://lhcathome.cern.ch/lhcathome/download/default.local -O /etc/cvmfs/default.local |
Send message Joined: 9 Jul 18 Posts: 11 Credit: 604,030 RAC: 0 |
I thought I did before, but now nothing happens. |
Send message Joined: 9 Jul 18 Posts: 11 Credit: 604,030 RAC: 0 |
That worked. # Must be set: the repositories required for ATLAS/Theory # If "CVMFS_DEFAULT_DOMAIN" is set to "cern.ch" like in /etc/cvmfs/default.d/50-cern.conf # simple names will expand to FQRNs, # e.g. atlas -> atlas.cern.ch CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice" # Must be set: to ensure openhtc.io will be used CVMFS_USE_CDN=yes # If a local proxy <my_proxy> is available that listens at port <proxy_port>. # Replace <my_proxy> with either: # - the computer name of your local proxy; requires a well configured local name resolution # - the FQDN of your local proxy; requires a well configured local name resolution # - the IP of your local proxy # Replace <proxy_port> with your proxy's TCP port; Squid's default would be 3128 # Pattern: CVMFS_HTTP_PROXY="http://<my_proxy>:<proxy_port>;DIRECT" # Example: CVMFS_HTTP_PROXY="http://squid:3128;DIRECT" # Only recommended for clusters <5 worker nodes: # uncomment the next line if no local proxy is available. # CVMFS_HTTP_PROXY="auto;DIRECT" # further options CVMFS_KCACHE_TIMEOUT=2 CVMFS_MAX_RETRIES=3 |
Send message Joined: 12 Jun 18 Posts: 126 Credit: 53,906,164 RAC: 26,149 |
I have one computer that fails the atlas probe. I've deleted all cvmfs and reinstalled it several times. My other computers continue to run atlas fine. Can anyone see what's wrong? Linux Mint 20.3 Ubuntu 20.04: aurum@Rig-05:~$ cvmfs_config probe Probing /cvmfs/atlas.cern.ch... Failed! Probing /cvmfs/atlas-condb.cern.ch... OK Probing /cvmfs/grid.cern.ch... OK Probing /cvmfs/cernvm-prod.cern.ch... OK Probing /cvmfs/sft.cern.ch... Failed! Probing /cvmfs/alice.cern.ch... OK aurum@Rig-05:~$ sudo cvmfs_config showconfig -s atlas.cern.ch && systemctl status autofs.service CVMFS_REPOSITORY_NAME=atlas.cern.ch CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf CVMFS_CACHE_DIR=/var/lib/cvmfs/shared CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf CVMFS_CLIENT_PROFILE=single # from /etc/cvmfs/default.conf CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf CVMFS_HTTP_PROXY='auto;DIRECT' # from /etc/cvmfs/default.local CVMFS_KCACHE_TIMEOUT=2 # from /etc/cvmfs/default.local CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf CVMFS_MAX_RETRIES=3 # from /etc/cvmfs/default.local CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf CVMFS_REPOSITORIES=atlas,atlas-condb,grid,cernvm-prod,sft,alice # from /etc/cvmfs/default.local CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf aurum@Rig-05:~$ sudo cvmfs_config chksetup Warning: failed to resolve auto proxy for http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch/.cvmfspublished etc... |
Send message Joined: 15 Jun 08 Posts: 2425 Credit: 227,502,920 RAC: 129,877 |
Your ATLAS logs (on all computers) show that you are running a recent CVMFS version but it appears that the configuration is a mix of older and newer options. You will need to update the cvmfs-config package and rework the configuration as mentioned here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594&postid=48539 Best would be to ensure no ATLAS/Theory task is currently running so you can reboot after the changes are made. In addition the proxy "http://131.225.188.245:3125" reported in the logs also points out a misconfiguration as this is a Squid at Fermilab. Your packets currently travel along a very long route: your LAN <--> Fermilab proxy <--> Cloudflare proxy (close to Fermilab) <--> CVMFS at Fermilab The intended route would be: your LAN (best case: your proxy!) <--> Cloudflare proxy (close to your LAN) <--> CVMFS at Fermilab Since ATLAS makes heavy use of the Frontier service (beside CVMFS) and you are running a large cluster it is recommended that you run a local Squid. Unlike CVMFS Frontier does not have it's own local cache. |
Send message Joined: 21 Feb 11 Posts: 59 Credit: 543,728 RAC: 19 |
For me ping to 131.225.188.245 is average 152ms. |
Send message Joined: 15 Jun 08 Posts: 2425 Credit: 227,502,920 RAC: 129,877 |
For me ping to 131.225.188.245 is average 152ms. The rtt to only one of the nodes in the chain is not relevant. The fact is that if your local CVMFS client reports 131.225.188.245:3125 it points out a misconfiguration. To heal that misconfiguration use the recent packets (client and config) and add the recommended modifications. At the end this makes CVMFS as robust and fast as possible. |
Send message Joined: 12 Jun 18 Posts: 126 Credit: 53,906,164 RAC: 26,149 |
/etc/cvmfs/default.local (to be used without a local proxy): # set quota limit between 4000 and 10000 CVMFS_QUOTA_LIMIT=6000 This already appears in my default.conf file as: # 4GB soft limit for the files in CVMFS_CACHE_BASE CVMFS_QUOTA_LIMIT=4000 Should it be in both files or just one and should the values be the same? Also, should this be the only line? /etc/cvmfs/config.d/cvmfs-config.cern.ch.local # usually this is the only option that should be set in this file CVMFS_CONFIG_REPO_REQUIRED=no Mine currently is this and it passed probe: # In order to bootstrap the network connection, we use a DIRECT proxy # for the config repository only if no other proxy has been set. # CVMFS_HTTP_PROXY="${CVMFS_HTTP_PROXY:=DIRECT}" # Usually this is the only option that should be set in this file CVMFS_CONFIG_REPO_REQUIRED=no |
Send message Joined: 15 Jun 08 Posts: 2425 Credit: 227,502,920 RAC: 129,877 |
If an option appears in multiple config files the last appearence will be used. The order can be found e.g. in /etc/cvmfs/cern.ch.conf. "CVMFS_QUOTA_LIMIT=4000" is set as default in /etc/cvmfs/default.conf. That's the maximum size of the disk cache used by CVMFS. 4 GB typically result in a hitrate around 95 % for ATLAS/Theory native. If you increase this to 6 GB ("CVMFS_QUOTA_LIMIT=6000" in /etc/cvmfs/default.local) you may get a hitrate around 99 %. A higher quota does not significantly increase the hitrate. Lines starting with "#" are comments and will not be used for the configuration. Hence, your /etc/cvmfs/config.d/cvmfs-config.cern.ch.local sets nothing but "CVMFS_CONFIG_REPO_REQUIRED=no" since you posted: Mine currently is this and it passed probe: Looks like you copied the *.conf file to *.local and made the marked line a comment. This should be avoided. Use *.local only to modify options from the corresponding *.conf file or (like here) to set an option for the specific target. |
©2024 CERN