LHC@home

Atlas WU failed after about 10 minutes with the following error.

<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
22:30:59 (268507): wrapper (7.7.26015): starting
22:30:59 (268507): wrapper: running run_atlas (--nthreads 6)
[2023-04-13 22:30:59] Arguments: --nthreads 6
[2023-04-13 22:30:59] Threads: 6
[2023-04-13 22:30:59] Checking for CVMFS
[2023-04-13 22:31:01] Probing /cvmfs/atlas.cern.ch... Failed!
[2023-04-13 22:31:01] Probing /cvmfs/atlas-condb.cern.ch... Failed!
[2023-04-13 22:31:01] cvmfs_config probe atlas.cern.ch atlas-condb.cern.ch failed, aborting the job
22:41:01 (268507): run_atlas exited; CPU time 0.076063
22:41:01 (268507): app exit status: 0x1
22:41:01 (268507): called boinc_finish(195)

</stderr_txt>
]]>

I did run sudo cvmfs_config chksetup before starting with a return of "OK" so I thought I was good. Any idea what I'm missing? Running Linux Ubuntu 22.04.2 LTS [5.19.0-32-generic|libc 2.35] with VB 7.0.6-155176

Does cvmfs_config probe work?

Is it your first try to use CVMFS?
If so, did you follow the advices given here?

https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5595

Run "[sudo] cvmfs_config probe" before you try another ATLAS task.
As long as the probes fail your ATLAS tasks will all fail.

If you can't get it running, post the output of:
[sudo] cvmfs_config showconfig -s atlas.cern.ch
systemctl status autofs.service



BTW:
Your computer details page shows you are running BOINC version 7.18.1.
This is an Android only version and needs to be replaced.
See:
https://boinc.berkeley.edu/forum_thread.php?id=14650

No, same results when I run it.

Did you run "cvmfs_config setup"?

$ sudo cvmfs_config showconfig -s atlas.cern.ch systemctl status autofs.service
CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_FALLBACK_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_HIDE_MAGIC_XATTRS=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAGIC_XATTRS_VISIBILITY=rootonly # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_SEND_INFO_HEADER=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SERVER_URL='http://cvmfs-stratum-one.cern.ch:8000/cvmfs/atlas.cern.ch;http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1bnl.opensciencegrid.org:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch;http://cvmfsrep.grid.sinica.edu.tw:8000/cvmfs/atlas.cern.ch;http://cvmfs-stratum-one.ihep.ac.cn:8000/cvmfs/atlas.cern.ch;http://cvmfs-s1.hpc.swin.edu.au:8000/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf

You are supposed to separate commands with && like this: sudo cvmfs_config showconfig -s atlas.cern.ch && systemctl status autofs.service
What is in /etc/cvmfs/default.local ?
If it is empty run this:
sudo wget https://lhcathome.cern.ch/lhcathome/download/default.local -O /etc/cvmfs/default.local

I thought I did before, but now nothing happens.

That worked.

# Must be set: the repositories required for ATLAS/Theory
# If "CVMFS_DEFAULT_DOMAIN" is set to "cern.ch" like in /etc/cvmfs/default.d/50-cern.conf
# simple names will expand to FQRNs,
# e.g. atlas -> atlas.cern.ch
CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice"

# Must be set: to ensure openhtc.io will be used
CVMFS_USE_CDN=yes

# If a local proxy <my_proxy> is available that listens at port <proxy_port>.
# Replace <my_proxy> with either:
# - the computer name of your local proxy; requires a well configured local name resolution
# - the FQDN of your local proxy; requires a well configured local name resolution
# - the IP of your local proxy
# Replace <proxy_port> with your proxy's TCP port; Squid's default would be 3128
# Pattern: CVMFS_HTTP_PROXY="http://<my_proxy>:<proxy_port>;DIRECT"
# Example: CVMFS_HTTP_PROXY="http://squid:3128;DIRECT"

# Only recommended for clusters <5 worker nodes:
# uncomment the next line if no local proxy is available.
# CVMFS_HTTP_PROXY="auto;DIRECT"

# further options
CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3

I have one computer that fails the atlas probe. I've deleted all cvmfs and reinstalled it several times. My other computers continue to run atlas fine. Can anyone see what's wrong? Linux Mint 20.3 Ubuntu 20.04:
aurum@Rig-05:~$ cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... Failed!
Probing /cvmfs/alice.cern.ch... OK
aurum@Rig-05:~$ sudo cvmfs_config showconfig -s atlas.cern.ch && systemctl status autofs.service
CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=single # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='auto;DIRECT' # from /etc/cvmfs/default.local
CVMFS_KCACHE_TIMEOUT=2 # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=3 # from /etc/cvmfs/default.local
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas,atlas-condb,grid,cernvm-prod,sft,alice # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf
aurum@Rig-05:~$ sudo cvmfs_config chksetup
Warning: failed to resolve auto proxy for http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch/.cvmfspublished
etc...

Your ATLAS logs (on all computers) show that you are running a recent CVMFS version but it appears that the configuration is a mix of older and newer options.
You will need to update the cvmfs-config package and rework the configuration as mentioned here:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594&postid=48539

Best would be to ensure no ATLAS/Theory task is currently running so you can reboot after the changes are made.


In addition the proxy "http://131.225.188.245:3125" reported in the logs also points out a misconfiguration as this is a Squid at Fermilab.
Your packets currently travel along a very long route:
your LAN <--> Fermilab proxy <--> Cloudflare proxy (close to Fermilab) <--> CVMFS at Fermilab

The intended route would be:
your LAN (best case: your proxy!) <--> Cloudflare proxy (close to your LAN) <--> CVMFS at Fermilab


Since ATLAS makes heavy use of the Frontier service (beside CVMFS) and you are running a large cluster it is recommended that you run a local Squid.
Unlike CVMFS Frontier does not have it's own local cache.

For me ping to 131.225.188.245 is average 152ms.

For me ping to 131.225.188.245 is average 152ms.

The rtt to only one of the nodes in the chain is not relevant.
The fact is that if your local CVMFS client reports 131.225.188.245:3125 it points out a misconfiguration.
To heal that misconfiguration use the recent packets (client and config) and add the recommended modifications.

At the end this makes CVMFS as robust and fast as possible.

/etc/cvmfs/default.local (to be used without a local proxy):
# set quota limit between 4000 and 10000
CVMFS_QUOTA_LIMIT=6000

This already appears in my default.conf file as:
# 4GB soft limit for the files in CVMFS_CACHE_BASE
CVMFS_QUOTA_LIMIT=4000

Should it be in both files or just one and should the values be the same?

Also, should this be the only line?
/etc/cvmfs/config.d/cvmfs-config.cern.ch.local
# usually this is the only option that should be set in this file
CVMFS_CONFIG_REPO_REQUIRED=no

Mine currently is this and it passed probe:
# In order to bootstrap the network connection, we use a DIRECT proxy
# for the config repository only if no other proxy has been set.
# CVMFS_HTTP_PROXY="${CVMFS_HTTP_PROXY:=DIRECT}"

# Usually this is the only option that should be set in this file
CVMFS_CONFIG_REPO_REQUIRED=no

If an option appears in multiple config files the last appearence will be used.
The order can be found e.g. in /etc/cvmfs/cern.ch.conf.

"CVMFS_QUOTA_LIMIT=4000" is set as default in /etc/cvmfs/default.conf.
That's the maximum size of the disk cache used by CVMFS.
4 GB typically result in a hitrate around 95 % for ATLAS/Theory native.
If you increase this to 6 GB ("CVMFS_QUOTA_LIMIT=6000" in /etc/cvmfs/default.local) you may get a hitrate around 99 %.
A higher quota does not significantly increase the hitrate.


Lines starting with "#" are comments and will not be used for the configuration.

Hence, your /etc/cvmfs/config.d/cvmfs-config.cern.ch.local sets nothing but "CVMFS_CONFIG_REPO_REQUIRED=no" since you posted:

Mine currently is this and it passed probe:
# In order to bootstrap the network connection, we use a DIRECT proxy
# for the config repository only if no other proxy has been set.
# CVMFS_HTTP_PROXY="${CVMFS_HTTP_PROXY:=DIRECT}"

# Usually this is the only option that should be set in this file
CVMFS_CONFIG_REPO_REQUIRED=no

Looks like you copied the *.conf file to *.local and made the marked line a comment.
This should be avoided.
Use *.local only to modify options from the corresponding *.conf file or (like here) to set an option for the specific target.