LHC@home

Ok, me again. I have an odd problem with a rig I just threw together.
I followed the same installation steps for CVMFS as my last rig, copy and paste kind of "the same", but this one jsut won't behave. The catch is, it's only the Atlas repositories that it refuses to load.
To illustrate:
[CODE]
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK
[/CODE]
(extra repositories included to illustrate)
It's only the first two that continually fail, everything else loads without issue.
I'm quite capable of messing most things up, but I cannot for the life of me work out what I could have done to achieve this.
I let it run a Theory unit for a while and it accesses the repositories via my Squid proxy without issue, just not those two Atlas ones.

Is it this computer?
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10800403

Did you reload the affected repositories?

cvmfs_config reload atlas.cern.ch
cvmfs_config reload atlas-condb.cern.ch

Did you restart the automounter service (or reboot)?

If this doesn't solve the problem you may compare the CVMFS settings of atlas.cern.ch and grid.cern.ch:

cvmfs_config showconfig -s atlas.cern.ch
cvmfs_config showconfig -s grid.cern.ch

Yes, it's that computer.

Both those reload commands give a "Failed" result.

I've restarted the serve and rebooted the computer multiple times each.

I'm looking at the output of the showconfig commands now and will report if there's any differences.

Ok, the difference between the two:

# cvmfs_config showconfig -s atlas.cern.ch >> 1.txt
# cvmfs_config showconfig -s grid.cern.ch >> 2.txt
# diff 1.txt 2.txt
1c1
< CVMFS_REPOSITORY_NAME=atlas.cern.ch
---
> CVMFS_REPOSITORY_NAME=grid.cern.ch
24c24
< CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
---
> CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/grid.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf

Looks like a wrong server list (but why is not yet clear).

Could you post the complete output of:

cvmfs_config showconfig -s atlas.cern.ch

CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://192.168.1.3:3128;DIRECT' # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,sft-nightlies.cern.ch,unpacked.cern.ch,grid,cernvm-prod,sft,alice # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf

Did you run "cvmfs_config setup" once after installation?

I'm missing the config repository "cvmfs-config.cern.ch" which is set but not mounted.
You may try to force it using this line in /etc/cvmfs/default.local:

CVMFS_CONFIG_REPO_REQUIRED=yes

As a result a couple of other options should be loaded from the config repo, e.g. an up to date server list or:

CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf

(Don't set this option in your local config files. It just indicates that CVMFS has access to the web repo)

Yes, I ran that.

After adding that line to default.local and restarting autofs I get:
[CODE]
#cvmfs_config probe
required configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d
Failed to read CernVM-FS configuration
[/CODE]

https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html

From the troubleshooting section:

In order to exclude autofs/automounter as a source of problems, you can try to mount repository.cern.ch manually with the following

# mount -t cvmfs atlas.cern.ch /mnt/cvmfs
CernVM-FS: running with credentials 116:118
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)

From the troubleshooting section:
In order to exclude a corrupted local cache as a source of problems, run

cvmfs_config wipecache

In addition, remove this line (suggested in my previous post):
CVMFS_CONFIG_REPO_REQUIRED=yes

Instead add "cvmfs-config.cern.ch" as 1st entry to the repository list in "/etc/cvmfs/default.local".
=> run "cvmfs_config reload"

Prior to reading this I decided to do a full OS reinstall on the system with the simplest possible configuration of everything. I followed the exact same steps I used before, even copying the default.local file over from a working system exactly as before.
It now appears to be functioning properly.
This annoys me greatly because I've done the same thing with different results.
On the plus side, it's working. All repositories are loading as they should without issue.

Thanks for your help, I just hate that we couldn't get s definitive "this is wrong" answer so that others might find it.

Good to know it's working.

Your computers are located in Australia, right?
In connection with the fact that you are using a local Squid you may test the following settings:
in /etc/cvmfs/default.local

CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3

in /etc/cvmfs/domain.d/cern.ch.local

CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no

Then run "cvmfs_config reload" and post the output of:
cvmfs_config stat




There's a "new" CVMFS server alias "s1swinburne-cvmfs.openhtc.io" connecting to the datacenter here:
https://en.wikipedia.org/wiki/Swinburne_University_of_Technology

<edit>
Sorry, must be:

CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no

</edit>

Yes, I'm in Australia

# cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4821 164 27204 19 3 1 1259890 4096000 0 130560 0 16 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/cvmfs-config.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4911 164 35952 104984 3 1 1259890 4096000 732 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 5469 163 29024 23003 3 1 1259890 4096000 7 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/sft.cern.ch http://192.168.1.3:3128 1

Will these settings improve network performance for the rest of my clients or is this more a diagnostic thing?

Oh and:
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

Not sure what exactly you mean with "these settings".
Hence, I repeat the suggested ones.


Since you are located in Australia and you are using a local Squid the most efficient settings are:
in /etc/cvmfs/default.local

CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3
CVMFS_USE_CDN=yes

in /etc/cvmfs/domain.d/cern.ch.local

CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no

All your CVMFS clients should be configured using these options.

Not sure what exactly you mean with "these settings".
Hence, I repeat the suggested ones.


Since you are located in Australia and you are using a local Squid the most efficient settings are:
in /etc/cvmfs/default.local

CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3
CVMFS_USE_CDN=yes

in /etc/cvmfs/domain.d/cern.ch.local

CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no

All your CVMFS clients should be configured using these options.

"These settings" the ones you just gave me.