Message boards : Number crunching : CVMFS issue - probably configuration but ...
Message board moderation

To post messages, you must log in.

AuthorMessage
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46830 - Posted: 31 May 2022, 6:46:32 UTC

Ok, me again. I have an odd problem with a rig I just threw together.
I followed the same installation steps for CVMFS as my last rig, copy and paste kind of "the same", but this one jsut won't behave. The catch is, it's only the Atlas repositories that it refuses to load.
To illustrate:
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

(extra repositories included to illustrate)
It's only the first two that continually fail, everything else loads without issue.
I'm quite capable of messing most things up, but I cannot for the life of me work out what I could have done to achieve this.
I let it run a Theory unit for a while and it accesses the repositories via my Squid proxy without issue, just not those two Atlas ones.
ID: 46830 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46831 - Posted: 31 May 2022, 8:20:34 UTC - in response to Message 46830.  

Is it this computer?
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10800403

Did you reload the affected repositories?
cvmfs_config reload atlas.cern.ch
cvmfs_config reload atlas-condb.cern.ch


Did you restart the automounter service (or reboot)?

If this doesn't solve the problem you may compare the CVMFS settings of atlas.cern.ch and grid.cern.ch:
cvmfs_config showconfig -s atlas.cern.ch
cvmfs_config showconfig -s grid.cern.ch
ID: 46831 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46832 - Posted: 31 May 2022, 8:25:31 UTC - in response to Message 46831.  

Yes, it's that computer.

Both those reload commands give a "Failed" result.

I've restarted the serve and rebooted the computer multiple times each.

I'm looking at the output of the showconfig commands now and will report if there's any differences.
ID: 46832 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46833 - Posted: 31 May 2022, 8:29:16 UTC

Ok, the difference between the two:

# cvmfs_config showconfig -s atlas.cern.ch >> 1.txt
# cvmfs_config showconfig -s grid.cern.ch >> 2.txt
# diff 1.txt 2.txt
1c1
< CVMFS_REPOSITORY_NAME=atlas.cern.ch
---
> CVMFS_REPOSITORY_NAME=grid.cern.ch
24c24
< CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
---
> CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/grid.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
ID: 46833 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46834 - Posted: 31 May 2022, 8:52:13 UTC - in response to Message 46833.  

Looks like a wrong server list (but why is not yet clear).

Could you post the complete output of:
cvmfs_config showconfig -s atlas.cern.ch
ID: 46834 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46835 - Posted: 31 May 2022, 8:54:00 UTC - in response to Message 46834.  

CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://192.168.1.3:3128;DIRECT' # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,sft-nightlies.cern.ch,unpacked.cern.ch,grid,cernvm-prod,sft,alice # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf
ID: 46835 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46836 - Posted: 31 May 2022, 9:51:53 UTC - in response to Message 46835.  

Did you run "cvmfs_config setup" once after installation?

I'm missing the config repository "cvmfs-config.cern.ch" which is set but not mounted.
You may try to force it using this line in /etc/cvmfs/default.local:
CVMFS_CONFIG_REPO_REQUIRED=yes



As a result a couple of other options should be loaded from the config repo, e.g. an up to date server list or:
CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf

(Don't set this option in your local config files. It just indicates that CVMFS has access to the web repo)
ID: 46836 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46837 - Posted: 31 May 2022, 11:53:18 UTC - in response to Message 46836.  

Yes, I ran that.

After adding that line to default.local and restarting autofs I get:
#cvmfs_config probe
required configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d
Failed to read CernVM-FS configuration
ID: 46837 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,091,089
RAC: 103,567
Message 46839 - Posted: 31 May 2022, 17:13:41 UTC - in response to Message 46837.  

ID: 46839 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46840 - Posted: 31 May 2022, 21:02:11 UTC - in response to Message 46839.  

https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html


From the troubleshooting section:

In order to exclude autofs/automounter as a source of problems, you can try to mount repository.cern.ch manually with the following

# mount -t cvmfs atlas.cern.ch /mnt/cvmfs
CernVM-FS: running with credentials 116:118
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
ID: 46840 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46842 - Posted: 1 Jun 2022, 4:35:48 UTC - in response to Message 46840.  

From the troubleshooting section:
In order to exclude a corrupted local cache as a source of problems, run
cvmfs_config wipecache




In addition, remove this line (suggested in my previous post):
CVMFS_CONFIG_REPO_REQUIRED=yes

Instead add "cvmfs-config.cern.ch" as 1st entry to the repository list in "/etc/cvmfs/default.local".
=> run "cvmfs_config reload"
ID: 46842 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46843 - Posted: 1 Jun 2022, 5:52:24 UTC

Prior to reading this I decided to do a full OS reinstall on the system with the simplest possible configuration of everything. I followed the exact same steps I used before, even copying the default.local file over from a working system exactly as before.
It now appears to be functioning properly.
This annoys me greatly because I've done the same thing with different results.
On the plus side, it's working. All repositories are loading as they should without issue.

Thanks for your help, I just hate that we couldn't get s definitive "this is wrong" answer so that others might find it.
ID: 46843 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46844 - Posted: 1 Jun 2022, 7:01:54 UTC - in response to Message 46843.  
Last modified: 1 Jun 2022, 7:11:16 UTC

Good to know it's working.

Your computers are located in Australia, right?
In connection with the fact that you are using a local Squid you may test the following settings:
in /etc/cvmfs/default.local
CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3

in /etc/cvmfs/domain.d/cern.ch.local
CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no


Then run "cvmfs_config reload" and post the output of:
cvmfs_config stat




There's a "new" CVMFS server alias "s1swinburne-cvmfs.openhtc.io" connecting to the datacenter here:
https://en.wikipedia.org/wiki/Swinburne_University_of_Technology

<edit>
Sorry, must be:
CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no

</edit>
ID: 46844 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46845 - Posted: 1 Jun 2022, 8:45:00 UTC - in response to Message 46844.  
Last modified: 1 Jun 2022, 8:45:25 UTC

Yes, I'm in Australia

# cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4821 164 27204 19 3 1 1259890 4096000 0 130560 0 16 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/cvmfs-config.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4911 164 35952 104984 3 1 1259890 4096000 732 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 5469 163 29024 23003 3 1 1259890 4096000 7 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/sft.cern.ch http://192.168.1.3:3128 1
ID: 46845 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46846 - Posted: 1 Jun 2022, 8:46:11 UTC - in response to Message 46844.  
Last modified: 1 Jun 2022, 8:47:06 UTC

Will these settings improve network performance for the rest of my clients or is this more a diagnostic thing?

Oh and:
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK
ID: 46846 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,922,319
RAC: 138,016
Message 46847 - Posted: 1 Jun 2022, 10:28:56 UTC - in response to Message 46846.  

Not sure what exactly you mean with "these settings".
Hence, I repeat the suggested ones.


Since you are located in Australia and you are using a local Squid the most efficient settings are:
in /etc/cvmfs/default.local
CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3
CVMFS_USE_CDN=yes

in /etc/cvmfs/domain.d/cern.ch.local
CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no


All your CVMFS clients should be configured using these options.
ID: 46847 · Report as offensive     Reply Quote
Dark Angel
Avatar

Send message
Joined: 7 Aug 11
Posts: 62
Credit: 21,011,653
RAC: 8,819
Message 46848 - Posted: 1 Jun 2022, 21:55:52 UTC - in response to Message 46847.  

Not sure what exactly you mean with "these settings".
Hence, I repeat the suggested ones.


Since you are located in Australia and you are using a local Squid the most efficient settings are:
in /etc/cvmfs/default.local
CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3
CVMFS_USE_CDN=yes

in /etc/cvmfs/domain.d/cern.ch.local
CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no


All your CVMFS clients should be configured using these options.


"These settings" the ones you just gave me.
ID: 46848 · Report as offensive     Reply Quote

Message boards : Number crunching : CVMFS issue - probably configuration but ...


©2024 CERN