1) Message boards : CMS Application : Is there a Native version in the pipeline? (Message 46898)
Posted 10 days ago by Dark Angel
Post:
You mean one of these?: https://www.amd.com/en/products/cpu/amd-epyc-7742

Dual CPU configuration 128t for each CPU gives 256 threads.
The RAM alone to run it would cost a fortune!
2) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46848)
Posted 24 days ago by Dark Angel
Post:
Not sure what exactly you mean with "these settings".
Hence, I repeat the suggested ones.


Since you are located in Australia and you are using a local Squid the most efficient settings are:
in /etc/cvmfs/default.local
CVMFS_KCACHE_TIMEOUT=2
CVMFS_MAX_RETRIES=3
CVMFS_USE_CDN=yes

in /etc/cvmfs/domain.d/cern.ch.local
CVMFS_SERVER_URL="http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;$CVMFS_SERVER_URL"
CVMFS_USE_GEOAPI=no


All your CVMFS clients should be configured using these options.


"These settings" the ones you just gave me.
3) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46846)
Posted 24 days ago by Dark Angel
Post:
Will these settings improve network performance for the rest of my clients or is this more a diagnostic thing?

Oh and:
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK
4) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46845)
Posted 24 days ago by Dark Angel
Post:
Yes, I'm in Australia

# cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4821 164 27204 19 3 1 1259890 4096000 0 130560 0 16 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/cvmfs-config.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 4911 164 35952 104984 3 1 1259890 4096000 732 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch http://192.168.1.3:3128 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.9.2.0 5469 163 29024 23003 3 1 1259890 4096000 7 130560 0 0 100.000 0 0 http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/sft.cern.ch http://192.168.1.3:3128 1
5) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46843)
Posted 24 days ago by Dark Angel
Post:
Prior to reading this I decided to do a full OS reinstall on the system with the simplest possible configuration of everything. I followed the exact same steps I used before, even copying the default.local file over from a working system exactly as before.
It now appears to be functioning properly.
This annoys me greatly because I've done the same thing with different results.
On the plus side, it's working. All repositories are loading as they should without issue.

Thanks for your help, I just hate that we couldn't get s definitive "this is wrong" answer so that others might find it.
6) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46840)
Posted 25 days ago by Dark Angel
Post:
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html


From the troubleshooting section:

In order to exclude autofs/automounter as a source of problems, you can try to mount repository.cern.ch manually with the following

# mount -t cvmfs atlas.cern.ch /mnt/cvmfs
CernVM-FS: running with credentials 116:118
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
7) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46837)
Posted 25 days ago by Dark Angel
Post:
Yes, I ran that.

After adding that line to default.local and restarting autofs I get:
#cvmfs_config probe
required configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d
Failed to read CernVM-FS configuration
8) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46835)
Posted 25 days ago by Dark Angel
Post:
CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://192.168.1.3:3128;DIRECT' # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,atlas-nightlies.cern.ch,sft.cern.ch,sft-nightlies.cern.ch,unpacked.cern.ch,grid,cernvm-prod,sft,alice # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf
9) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46833)
Posted 25 days ago by Dark Angel
Post:
Ok, the difference between the two:

# cvmfs_config showconfig -s atlas.cern.ch >> 1.txt
# cvmfs_config showconfig -s grid.cern.ch >> 2.txt
# diff 1.txt 2.txt
1c1
< CVMFS_REPOSITORY_NAME=atlas.cern.ch
---
> CVMFS_REPOSITORY_NAME=grid.cern.ch
24c24
< CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
---
> CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/grid.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/grid.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf
10) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46832)
Posted 25 days ago by Dark Angel
Post:
Yes, it's that computer.

Both those reload commands give a "Failed" result.

I've restarted the serve and rebooted the computer multiple times each.

I'm looking at the output of the showconfig commands now and will report if there's any differences.
11) Message boards : Number crunching : CVMFS issue - probably configuration but ... (Message 46830)
Posted 25 days ago by Dark Angel
Post:
Ok, me again. I have an odd problem with a rig I just threw together.
I followed the same installation steps for CVMFS as my last rig, copy and paste kind of "the same", but this one jsut won't behave. The catch is, it's only the Atlas repositories that it refuses to load.
To illustrate:
# cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/sft-nightlies.cern.ch... OK
Probing /cvmfs/unpacked.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

(extra repositories included to illustrate)
It's only the first two that continually fail, everything else loads without issue.
I'm quite capable of messing most things up, but I cannot for the life of me work out what I could have done to achieve this.
I let it run a Theory unit for a while and it accesses the repositories via my Squid proxy without issue, just not those two Atlas ones.
12) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46816)
Posted 24 May 2022 by Dark Angel
Post:
Ok, an update.
I completely disabled the thumbnailer service in the system and the issue hasn't happened since.
At this stage, unless something changes, I suspect that was the problem. The thumbnail creation service was trying to handle the CVMFS directories and getting overloaded.
13) Message boards : CMS Application : Is there a Native version in the pipeline? (Message 46815)
Posted 24 May 2022 by Dark Angel
Post:
That's it, straight question. Is there a native version in development/under consideration/even possible for CMS?
14) Message boards : ATLAS application : ATLAS badges (Message 46807)
Posted 20 May 2022 by Dark Angel
Post:
Reviving an old thread but it is on topic

Id there any chance of adding further badges at the higher end of the scale? There are people with achievements far in excess of the maximum 5 million points badge.
15) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46762)
Posted 9 May 2022 by Dark Angel
Post:
Nope, it did it again.
Nothing in the error logs, just the file manager and trash applet locked up hard.
Filesystem access from the terminal is normal.
System has all updates installed.
Everything normal again after reboot.
16) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46759)
Posted 8 May 2022 by Dark Angel
Post:
I have no idea if this is relevant, but it was worth a try.
I noticed that on my two Mint rigs both were using the Bionic cern repositories rather than the Focal which the rest of the system, and my two Ubuntu Server headless rigs, use.
I have changed the settings so it is consistent, updated cvmfs and fuse2, and rebooted. I guess I'll see if that makes things better, worse, or has no effect at all.
17) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46758)
Posted 7 May 2022 by Dark Angel
Post:
It did it again tonight. Places applet/file manager and recycle bin/Trash applet both locked up solid. No errors in logs beside the same things. Only ever does it running this.

Those commands return the correct outputs now, but I did forget to run them while the applets were locked up.
18) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46753)
Posted 5 May 2022 by Dark Angel
Post:
I find a lot of stuff like this:

May 5 17:00:16 Zen cvmfs2: (sft-nightlies.cern.ch) switching host from http://s1ihep-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch to http://s1unl-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch (host returned HTTP error)
May 5 17:00:17 Zen cvmfs2: (sft-nightlies.cern.ch) switching host from http://s1unl-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch to http://s1fnal-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch (host returned HTTP error)
May 5 17:00:17 Zen cvmfs2: (sft-nightlies.cern.ch) switching host from http://s1fnal-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch to http://s1bnl-cvmfs.openhtc.io/cvmfs/sft-nightlies.cern.ch (host returned HTTP error)

and a few from the Tracker-Miner indexing service (which I've now uninstalled)

I'll wait and see if it happens again.
19) Message boards : ATLAS application : Atlas apparently affecting File Manager operation (Message 46750)
Posted 5 May 2022 by Dark Angel
Post:
It doesn't help that it's intermittent as well. The file manager (the application is Thunar but I access it using the XFCE Places toolbar applet) works fine most of the time, but after a period of time, which could be several days, it starts to slow down until it stops working. Sometimes I only notice it when the trash applet stops working.
20) Message boards : ATLAS application : Singularity errors (Message 46746)
Posted 5 May 2022 by Dark Angel
Post:
Ok, further to this issue: I seem to have it working again.

In the end I had to completely wipe any trace of singularity, Apptainer, and cvmfs from the system, deleting all files and folders from the system, reboot, the reinstall cvmfs from scratch.
At this point it had a massive hissy fit regarding missing folders and keys, and for some reason refused to populate the /etc/cvmfs directory structure, so I copied it whole from a working system, changed the owner to cvmfs:cvmfs, restarted autofs, and finally got cvmfs_config probe to return something usable.
Then I installed Apptainer and it appears to be working normally again.


Next 20


©2022 CERN