LHC@home

It used to work, years ago. It currently works on three machines running Proxmox (Debian derivative).

It fails whether or not I have apptainer installed locally.

> sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

> sudo cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1004219 406 33564 41 1 1 5410278 10240001 0 130560 1 517 98.263 156 0 http://s1fnal-cvmfs.openhtc.io/cvmfs/cvmfs-config.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1007193 400 43204 150158 1 17 5410278 10240001 0 130560 0 104775 96.468 170480 693 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat atlas-condb.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1007343 400 32168 15419 1 1 5410278 10240001 0 130560 0 14 70.588 9513 1506 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/atlas-condb.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat cernvm-prod.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1007489 399 69056 276 2 6 5410278 10240001 0 130560 1 0 45.455 71748 150 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/cernvm-prod.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat grid.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1007637 399 66816 28125 3 49 5410278 10240001 0 130560 1 0 48.980 86338 179 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/grid.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1007762 399 32520 33181 0 1 5410278 10240001 0 130560 0 7 66.667 34 4 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/sft.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat alice.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.13.2.0 1012935 397 1071100 23176 1 2360 5410278 10240001 0 523776 0 0 62.367 2254522 2593 http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/alice.cern.ch DIRECT 1

You can see an example of a failure here: https://lhcathome.cern.ch/lhcathome/result.php?resultid=426164808

I've basically exhausted my attempts at troubleshooting things locally. CVMFS is installed and appears properly configured, it fails the same way whether or not I have apptainer installed, and I've tried fiddling with overriding various settings via

systemctl edit boinc-client.service

. If I understood the steps the program was taking better, I might be able to run apptainer correctly manually to see if that gets me any better output.

Let me know if there are any troubleshooting steps I'm missing.

Just on this:

I might be able to run apptainer correctly manually to see if...

I've not had to do this since Apptainer was still Singularity, but: you can manually run the same steps as the script: it basically just does a

apptainer <apptainer-container> hostname

which should return the system hostname. Make sure to do this as the same user that runs boinc.

From the log you linked:

[2025-09-01 12:15:16] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-09-01 12:15:17] HolyPie
[2025-09-01 12:15:17] apptainer works

so apptainer per se is unlikely to be the problem.

Hth

Henry

---