LHC@home

[2021-04-25 03:37:21] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
[2021-04-25 03:37:21] Checking for singularity binary...
[2021-04-25 03:37:21] Singularity is not installed, using version from CVMFS
[2021-04-25 03:37:21] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
[2021-04-25 03:37:24] INFO: Converting SIF file to temporary sandbox... Rig-48 INFO: Cleaning up image...
[2021-04-25 03:37:24] Singularity works
[2021-04-25 03:37:24] Starting ATLAS job with PandaID=5037675747
[2021-04-25 03:37:24] Running command: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec --pwd /var/lib/boinc-client/slots/2 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh
[2021-04-25 03:37:26] Job failed
[2021-04-25 03:37:26] INFO:    Converting SIF file to temporary sandbox...
[2021-04-25 03:37:26] INFO:    Cleaning up image...
[2021-04-25 03:37:26] FATAL:   container creation failed: hook function for tag prelayer returns error: failed to create /var/lib/alternatives directory: mkdir /var/lib/alternatives: read-only file system
[2021-04-25 03:37:26] ./runtime_log.err
[2021-04-25 03:37:26] ./runtime_log
03:47:26 (4413): run_atlas exited; CPU time 17.180037
03:47:26 (4413): app exit status: 0x1
03:47:26 (4413): called boinc_finish(195)

root@Rig-48:/usr/local/bin/singularity# ./mconfig && \
>     make -C builddir && \
>     make -C builddir install
Configuring for project `singularity' with languages: C, Golang
=> running pre-basechecks project specific checks ...
=> running base system checks ...
 checking: host C compiler... cc
 checking: host C++ compiler... c++
 checking: host Go compiler (at least version 1.13)... not found!
mconfig: could not complete configuration

aurum@Rig-48:~$ go version
go version go1.16.3 linux/amd64

what do you get when you run "singularity --version"?

$ singularity --version
Command 'singularity' not found, but can be installed with:
sudo apt install singularity

[2021-04-24 13:21:36] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname

cvmfs_config stat
/usr/bin/cvmfs_config: line 907: cd: /cvmfs/atlas.cern.ch: Transport endpoint is not connected

<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
09:59:55 (52264): wrapper (7.7.26015): starting
09:59:55 (52264): wrapper: running run_atlas (--nthreads 1)
[2021-04-24 09:59:55] Arguments: --nthreads 1
[2021-04-24 09:59:55] Threads: 1
[2021-04-24 09:59:55] Checking for CVMFS
[2021-04-24 09:59:58] Probing /cvmfs/atlas.cern.ch... OK
[2021-04-24 10:00:00] Probing /cvmfs/atlas-condb.cern.ch... OK
[2021-04-24 10:00:01] Probing /cvmfs/grid.cern.ch... OK
[2021-04-24 10:00:02] Probing /cvmfs/cernvm-prod.cern.ch... OK
[2021-04-24 10:00:02] Probing /cvmfs/grid.cern.ch... OK
[2021-04-24 10:00:03] Probing /cvmfs/sft.cern.ch... OK
[2021-04-24 10:00:04] Probing /cvmfs/alice.cern.ch... OK
[2021-04-24 10:00:05] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2021-04-24 10:00:05] 2.8.1.0 52564 0 24632 83323 3 1 473981 4194304 0 130560 0 0 100.000 0 0 http://s1unl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1
[2021-04-24 10:00:05] CVMFS is ok
[2021-04-24 10:00:05] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
[2021-04-24 10:00:05] Checking for singularity binary...
[2021-04-24 10:00:05] Singularity is not installed, using version from CVMFS
[2021-04-24 10:00:05] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
[2021-04-24 10:00:10] INFO: Converting SIF file to temporary sandbox... Rig-46 INFO: Cleaning up image...
[2021-04-24 10:00:10] Singularity works
[2021-04-24 10:00:10] Starting ATLAS job with PandaID=5035383237
[2021-04-24 10:00:10] Running command: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec --pwd /var/lib/boinc-client/slots/35 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh
[2021-04-24 10:00:17] Job failed
[2021-04-24 10:00:17] INFO:    Converting SIF file to temporary sandbox...
[2021-04-24 10:00:17] INFO:    Cleaning up image...
[2021-04-24 10:00:17] FATAL:   container creation failed: hook function for tag prelayer returns error: failed to create /var/lib/alternatives directory: mkdir /var/lib/alternatives: read-only file system
[2021-04-24 10:00:17] ./runtime_log
[2021-04-24 10:00:17] ./runtime_log.err
10:10:17 (52264): run_atlas exited; CPU time 11.489694
10:10:17 (52264): app exit status: 0x1
10:10:17 (52264): called boinc_finish(195)
</stderr_txt>

[2021-04-24 09:34:41] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
[2021-04-24 09:34:41] Checking for singularity binary...
[2021-04-24 09:34:41] Singularity is not installed, using version from CVMFS
[2021-04-24 09:34:41] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
[2021-04-24 09:34:42] Singularity isnt working: INFO:    Converting SIF file to temporary sandbox...
[2021-04-24 09:34:42] FATAL:   while extracting /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img: root filesystem extraction failed: extract command failed: WARNING: passwd file doesn't exist in container, not updating
[2021-04-24 09:34:42] WARNING: group file doesn't exist in container, not updating
[2021-04-24 09:34:42] WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
[2021-04-24 09:34:42] WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
[2021-04-24 09:34:42] WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
[2021-04-24 09:34:42] WARNING: Skipping mount /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.7.2/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
[2021-04-24 09:34:42] WARNING: Skipping mount /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.7.2/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
[2021-04-24 09:34:42] WARNING: Skipping mount /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.7.2/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
[2021-04-24 09:34:42] 
[2021-04-24 09:34:42] FATAL ERROR:write_file: failed to create file /image/root/usr/include/c++/4.8.2/ext/pb_ds/detail/cc_hash_table_map_/erase_fn_imps.hpp, because Too many open files
[2021-04-24 09:34:42] Parallel unsquashfs: Using 36 processors
[2021-04-24 09:34:42] 41269 inodes (41434 blocks) to write
[2021-04-24 09:34:42] 
[2021-04-24 09:34:42] : exit status 1
09:44:43 (47648): run_atlas exited; CPU time 1.082061
09:44:43 (47648): app exit status: 0x1
09:44:43 (47648): called boinc_finish(195)
</stderr_txt>

The feedback I got said there was a configuration change related to setuid in the latest singularity, and he pointed me to this page.

openSUSE and SUSE have a small difference with upstream default. 
This means the SUID root binaries distributed by singularity are executable only by users belonging to the group 'singularity'.
Otherwise, users will get an error message like this one:
FATAL:   while executing /usr/lib/singularity/bin/starter-suid: permission denied
To add a user to the group singularity, execute (as root):
 # usermod -a -G singularity <user_login>

Regarding the cvmfs 18.04 packages: they are selected by the cvmfs-release package, which is responsible for picking a suitable package repository. It uses

lsb-release

to figure out where it runs. Officially, only the Ubuntu LTS releases are supported. So, "focal" is a recognized release but "ulyssa" is not. For unsupported distributions, the cvmfs-release tries to fall back to a package repository with compatible binaries. In this case, it picks the repository with the 18.04 binaries. While somewhat less than perfect, these binaries should nevertheless work fine on Mint 20, so there is nothing to worry about.

Yes, unfortunately the CERN world is rather CentOS-centric and so other Linux flavours like Ubuntu don't work out of the box.
So the options for running ATLAS WU are (in rough order of increasing difficulty):
- Run the Vbox app
- Run a pre-made Docker container with no configuration required
- Set up a CentOS VM and run BOINC inside
- Install (compile if necessary) CVMFS and Singularity to run real native tasks
- Wipe Ubuntu and install CentOS 7 (just a joke :)

wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb ; \
 sudo dpkg -i cvmfs-release-latest_all.deb ; \
 rm -f cvmfs-release-latest_all.deb ; \
 sudo apt-get update ; \
 sudo apt-get install cvmfs ; \
 sudo apt install glibc-doc open-iscsi watchdog

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/default.local -O /etc/cvmfs/default.local ; \
 sudo cvmfs_config setup ; \
 sudo echo "/cvmfs /etc/auto.cvmfs" > /etc/auto.master.d/cvmfs.autofs ; \
 sudo systemctl restart autofs ; \
 cvmfs_config probe

sudo cvmfs_config reload

aurum@Rig-25:~$ dpkg -l |grep -i cvmfs
ii  cvmfs                                      2.8.0~1+ubuntu18.04                   amd64        CernVM File System
ii  cvmfs-config-default                       2.0-1                                 all          CernVM File System Default Configuration
ii  cvmfs-release                              3.2-1                                 all          This package contains the apt configuration for the CernVM File System packages.

root@Rig-25:/home/aurum/singularity# ./mconfig && \
>     make -C builddir && \
>     sudo make -C builddir install
Configuring for project `singularity' with languages: C, Golang
=> running pre-basechecks project specific checks ...
=> running base system checks ...
 checking: host C compiler... cc
 checking: host C++ compiler... c++
 checking: host Go compiler (at least version 1.13)... not found!
mconfig: could not complete configuration

Setting up cvmfs-release (3.2-1) ...
Warning: this distribution is not supported. Using Ubuntu 18.04 packages as fallback.

Docker Container
The CernVM-FS service container can expose the /cvmfs directory tree to the host. Import the container with
docker pull cvmfs/service

sudo apt install docker.io -y
sudo docker pull cvmfs/service

Running Theory (or ATLAS) native still requires lots of expert knownlege, additional settings and more babysitting.
They pay back more efficient tasks and less total RAM requirements, especially if many task run concurrently on a computer with many cores.

Volunteers who don't want to spend that additional work should run the vbox apps.

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
# see ACL definition above
# Examples:
# http_access allow crunchers
# http_access allow localnet
http_access allow localhost
http_access deny all
http_access allow crunch

# http_port
# Don't bind it to an IP accessible from outside unless you know what you're doing. E.g.,
http_port 192.168.1.227:3128

CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice"
CVMFS_SEND_INFO_HEADER=yes
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY="http://192.168.1.227:3128"

<dont_use_vbox>1</dont_use_vbox>
<vbox_window>0</vbox_window>

In addition your local squid has to be set in your BOINC client as this is used to configure all LHC vbox tasks as well as ATLAS (native)'s Frontier client.