LHC@home

Haven't been here in a while. The instructions are far more complex and confusing than they were when I last ran ATLAS. Now every WU fails with computation error even though I passed cvmfs config probe.
When setting it up I get:

Setting up cvmfs-release (3.2-1) ...
Warning: this distribution is not supported. Using Ubuntu 18.04 packages as fallback.

I'm running Linux Mint 20.1 Ulyssa base: Ubuntu 20.04 focal. Is this my problem that I have Ubuntu 20.04 ???

This is called a "native application" but it doesn't conform to the definition:
"Something running on a computer natively means that it is running without any external layer requiring fewer software layers."

In the instructions it says:

Docker Container
The CernVM-FS service container can expose the /cvmfs directory tree to the host. Import the container with
docker pull cvmfs/service

https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html#docker-container
To get this command to run I had to execute these 2 commands since I never use docker:

sudo apt install docker.io -y
sudo docker pull cvmfs/service

Do I need docker or not???

The previous instructions used this command but the current instructions don't mention it:
sudo apt install glibc-doc open-iscsi watchdog
When I follow the current instructions it said something about watchdog being recommended. I have synaptic options set to consider recommends as dependencies but watchdog and/or open-iscsi did not load. So I ran sudo apt install glibc-doc open-iscsi watchdog.
Is this needed or not???

This section is confusing since it tells me to do something that has already been done by:
sudo wget https://lhcathome.cern.ch/lhcathome/download/default.local -O /etc/cvmfs/default.local
CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice"
So I took to mean I needed to open default.local and edit it and entered this which may be one of my problems:
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch
CVMFS_CLIENT_PROFILE=single
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html#create-default-local

---

Haven't been here in a while. The instructions are far more complex and confusing than they were when I last ran ATLAS.

It is worse than that. They change every time you try. But it can work with Ubuntu 20.04.2.
Here is more or less what I use, though you will probably encounter dependency errors that you will have to fix depending on what you have already.
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5342&postid=41861#41861

But wait, there's more. You need Singularity too for native ATLAS. That is where the fun begins.
There is supposed to be a CERN version that you can use, but I have not found it.
I compile it myself. https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5616&postid=44485#44485

However, the installation of GO has been simplified.
You can just use:

sudo snap install go --classic
Result: go 1.16.2 from Michael Hudson-Doyle (mwhudson) installed

go get -u github.com/golang/dep/cmd/dep
(wait for the download!)

Check version: go version

And you will probably have dependency problems also that you will need to address by downgrading versions.
For example, to correct broken packages (examples):
sudo apt install libseccomp2=2.4.3-1ubuntu1
sudo apt install libssl1.1=1.1.1f-1ubuntu2


That should work, more or less, unless Mint causes extra problems over and above Ubuntu 20.04.2, which is what I am using on several machines now.

I can't get this command to work:
sudo snap install go --classic
sudo: snap: command not found

But when I search in the Synaptic Package Manager it shows I have snap installed.
SNAP is a general purpose gene finding program suitable for both eukaryotic and prokaryotic genomes. SNAP is an acronym for Semi-HMM-based Nucleic Acid Parser.

But when I search SPM for go there is no standalone program called go but there are thousands of programs containing go.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.
Maybe they did not meet their GDPR requirements.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Try this.
https://sylabs.io/guides/3.7/user-guide/quick_start.html

I would post more, but the LHC website keeps going out.

Failed saying no go compiler:

root@Rig-25:/home/aurum/singularity# ./mconfig && \
>     make -C builddir && \
>     sudo make -C builddir install
Configuring for project `singularity' with languages: C, Golang
=> running pre-basechecks project specific checks ...
=> running base system checks ...
 checking: host C compiler... cc
 checking: host C++ compiler... c++
 checking: host Go compiler (at least version 1.13)... not found!
mconfig: could not complete configuration

This is unimaginably complicated.

Native apps are there to support CERN partners who usually run CentOS.
They may run on other Linux flavours but if it's too much work for you feel free to use the VBox app instead.

Yes, unfortunately the CERN world is rather CentOS-centric and so other Linux flavours like Ubuntu don't work out of the box.

So the options for running ATLAS WU are (in rough order of increasing difficulty):

- Run the Vbox app
- Run a pre-made Docker container with no configuration required
- Set up a CentOS VM and run BOINC inside
- Install (compile if necessary) CVMFS and Singularity to run real native tasks
- Wipe Ubuntu and install CentOS 7 (just a joke :)

Atlas Native tasks are running fine under Ubuntu 20.04.2 without singularity.

[2021-03-28 08:34:43] CVMFS is ok
[2021-03-28 08:34:43] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
[2021-03-28 08:34:43] Checking for singularity binary...
[2021-03-28 08:34:43] Singularity is not installed, using version from CVMFS
[2021-03-28 08:34:43] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
[2021-03-28 08:34:45] INFO:  Convert SIF file to sandbox... fonck INFO:  Cleaning up image...
[2021-03-28 08:34:45] Singularity works

Installed packages :

$ dpkg -l |grep -i cvmfs
ii  cvmfs                                                       2.8.0~1+ubuntu20.04                   amd64        CernVM File System
ii  cvmfs-config-default                                        2.0-1                                 all          CernVM File System Default Configuration
ii  cvmfs-fuse3                                                 2.8.0~1+ubuntu20.04                   amd64        CernVM File System fuse3 libraries
ii  cvmfs-release                                               3.2-1                                 all          This package contains the apt configuration for the CernVM File System packages.

aurum@Rig-25:~$ dpkg -l |grep -i cvmfs
ii  cvmfs                                      2.8.0~1+ubuntu18.04                   amd64        CernVM File System
ii  cvmfs-config-default                       2.0-1                                 all          CernVM File System Default Configuration
ii  cvmfs-release                              3.2-1                                 all          This package contains the apt configuration for the CernVM File System packages.

Looks like I'm missing cvmfs-fuse3. Now if I can just figure out how to install that...

Yes, unfortunately the CERN world is rather CentOS-centric and so other Linux flavours like Ubuntu don't work out of the box.
So the options for running ATLAS WU are (in rough order of increasing difficulty):
- Run the Vbox app
- Run a pre-made Docker container with no configuration required
- Set up a CentOS VM and run BOINC inside
- Install (compile if necessary) CVMFS and Singularity to run real native tasks
- Wipe Ubuntu and install CentOS 7 (just a joke :)

1. By definition ATLAS is NOT a native application and you should not call it one.
2. I will never run anything in a VirtualCatBox. I made that mistake before and I'll never do it again.
3. ATLAS ran fine for me before and most of the 37 million points I have here were running ATLAS but NOT in a VirtualCatBox.
4. I believe the current instructions are posted in more than one place and contradict themselves. Please clean up behind yourselves.
5. These were all the steps that it previously took to run ATLAS for Linux Mint 19.3 Tricia base: Ubuntu 18.04 bionic:
For new & reinstalls:

wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb ; \
 sudo dpkg -i cvmfs-release-latest_all.deb ; \
 rm -f cvmfs-release-latest_all.deb ; \
 sudo apt-get update ; \
 sudo apt-get install cvmfs ; \
 sudo apt install glibc-doc open-iscsi watchdog

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/default.local -O /etc/cvmfs/default.local ; \
 sudo cvmfs_config setup ; \
 sudo echo "/cvmfs /etc/auto.cvmfs" > /etc/auto.master.d/cvmfs.autofs ; \
 sudo systemctl restart autofs ; \
 cvmfs_config probe

sudo cvmfs_config reload

6. cvmfs is installed 2.8.0~1+ubuntu18.04 (50.4 MB).
7. It says fuse was installed yet earlier post indicates it's not there. Maybe there's a problem with what's in the Linux repository?
/.
/etc
/etc/auto.cvmfs
/etc/bash_completion.d
/etc/bash_completion.d/cvmfs
/etc/cvmfs
/etc/cvmfs/config.d
/etc/cvmfs/config.sh
/etc/cvmfs/default.conf
/etc/cvmfs/default.d
/etc/cvmfs/default.d/README
/sbin
/sbin/mount.cvmfs
/usr
/usr/bin
/usr/bin/cvmfs2
/usr/bin/cvmfs_config
/usr/bin/cvmfs_fsck
/usr/bin/cvmfs_talk
/usr/lib
/usr/lib/libcvmfs_fuse.so
/usr/lib/libcvmfs_fuse.so.2.8.0
/usr/lib/libcvmfs_fuse_debug.so
/usr/lib/libcvmfs_fuse_debug.so.2.8.0
/usr/lib/libcvmfs_fuse_stub.so
/usr/lib/libcvmfs_fuse_stub.so.2.8.0
/usr/libexec
/usr/libexec/cvmfs
/usr/libexec/cvmfs/authz
/usr/libexec/cvmfs/authz/cvmfs_allow_helper
/usr/libexec/cvmfs/authz/cvmfs_deny_helper
/usr/libexec/cvmfs/auto.cvmfs
/usr/libexec/cvmfs/cache
/usr/libexec/cvmfs/cache/cvmfs_cache_posix
/usr/libexec/cvmfs/cache/cvmfs_cache_ram
/usr/share
/usr/share/doc
/usr/share/doc/cvmfs
/usr/share/doc/cvmfs-2.8.0
/usr/share/doc/cvmfs-2.8.0/AUTHORS
/usr/share/doc/cvmfs-2.8.0/COPYING
/usr/share/doc/cvmfs-2.8.0/ChangeLog.gz
/usr/share/doc/cvmfs-2.8.0/README.md
/usr/share/doc/cvmfs/changelog.Debian.gz
/usr/share/doc/cvmfs/changelog.gz
/usr/share/doc/cvmfs/copyright

---

Regarding the cvmfs 18.04 packages: they are selected by the cvmfs-release package, which is responsible for picking a suitable package repository. It uses

lsb-release

to figure out where it runs. Officially, only the Ubuntu LTS releases are supported. So, "focal" is a recognized release but "ulyssa" is not. For unsupported distributions, the cvmfs-release tries to fall back to a package repository with compatible binaries. In this case, it picks the repository with the 18.04 binaries. While somewhat less than perfect, these binaries should nevertheless work fine on Mint 20, so there is nothing to worry about.

Regarding the cvmfs 18.04 packages: they are selected by the cvmfs-release package, which is responsible for picking a suitable package repository. It uses

lsb-release

to figure out where it runs. Officially, only the Ubuntu LTS releases are supported. So, "focal" is a recognized release but "ulyssa" is not. For unsupported distributions, the cvmfs-release tries to fall back to a package repository with compatible binaries. In this case, it picks the repository with the 18.04 binaries. While somewhat less than perfect, these binaries should nevertheless work fine on Mint 20, so there is nothing to worry about.

That's not the issue. The issue is that LHC ATLAS Fails 100% of the time. What are the correct instructions to make ATLAS run Linux native WUs???

Here's what I'm trying:

wget https://ecsft.cern.ch/dist/cvmfs/cvmfs-release/cvmfs-release-latest_all.deb ; \
sudo dpkg -i cvmfs-release-latest_all.deb ; \
rm -f cvmfs-release-latest_all.deb ; \
sudo apt-get update ; \
sudo apt-get install cvmfs ; \
sudo apt install glibc-doc open-iscsi watchdog

sudo wget https://lhcathomedev.cern.ch/lhcathome-dev/download/default.local -O /etc/cvmfs/default.local ; \
sudo cvmfs_config setup ; \
sudo echo "/cvmfs /etc/auto.cvmfs" > /etc/auto.master.d/cvmfs.autofs ; \
sudo systemctl restart autofs ; \
cvmfs_config probe

sudo xed /etc/cvmfs/default.local
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,grid.cern.ch,sft.cern.ch,alice.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY=DIRECT
CVMFS_USE_CDN=yes
CVMFS_USE_GEOAPI=no

sudo xed /etc/cvmfs/domain.d/cern.ch.local
CVMFS_USE_GEOAPI=no

sudo cvmfs_config reload

What are the correct instructions to make ATLAS run Linux native WUs???

One difference I see is that I use for the Cloudfare servers:
sudo gedit /etc/cvmfs/domain.d/cern.ch.local

Add the following:

CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1unl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io/cvmfs/@fqrn@"

And you need the nightlies:
sudo gedit /etc/cvmfs/config.d/atlas-nightlies.cern.ch.local

Add the following:

CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@"

Also, are you getting error messages about broken packages?
You will need to deal with them on a case-by-case basis.

To check CVMFS is functioning correctly run the following command:
cvmfs_config probe

Which should produce the following output:
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

If the probe fails, try to restart autofs with: sudo systemctl restart autofs

I added those and it still fails.

<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
09:59:55 (52264): wrapper (7.7.26015): starting
09:59:55 (52264): wrapper: running run_atlas (--nthreads 1)
[2021-04-24 09:59:55] Arguments: --nthreads 1
[2021-04-24 09:59:55] Threads: 1
[2021-04-24 09:59:55] Checking for CVMFS
[2021-04-24 09:59:58] Probing /cvmfs/atlas.cern.ch... OK
[2021-04-24 10:00:00] Probing /cvmfs/atlas-condb.cern.ch... OK
[2021-04-24 10:00:01] Probing /cvmfs/grid.cern.ch... OK
[2021-04-24 10:00:02] Probing /cvmfs/cernvm-prod.cern.ch... OK
[2021-04-24 10:00:02] Probing /cvmfs/grid.cern.ch... OK
[2021-04-24 10:00:03] Probing /cvmfs/sft.cern.ch... OK
[2021-04-24 10:00:04] Probing /cvmfs/alice.cern.ch... OK
[2021-04-24 10:00:05] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2021-04-24 10:00:05] 2.8.1.0 52564 0 24632 83323 3 1 473981 4194304 0 130560 0 0 100.000 0 0 http://s1unl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1
[2021-04-24 10:00:05] CVMFS is ok
[2021-04-24 10:00:05] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
[2021-04-24 10:00:05] Checking for singularity binary...
[2021-04-24 10:00:05] Singularity is not installed, using version from CVMFS
[2021-04-24 10:00:05] Checking singularity works with /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
[2021-04-24 10:00:10] INFO: Converting SIF file to temporary sandbox... Rig-46 INFO: Cleaning up image...
[2021-04-24 10:00:10] Singularity works
[2021-04-24 10:00:10] Starting ATLAS job with PandaID=5035383237
[2021-04-24 10:00:10] Running command: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec --pwd /var/lib/boinc-client/slots/35 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh
[2021-04-24 10:00:17] Job failed
[2021-04-24 10:00:17] INFO:    Converting SIF file to temporary sandbox...
[2021-04-24 10:00:17] INFO:    Cleaning up image...
[2021-04-24 10:00:17] FATAL:   container creation failed: hook function for tag prelayer returns error: failed to create /var/lib/alternatives directory: mkdir /var/lib/alternatives: read-only file system
[2021-04-24 10:00:17] ./runtime_log
[2021-04-24 10:00:17] ./runtime_log.err
10:10:17 (52264): run_atlas exited; CPU time 11.489694
10:10:17 (52264): app exit status: 0x1
10:10:17 (52264): called boinc_finish(195)
</stderr_txt>