LHC@home

We just released a new version of ATLAS native, which updates the image used by singularity to CentOS 7. This is the only difference from 2.71 and is done to keep in sync with the new update of the vbox version to CentOS 7.

The 2.72 are all failing for me after 10 minutes, with
195 (0x000000C3) EXIT_CHILD_FAILED.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10609230&offset=0&show_names=0&state=6&appid=14

I had no problems with 2.71.

2019-10-10 04:03:26,698: Checking Singularity...
2019-10-10 04:03:26,712: Singularity is installed, version 2.6.1-dist
2019-10-10 04:03:26,712: Testing the function of Singularity...
2019-10-10 04:03:26,712: Checking singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
2019-10-10 04:03:26,741: Singularity isnt working: ERROR : Unknown image format/type: /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
ABORT : Retval = 255

Your singularity version (2.6.1) is very old. Maybe the new image needs a more current version in order to work. You should update your singularity version.

For me, native version 2.72 looks like it is working without a problem:

2019-10-09 20:09:23,818: singularity image is /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img
2019-10-09 20:09:23,819: sys.argv = ['run_atlas', '--nthreads', '2']
2019-10-09 20:09:23,820: THREADS=2
2019-10-09 20:09:23,821: Checking for CVMFS
2019-10-09 20:09:39,404: CVMFS is installed
2019-10-09 20:09:39,404: Checking Singularity...
2019-10-09 20:09:40,399: Singularity is installed, version singularity version 3.4.1+324-g54b182afd
2019-10-09 20:09:40,399: Testing the function of Singularity...
2019-10-09 20:09:40,399: Checking singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
2019-10-09 20:10:03,571: Singularity Works...
2019-10-09 20:10:03,572: copy /home/boinc/boinc1/slots/0/shared/ATLAS.root_0
2019-10-09 20:10:03,872: copy /home/boinc/boinc1/slots/0/shared/RTE.tar.gz
2019-10-09 20:10:03,873: copy /home/boinc/boinc1/slots/0/shared/input.tar.gz
2019-10-09 20:10:03,873: copy /home/boinc/boinc1/slots/0/shared/start_atlas.sh
2019-10-09 20:10:03,873: export ATHENA_PROC_NUMBER=2;
2019-10-09 20:10:04,243: start atlas job with PandaID=4503174146
2019-10-09 20:10:04,243: cmd = singularity exec --pwd /home/boinc/boinc1/slots/0 -B /cvmfs,/home /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh > runtime_log 2> runtime_log.err

The first 2.72 task is running for over 10 hours now and everything looks fine.

Your singularity version (2.6.1) is very old. Maybe the new image needs a more current version in order to work.

Indeed.
Looks like v2.72 requires a more recent singularity than 2.6.1 (which is also included in opensuse leap 15.1).

To solve the problems on my hosts I removed the local singularity and fresh tasks now use singularity from CVMFS.

To solve the problems on my hosts I removed the local singularity and fresh tasks now use singularity from CVMFS.

What commands did you use to remove it? I tried that on two other machines, and ended up with nothing that worked.
"singularity --version" then showed nothing.

Since I started long ago with a manual installation and that might be mixed with an installation from the distribution package I manually removed the files and directories.

On my hosts the following places were affected:
/etc/singularity
/usr/bin/singularity
/usr/local/bin/singularity
/usr/local/bin/run-singularity
/usr/local/lib/singularity

Renaming all of them should do the trick.
Then request a fresh ATLAS task and check the stderr.txt for the following lines:

2019-10-10 09:59:42,014: Singularity seems to be installed but not working: sh: singularity: Kommando nicht gefunden.
2019-10-10 09:59:42,014: Will use version from CVMFS
2019-10-10 09:59:42,014: Testing the function of Singularity...
2019-10-10 09:59:42,014: Checking singularity with cmd:/cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
2019-10-10 09:59:55,072: Singularity Works...

<edit>
forgot to mention
/usr/local/lib64/singularity
</edit>

Looking around the web it seems that singularity does not guarantee any backwards compatibility, so if you create an image with one version it is not guaranteed to work with previous versions. I think the centos7 version was built with version 3 so will probably not work with 2.x.

If you have an old 2.x version I would recommend uninstalling it as described above so that the one from CVMFS is used, since this should be guaranteed to work with ATLAS images. If you do this please remember to check that user namespaces are enabled as described here for the theory native app.

OK, this is a bit of a long story, but after removing various directories I never could get the CVMFS version to work that I can see.
So I installed the one-line version https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5151&postid=40040, and at least got singularity version 3.2.1-1.
But even that did not appear to run, though I may not have waited long enough; it was slow starting.

So I just did "sudo apt install singularity", and got a large bunch of files I had never seen before, and probably are not necessary.
But it now works, so that will do.

Thanks to all.

While 2.72 is running fine on my Ubunu 16.04 machine now, the same cannot be said for my Ubuntu 18.04 machines.
The work units basically do not run, but error out in 10 minutes.

i7-8700:https://lhcathome.cern.ch/lhcathome/results.php?hostid=10612434
Note that the stderr output says that "Singularity is installed, version singularity version 3.2.1-1".

i7-9700:https://lhcathome.cern.ch/lhcathome/results.php?hostid=10607999
Note that the stderr output says that "Singularity is not installed, using version from CVMFS".

Also note that there is a high error rate on all of these work units on other machines too, but I see a few that work.
Something is amiss.

Try remove singularity and use the one build in vm. It worked for me with 18.10.

$ sudo rm -rf \
    /usr/local/libexec/singularity \
    /usr/local/var/singularity \
    /usr/local/etc/singularity \
    /usr/local/bin/singularity \
    /usr/local/bin/run-singularity \
    /usr/local/etc/bash_completion.d/singularity

https://sylabs.io/guides/3.0/user-guide/installation.html#remove-an-old-version

At this point, switching between one singularity an another probably won't save it. The problem seems to be at a deeper level.
I will wipe out one of the machines and start over. That might fix it. Thanks

LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.6.1810 (Core)
Release: 7.6.1810
Boinc 7.16.1

-10-15 11:19:31,539: start atlas job with PandaID=4505354066
2019-10-15 11:19:31,539: cmd = singularity exec --pwd /var/lib/boinc/slots/0 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh > runtime_log 2> runtime_log.err
2019-10-16 11:31:58,538: running cmd return value is 0
2019-10-16 11:31:58,540: Moving ./HITS.19056301._009925.pool.root.1 to shared/HITS.pool.root.1
2019-10-16 11:31:58,540: HITS result file:
2019-10-16 11:31:58,558: -rw-------. 1 boinc boinc 243621250 16. Okt 11:30 shared/HITS.pool.root.1

First task -native CentOS76 with CentOS inside for Atlas-Task?

2019-10-15 11:19:31,539: cmd = singularity exec --pwd /var/lib/boinc/slots/0 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh > runtime_log 2> runtime_log.err
2019-10-16 11:31:58,540: Moving ./HITS.19056301._009925.pool.root.1 to shared/HITS.pool.root.1

First task -native CentOS76 with CentOS inside for Atlas-Task?

Operating System: Linux CentOS Linux CentOS Linux 7 (Core) [3.10.0-1062.1.2.el7.x86_64|libc 2.17 (GNU libc)]:

2019-10-14 18:50:34,710: cmd = singularity exec --pwd /var/lib/boinc/slots/0 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img sh start_atlas.sh > runtime_log 2> runtime_log.err
2019-10-15 00:34:57,894: running cmd return value is 0
2019-10-15 00:34:57,895: Moving ./HITS.19056367._007754.pool.root.1 to shared/HITS.pool.root.1

:)

---

Henry,
you are one step further and have one PC with CentOs8. ;-)

Yes, but Vbox only: I'm still waiting for the official CVMFS release for CentOS 8 - I'm not brave enough to get into building/installing it myself! :(

---