LHC@home

ATLAS native Linux version 2.70 was just release with the following changes:

- Singularity is now mandatory for all hosts (previously SLC6/CentOS 6 hosts could run without singularity)

- You no longer need to install singularity on your hosts because there is a version on CVMFS that can be used instead. Depending on your operating system, you may have to enable user namespaces for this to work. See more information on this post about theory native. This brings the ATLAS requirements nicely into line with theory native, i.e. all you need is CVMFS. If you have singularity locally installed this will be preferred to the CVMFS version.

- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity

Please let us know of any issues! And a big thanks to those on LHC-dev who helped test these changes.

If you have singularity locally installed this will be preferred to the CVMFS version.
- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity.

Would it be beneficial to uninstall a locally installed singularity, and install the CVMFS version?

Atlas native used to work for me a few months ago but doesn't now.
I have updated the default.local file that now includes the last line that wasn't there before.

$ more default.local
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.ce
rn.ch,sft.cern.ch,alice.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY=DIRECT
CVMFS_SEND_INFO_HEADER=yes

Have wiped the cache, reloaded, etc.

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

$ singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
ERROR : Image path /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img doesn't exist: No such file or directory
ABORT : Retval = 255

$ ls /cvmfs
alice.cern.ch atlas-condb.cern.ch grid.cern.ch
atlas.cern.ch cernvm-prod.cern.ch sft.cern.ch

$ ls /cvmfs/atlas.cern.ch/repo/
ATLASLocalRootBase benchmarks conditions containers dev sw tools

There is no images directory !

Is this an easy fix and how do I get it to work please ?

- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity

It looks like this doesn't work with older versions of singularity, so I have reverted back to the previous image with the newly released 2.71.

$ singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
ERROR : Image path /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img doesn't exist: No such file or directory
ABORT : Retval = 255

This was the path used in old versions of ATLAS native. Since version 2.59 we use the image at

/cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-slc6.img

2.70 was using the unpacked image at

/cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-slc6

but this doesn't work for some versions of singularity so I rolled it back in 2.71. I suspect this is why you saw failures.

Hello!

Since the change to native Version 2.71 all tasks fail with error 195 (0x000000C3) EXIT_CHILD_FAILED.

I have a host with cvmfs installed only, no singularity.
Seems like the new version can not start the cvmfs included version of singularity.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=247196671

Cheers, djoser.

---

Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us

Hello!

Since the change to native Version 2.71 all tasks fail with error 195 (0x000000C3) EXIT_CHILD_FAILED.

I have a host with cvmfs installed only, no singularity.
Seems like the new version can not start the cvmfs included version of singularity.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=247196671

Cheers, djoser.

That's what David Cameron explained:
2.70 works without a local singularity installation but causes problems if (an older) singularity is already present.
Hence he rolled back to 2.60 which requires a local singularity but is now present as 2.71 for continuity reasons.

Thanks for the explaination.
I guess i did not get the messsage from David right, since i am not a native english speaker.

Too bad he had to roll it back, since 2.70 worked well for me...

---

Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us

I have a host with ... no singularity.

This is untested but might be a workaround if all required libs are present on your system:

[sudo] ln -s /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity /usr/bin/singularity

This is untested but might be a workaround if all required libs are present on your system:

[sudo] ln -s /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity /usr/bin/singularity

I uninstalled singularity 2.6.1 and ran that, but got:
ln: failed to create symbolic link '/usr/bin/singularity': File exists

EDIT: I then removed the singularity link in /usr/bin/, and tried again. It worked then.
Now I have: singularity version 3.2.1-1
Very nice. Thanks.

But the combination of 2.71 with singularity version 3.2.1-1 does not work for me. I can't tell whether this is the same problem previously reported, or a new one.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10612434

Note 1: They don't work on any of the vbox machines that have tried them either.
Note 2: It is getting hard to edit this forum. Maybe the servers are going out.
Note 3: #2 would be useful to give them time to fix the already-identified problems.

Thanks for the explaination.
I guess i did not get the messsage from David right, since i am not a native english speaker.

Too bad he had to roll it back, since 2.70 worked well for me...

Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

Unfortunately using the packed image requires unsquashfs installed to unpack it, which is why your tasks failed:

2019-09-28 06:37:51,532: Singularity isnt working: INFO:  Convert SIF file to sandbox...
FATAL:  while extracting /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-slc6.img: root filesystem extraction failed: could not extract squashfs data, unsquashfs not found

So you need to install the "squashfs-tools" package for this to work.

I'm thinking that what I might do is enforce that people use a version of singularity that works with the unpacked image, i.e. if you have a reasonably new locally installed version, use that, if not then use the CVMFS version.

Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

Unfortunately using the packed image requires unsquashfs installed to unpack it, which is why your tasks failed:
So you need to install the "squashfs-tools" package for this to work.

Hello David,

Thanks for taking the time to look into my problem.
I think your explaination was quite sufficient, only my english wasn't :-)

I will install the squashfs tools later today and try again!

Regards, djoser.

---

Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us

Short update:
After installing squashfs-tools everything works flawlessly :-)

---

Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us

Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

OK, so 2.71 should work with singularity version 3.2.1-1, which I have installed as suggested here:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5151&postid=40040#40040

And CVMFS checks out OK:

$ cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

But all I get are errors.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10607999&offset=0&show_names=0&state=6&appid=14

If it were bad work units, I would expect a lot of complaints here. So maybe it is an incompatibility between my installed singularity, and the one with CVMFS?

I dug out the error log from your last task (I'll try to put this info in the stderr in the future):

INFO:    Convert SIF file to sandbox...
INFO:    Cleaning up image...
FATAL:   container creation failed: mount ->/var error: can't remount /var: operation not permitted

Not sure exactly why that doesn't work for you. If you have singularity locally installed then the CVMFS one is not used so I don't think there is an incompatibility. Maybe you could try the version from CVMFS - you can easily test without uninstalling your local version by putting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin at the start of your PATH.

Not sure exactly why that doesn't work for you. If you have singularity locally installed then the CVMFS one is not used so I don't think there is an incompatibility. Maybe you could try the version from CVMFS - you can easily test without uninstalling your local version by putting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin at the start of your PATH.

Thanks, but I think the problem is deeper on this machine. I tried your change, and even uninstalled my previous version of singularity, but I still get the errors.
Looking back at my results, I see the errors go back to singularity 2.6.0, so they may not have been introduced by my upgrading to 3.2.1.1 (which is still what "singularity --version" shows by the way).

I think I will give it a rest and try something else later. There is nothing you can do on your end. I may need to re-install the OS.

EDIT: Thinking back, I believe I tried to upgrade from 2.6.0 earlier by compiling a later version myself. That was doomed to fail, and undoubtedly borked the machine, I just had not noticed yet. Thanks for your input.