Message boards : ATLAS application : ATLAS native version 2.70
Message board moderation

To post messages, you must log in.

AuthorMessage
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 40028 - Posted: 26 Sep 2019, 14:24:49 UTC
Last modified: 26 Sep 2019, 15:09:47 UTC

ATLAS native Linux version 2.70 was just release with the following changes:

- Singularity is now mandatory for all hosts (previously SLC6/CentOS 6 hosts could run without singularity)

- You no longer need to install singularity on your hosts because there is a version on CVMFS that can be used instead. Depending on your operating system, you may have to enable user namespaces for this to work. See more information on this post about theory native. This brings the ATLAS requirements nicely into line with theory native, i.e. all you need is CVMFS. If you have singularity locally installed this will be preferred to the CVMFS version.

- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity

Please let us know of any issues! And a big thanks to those on LHC-dev who helped test these changes.
ID: 40028 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 40029 - Posted: 26 Sep 2019, 16:53:02 UTC - in response to Message 40028.  

If you have singularity locally installed this will be preferred to the CVMFS version.
- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity.

Would it be beneficial to uninstall a locally installed singularity, and install the CVMFS version?
ID: 40029 · Report as offensive     Reply Quote
Profile PDW

Send message
Joined: 7 Aug 14
Posts: 14
Credit: 7,479,532
RAC: 50
Message 40031 - Posted: 27 Sep 2019, 8:02:47 UTC - in response to Message 40028.  

Atlas native used to work for me a few months ago but doesn't now.
I have updated the default.local file that now includes the last line that wasn't there before.

$ more default.local
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.ce
rn.ch,sft.cern.ch,alice.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY=DIRECT
CVMFS_SEND_INFO_HEADER=yes

Have wiped the cache, reloaded, etc.

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

$ singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
ERROR : Image path /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img doesn't exist: No such file or directory
ABORT : Retval = 255

$ ls /cvmfs
alice.cern.ch atlas-condb.cern.ch grid.cern.ch
atlas.cern.ch cernvm-prod.cern.ch sft.cern.ch

$ ls /cvmfs/atlas.cern.ch/repo/
ATLASLocalRootBase benchmarks conditions containers dev sw tools

There is no images directory !

Is this an easy fix and how do I get it to work please ?
ID: 40031 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 40032 - Posted: 27 Sep 2019, 8:09:27 UTC - in response to Message 40028.  

- The singularity image is now an "unpacked filesystem image" which speeds up the starting time of singularity


It looks like this doesn't work with older versions of singularity, so I have reverted back to the previous image with the newly released 2.71.
ID: 40032 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 40034 - Posted: 27 Sep 2019, 9:04:36 UTC - in response to Message 40031.  
Last modified: 27 Sep 2019, 9:04:59 UTC

$ singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
ERROR : Image path /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img doesn't exist: No such file or directory
ABORT : Retval = 255


This was the path used in old versions of ATLAS native. Since version 2.59 we use the image at

/cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-slc6.img

2.70 was using the unpacked image at

/cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-slc6

but this doesn't work for some versions of singularity so I rolled it back in 2.71. I suspect this is why you saw failures.
ID: 40034 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 145
Credit: 10,847,070
RAC: 0
Message 40037 - Posted: 28 Sep 2019, 6:19:37 UTC
Last modified: 28 Sep 2019, 6:32:18 UTC

Hello!

Since the change to native Version 2.71 all tasks fail with error 195 (0x000000C3) EXIT_CHILD_FAILED.

I have a host with cvmfs installed only, no singularity.
Seems like the new version can not start the cvmfs included version of singularity.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=247196671

Cheers, djoser.
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us
ID: 40037 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,931,335
RAC: 137,631
Message 40038 - Posted: 28 Sep 2019, 7:20:34 UTC - in response to Message 40037.  

Hello!

Since the change to native Version 2.71 all tasks fail with error 195 (0x000000C3) EXIT_CHILD_FAILED.

I have a host with cvmfs installed only, no singularity.
Seems like the new version can not start the cvmfs included version of singularity.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=247196671

Cheers, djoser.

That's what David Cameron explained:
2.70 works without a local singularity installation but causes problems if (an older) singularity is already present.
Hence he rolled back to 2.60 which requires a local singularity but is now present as 2.71 for continuity reasons.
ID: 40038 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 145
Credit: 10,847,070
RAC: 0
Message 40039 - Posted: 28 Sep 2019, 8:18:36 UTC - in response to Message 40038.  

Thanks for the explaination.
I guess i did not get the messsage from David right, since i am not a native english speaker.

Too bad he had to roll it back, since 2.70 worked well for me...
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us
ID: 40039 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,931,335
RAC: 137,631
Message 40040 - Posted: 28 Sep 2019, 11:38:05 UTC - in response to Message 40037.  

I have a host with ... no singularity.

This is untested but might be a workaround if all required libs are present on your system:
[sudo] ln -s /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity /usr/bin/singularity
ID: 40040 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 40041 - Posted: 28 Sep 2019, 19:16:58 UTC - in response to Message 40040.  
Last modified: 28 Sep 2019, 19:26:19 UTC

This is untested but might be a workaround if all required libs are present on your system:
[sudo] ln -s /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity /usr/bin/singularity

I uninstalled singularity 2.6.1 and ran that, but got:
ln: failed to create symbolic link '/usr/bin/singularity': File exists

EDIT: I then removed the singularity link in /usr/bin/, and tried again. It worked then.
Now I have: singularity version 3.2.1-1
Very nice. Thanks.
ID: 40041 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 40042 - Posted: 29 Sep 2019, 7:24:40 UTC
Last modified: 29 Sep 2019, 7:39:08 UTC

But the combination of 2.71 with singularity version 3.2.1-1 does not work for me. I can't tell whether this is the same problem previously reported, or a new one.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10612434

Note 1: They don't work on any of the vbox machines that have tried them either.
Note 2: It is getting hard to edit this forum. Maybe the servers are going out.
Note 3: #2 would be useful to give them time to fix the already-identified problems.
ID: 40042 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 40049 - Posted: 29 Sep 2019, 19:29:33 UTC - in response to Message 40039.  

Thanks for the explaination.
I guess i did not get the messsage from David right, since i am not a native english speaker.

Too bad he had to roll it back, since 2.70 worked well for me...


Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

Unfortunately using the packed image requires unsquashfs installed to unpack it, which is why your tasks failed:

2019-09-28 06:37:51,532: Singularity isnt working: INFO:  Convert SIF file to sandbox...
FATAL:  while extracting /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-slc6.img: root filesystem extraction failed: could not extract squashfs data, unsquashfs not found


So you need to install the "squashfs-tools" package for this to work.

I'm thinking that what I might do is enforce that people use a version of singularity that works with the unpacked image, i.e. if you have a reasonably new locally installed version, use that, if not then use the CVMFS version.
ID: 40049 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 145
Credit: 10,847,070
RAC: 0
Message 40055 - Posted: 30 Sep 2019, 14:43:23 UTC - in response to Message 40049.  

Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

Unfortunately using the packed image requires unsquashfs installed to unpack it, which is why your tasks failed:
So you need to install the "squashfs-tools" package for this to work.

Hello David,

Thanks for taking the time to look into my problem.
I think your explaination was quite sufficient, only my english wasn't :-)

I will install the squashfs tools later today and try again!

Regards, djoser.
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us
ID: 40055 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 145
Credit: 10,847,070
RAC: 0
Message 40058 - Posted: 1 Oct 2019, 11:17:47 UTC

Short update:
After installing squashfs-tools everything works flawlessly :-)
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us
ID: 40058 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 40059 - Posted: 1 Oct 2019, 16:33:43 UTC - in response to Message 40049.  

Sorry, I didn't explain very well what I rolled back in 2.71. I changed the singularity image that we use back to the one used with 2.60, because the unpacked image didn't work with older singularity releases.

The change to use singularity from CVMFS is still present in 2.71, so if you do not have singularity installed ("singularity" is not found in your PATH) then the CVMFS one will be used.

OK, so 2.71 should work with singularity version 3.2.1-1, which I have installed as suggested here:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5151&postid=40040#40040

And CVMFS checks out OK:
$ cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK


But all I get are errors.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10607999&offset=0&show_names=0&state=6&appid=14

If it were bad work units, I would expect a lot of complaints here. So maybe it is an incompatibility between my installed singularity, and the one with CVMFS?
ID: 40059 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 40066 - Posted: 2 Oct 2019, 8:42:58 UTC - in response to Message 40059.  

I dug out the error log from your last task (I'll try to put this info in the stderr in the future):

INFO:    Convert SIF file to sandbox...
INFO:    Cleaning up image...
FATAL:   container creation failed: mount ->/var error: can't remount /var: operation not permitted


Not sure exactly why that doesn't work for you. If you have singularity locally installed then the CVMFS one is not used so I don't think there is an incompatibility. Maybe you could try the version from CVMFS - you can easily test without uninstalling your local version by putting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin at the start of your PATH.
ID: 40066 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 40068 - Posted: 2 Oct 2019, 12:33:31 UTC - in response to Message 40066.  
Last modified: 2 Oct 2019, 13:14:15 UTC

Not sure exactly why that doesn't work for you. If you have singularity locally installed then the CVMFS one is not used so I don't think there is an incompatibility. Maybe you could try the version from CVMFS - you can easily test without uninstalling your local version by putting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin at the start of your PATH.

Thanks, but I think the problem is deeper on this machine. I tried your change, and even uninstalled my previous version of singularity, but I still get the errors.
Looking back at my results, I see the errors go back to singularity 2.6.0, so they may not have been introduced by my upgrading to 3.2.1.1 (which is still what "singularity --version" shows by the way).

I think I will give it a rest and try something else later. There is nothing you can do on your end. I may need to re-install the OS.

EDIT: Thinking back, I believe I tried to upgrade from 2.6.0 earlier by compiling a later version myself. That was doomed to fail, and undoubtedly borked the machine, I just had not noticed yet. Thanks for your input.
ID: 40068 · Report as offensive     Reply Quote

Message boards : ATLAS application : ATLAS native version 2.70


©2024 CERN