Message boards : ATLAS application : Singularity error in native tasks
Message board moderation

To post messages, you must log in.

AuthorMessage
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 309
Credit: 9,890,059
RAC: 8,904
Message 40729 - Posted: 29 Nov 2019, 16:01:53 UTC

Hi,

I just discovered one of my hosts has been failing ATLAS native WU for some time, with a singularity issue I have never seen before:
2019-11-29 04:24:14,886: FATAL:   container creation failed: hook function for tag layer returns error: failed to create /var/singularity/mnt/session/overlay-lowerdir/var/lib/boinc/slots directory: mkdir /var/singularity/mnt/session/overlay-lowerdir/var/lib/boinc/slots: no such file or directory

https://lhcathome.cern.ch/lhcathome/result.php?resultid=253370224

This is a standard CentOS 7 machine with singularity 3.5.0 from EPEL and is only used for crunching ATLAS WU. Of course I could uninstall singularity to use the one from CVMFS but I wondered if anyone had seen this issue and knows how to fix it?
ID: 40729 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 875
Credit: 32,389,270
RAC: 46,112
Message 40732 - Posted: 29 Nov 2019, 16:43:50 UTC - in response to Message 40729.  

Have also CentOs7 but with singularity 3.4.0-1.2.el7 from CVMFS.
Today CMS had got updates from Cern-IT (see news from Ivan).
ID: 40732 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 309
Credit: 9,890,059
RAC: 8,904
Message 40924 - Posted: 12 Dec 2019, 15:23:14 UTC

This host got an auto-update to singularity 3.5.1 a couple of days ago and since then tasks have been completing successfully. I suppose that 3.5.0 has some bugs. This is a good reason to use singularity from CVMFS since it is always validated to work with ATLAS tasks.
ID: 40924 · Report as offensive     Reply Quote

Message boards : ATLAS application : Singularity error in native tasks


©2020 CERN