Singularity error in native tasks

Author	Message
David Cameron Project administrator Project developer Project scientist Send message Joined: 13 May 14 Posts: 387 Credit: 15,314,184 RAC: 0	Message 40729 - Posted: 29 Nov 2019, 16:01:53 UTC Hi, I just discovered one of my hosts has been failing ATLAS native WU for some time, with a singularity issue I have never seen before: 2019-11-29 04:24:14,886: [31mFATAL: [0m container creation failed: hook function for tag layer returns error: failed to create /var/singularity/mnt/session/overlay-lowerdir/var/lib/boinc/slots directory: mkdir /var/singularity/mnt/session/overlay-lowerdir/var/lib/boinc/slots: no such file or directory https://lhcathome.cern.ch/lhcathome/result.php?resultid=253370224 This is a standard CentOS 7 machine with singularity 3.5.0 from EPEL and is only used for crunching ATLAS WU. Of course I could uninstall singularity to use the one from CVMFS but I wondered if anyone had seen this issue and knows how to fix it? ID: 40729 · Reply Quote

maeax Send message Joined: 2 May 07 Posts: 2090 Credit: 158,859,024 RAC: 126,466	Message 40732 - Posted: 29 Nov 2019, 16:43:50 UTC - in response to Message 40729. Have also CentOs7 but with singularity 3.4.0-1.2.el7 from CVMFS. Today CMS had got updates from Cern-IT (see news from Ivan). ID: 40732 · Reply Quote

David Cameron Project administrator Project developer Project scientist Send message Joined: 13 May 14 Posts: 387 Credit: 15,314,184 RAC: 0	Message 40924 - Posted: 12 Dec 2019, 15:23:14 UTC This host got an auto-update to singularity 3.5.1 a couple of days ago and since then tasks have been completing successfully. I suppose that 3.5.0 has some bugs. This is a good reason to use singularity from CVMFS since it is always validated to work with ATLAS tasks. ID: 40924 · Reply Quote

LHC@home