LHC@home

I was following the very good gyllic's guide to install everything I need to run ATLAS native.
However all WUs have failed with the following error message:
This is not SLC6, need to run with Singularity....
Checking Singularity...
sh: 1: singularity: not found
Singularity is not installed, aborting

For sure, singularity is installed:
$ sudo su - boinc
$ singularity

Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)
.....

Moreover if I try to check singularity manually by command:
singularity exec --debug -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
I'm getting the following output along with the error:
singularity --debug run /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-centos6.img VERBOSE [U=0,P=7990] singularity_message_level() Set messagelevel to: 5
DEBUG [U=0,P=7990] init() PIPE_EXEC_FD value: 7
VERBOSE [U=0,P=7990] init() Container runtime
VERBOSE [U=0,P=7990] is_suid() Check if we are running as setuid
DEBUG [U=0,P=7990] init() Overlay seems supported by kernel
DEBUG [U=0,P=7990] init() Read json configuration from pipe
DEBUG [U=0,P=7990] init() Set child signal mask
DEBUG [U=0,P=7990] init() Create socketpair for smaster communication channel
DEBUG [U=0,P=7990] init() Wait C and JSON runtime configuration from scontainer stage 1
DEBUG [U=0,P=7997] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=7997] init() Spawn scontainer stage 1
VERBOSE [U=0,P=7997] startup() Execute scontainer stage 1
DEBUG [U=0,P=7997] SContainer() Entering scontainer stage 1
DEBUG [U=0,P=7997] prepareRootCaps() Root full capabilities
DEBUG [U=0,P=7997] Init() Entering image format intializer
DEBUG [U=0,P=7997] Init() Check for image format sif
DEBUG [U=0,P=7997] Init() Opening /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-centos6.img in read-only mode: no write permissions
DEBUG [U=0,P=7997] Init() sif format initializer returns: SIF magic not found
DEBUG [U=0,P=7997] Init() Check for image format sandbox
DEBUG [U=0,P=7997] Init() sandbox format initializer returns: not a directory image
DEBUG [U=0,P=7997] Init() Check for image format squashfs
DEBUG [U=0,P=7997] Init() squashfs format initializer returns: not a valid squashfs image
DEBUG [U=0,P=7997] Init() Check for image format ext3
DEBUG [U=0,P=7990] init() Receiving configuration from scontainer stage 1
DEBUG [U=0,P=7990] init() Wait completion of scontainer stage1
VERBOSE [U=0,P=7990] priv_escalate() Get root privileges
VERBOSE [U=0,P=7990] create_namespace() Create mount namespace
DEBUG [U=0,P=7990] init() Create RPC socketpair for communication between scontainer and RPC server
VERBOSE [U=0,P=7990] init() Spawn smaster process
DEBUG [U=0,P=8075] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=8075] init() Spawn scontainer stage 2
VERBOSE [U=0,P=8075] create_namespace() Create mount namespace
VERBOSE [U=0,P=8076] init() Spawn RPC server
VERBOSE [U=0,P=7990] startup() Execute smaster process
DEBUG [U=0,P=7990] setupSessionLayout() Attempting to use overlayfs (enable overlay = try)
DEBUG [U=0,P=7990] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=0,P=7990] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=0,P=7990] addRootfsMount() Mounting block [ext3] image: /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-centos6.img
DEBUG [U=0,P=7990] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=0,P=7990] addKernelMount() Adding proc to mount list
DEBUG [U=0,P=7990] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=0,P=7990] addKernelMount() Adding sysfs to mount list
DEBUG [U=0,P=7990] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=0,P=7990] addDevMount() Adding dev to mount list
DEBUG [U=0,P=7990] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=0,P=7990] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=0,P=7990] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=0,P=7990] addHomeStagingDir() Staging home directory (/root) at /usr/local/var/singularity/mnt/session/root
DEBUG [U=0,P=7990] addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/root:/root] to list using layer: overlay
DEBUG [U=0,P=7990] isLayerEnabled() Using Layer system: overlay
DEBUG [U=0,P=7990] addTmpMount() Checking for 'mount tmp' in configuration file
DEBUG [U=0,P=7990] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=0,P=7990] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=0,P=7990] addResolvConfMount() Adding /etc/resolv.conf to mount list
DEBUG [U=0,P=7990] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=0,P=7990] create() Mount all
DEBUG [U=0,P=7990] mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session
VERBOSE [U=0,P=8076] startup() Serve RPC requests
DEBUG [U=0,P=7990] mountImage() Mounting loop device /dev/loop0 to /usr/local/var/singularity/mnt/session/rootfs
DEBUG [U=0,P=7990] mountGeneric() Mounting overlay to /usr/local/var/singularity/mnt/session/final
DEBUG [U=0,P=7990] CleanupContainer() Cleanup container
FATAL [U=0,P=7990] SMaster() container creation failed: mount error: can't mount overlay filesystem to /usr/local/var/singularity/mnt/session/final: invalid argument

Not sure if it's related to the first error.
Any help would be greatly appreciated.

Have you run the command "sudo make install" after compiling singularity?
Can you run singularity with your default user (i.e. with a different user than boinc)?
Which version of singularity do you use? Type "singularity --version" into your terminal and please post the output.
Which OS are you running? It shows Linux 3.19, current Debian is using 4.9.0 and olstable Debian is using 3.16 (the guide has only been tested on Debian stretch, but singularity obviously should work on other OS's as well). So please post the output of "uname -a" here as well.

Have you run the command "sudo make install" after compiling singularity?
Can you run singularity with your default user (i.e. with a different user than boinc)?
Which version of singularity do you use? Type "singularity --version" into your terminal and please post the output.
Which OS are you running? It shows Linux 3.19, current Debian is using 4.9.0 and olstable Debian is using 3.16 (the guide has only been tested on Debian stretch, but singularity obviously should work on other OS's as well). So please post the output of "uname -a" here as well.

Thank you for your time.
Yes, I run this command as I can see from history:
git clone https://github.com/sylabs/singularity.git
394 cd singularity
395 go get -u -v github.com/golang/dep/cmd/dep
396 cd $GOPATH/src/github.com/sylabs/singularity
397 ./mconfig
398 cd ./builddir
399 make
400 singularity --version
401 sudo make install
402 singularity --version

I can run singularity as default user:
sid@j1900c ~ $ singularity

Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)
....
And I can run it as root too.
The version is:
sid@j1900c ~ $ singularity --version
singularity version v3.0.0

Regarding Linux - I am running Linux Mint:
sid@j1900c ~ $ uname -a
Linux j1900c 3.19.0-32-generic #37~14.04.1-Ubuntu SMP Thu Oct 22 09:41:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Probably this is the root cause of the issue.

Thanks for your information.

I am no expert on these kind of things, but my first guess is that your kernel is missing the overlay filesystem module or it is failing to load it automatically (since this is only a guess, your problem might be located somewhere completely different). It should be in the kernel since (I think) version 3.18, but maybe it is not in yours.

To check that, please run the command "singularity --debug exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname" and then run the command "lsmod". Look if the module called "overlay" is shown in the output of the lsmod command (and that it is used).

If you dont see a module called overlay in the lsmod output, look into the folder /lib/modules/*your kernel*/kernel/fs/ and search for a folder called overlayfs (at least this is the path on debian).
I have not tested the following, so this is just to give you an idea on how you may be able to fix your problem: If the overlayfs directory is present, you can try to manually load it with the "modprobe" command and the according module name, so "modprobe overlay" or something like that. You can also type the module name into the /etc/modules file and restart your PC. This way the module should be automatically loaded at boot. If no such folder exists, your kernel probably does not have the overlay module which is needed (according to the singularity debug output). You maybe can try to use a backport kernel or upgrade your Mint 17.x installation to a newer one (which you should do because Mint 17.x is only supported until April 2019).

I have set up a virtual machine with Linux Mint 17.3 and singularity also did not work with the default kernel. I then updated the kernel from the default 3.19 to 4.4.0 generic, and now singularity with the "singularity --debug exec ..." command works. So it is propably something with the default kernel that singularity does not like.
My uname -a output is: Linux testing-VirtualBox 4.4.0-98-generic #121~14.04.1-Ubuntu SMP Wed Oct 11 11:54:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I have set up a virtual machine with Linux Mint 17.3 and singularity also did not work with the default kernel. I then updated the kernel from the default 3.19 to 4.4.0 generic, and now singularity with the "singularity --debug exec ..." command works. So it is propably something with the default kernel that singularity does not like.
My uname -a output is: Linux testing-VirtualBox 4.4.0-98-generic #121~14.04.1-Ubuntu SMP Wed Oct 11 11:54:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Thank you very much.
I'll try to update Linux kernel shortly.

I've followed the advice to change the kernel but an application was still failing.
Looking more attentively to the error message("singularity is not found") I decided to recreate "boinc" user from the scratch. That did the magic and now native application is working fine.
My guess - something was wrong in boinc user PATH or something like that.
Thanks for advice now I'm happy again.