1) Message boards : Number crunching : Setting up a local Squid to work with LHC@home - Comments and Questions (Message 45610)
Posted 4 Nov 2021 by [AF] Hydrosaure
Post:
With Squid version 5.0.6 I'm having difficulties uploading results to LHC.
In the logs I seem to be getting a lot of messages like these:

1635870641.453    135 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.101.107 text/plain
1635870643.445    120 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.101.107 -
1635871151.414    332 192.168.3.69 TCP_MISS/200 329 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.125.226 text/plain
1635871151.414    332 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.125.226 text/plain
1635871153.135    129 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.125.226 -
1635871153.137    132 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.125.226 -
1635874194.290    155 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 text/plain
1635874194.301    165 192.168.3.69 TCP_MISS/200 353 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 text/plain
1635874196.274    134 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 -
1635874196.284    144 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 -
1635877114.137     90 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 text/plain
1635877114.137     90 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 text/plain
1635877115.554    101 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 -
1635877115.561    108 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.75.236 -
1635883410.768    157 192.168.3.69 TCP_MISS/200 329 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.86.60 text/plain
1635883410.769    158 192.168.3.69 TCP_MISS/200 348 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.86.60 text/plain
1635883412.859    104 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.86.60 -
1635883412.884    129 192.168.3.69 TCP_MISS_ABORTED/100 0 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.86.60 -


Downgrading to version 4.17 solved the blocked upload.

Any idea what has evolved between these versions to explain this ?
2) Message boards : ATLAS application : Debian 11 (Message 45219)
Posted 17 Aug 2021 by [AF] Hydrosaure
Post:
Happy to report a working and completed ATLAS job on a fresh Debian 11 install.

Did you the same .deb as from a Debian 10.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=323787719
3) Message boards : Number crunching : Recommended CVMFS Configuration for Native Apps - Comments and Questions (Message 44240)
Posted 31 Jan 2021 by [AF] Hydrosaure
Post:
Hello,

My systems are all configured to use at least 4GB of local CVMFS cache and according to stats, they seem to reach pretty high hit ratios.
Here is just a look at a couple of them:


Would a proxy still help here ?

Also are there specific Squid options one should set in regards to caching CVMFS data (ie. max object size, retention or refresh policy) ?
4) Message boards : ATLAS application : error on Atlas native: 195 (0x000000C3) EXIT_CHILD_FAILED (Message 44236)
Posted 30 Jan 2021 by [AF] Hydrosaure
Post:
Thank you, will read and adjust my configs accordingly.
5) Message boards : ATLAS application : error on Atlas native: 195 (0x000000C3) EXIT_CHILD_FAILED (Message 44231)
Posted 30 Jan 2021 by [AF] Hydrosaure
Post:
...and yes installing a proxy is on my todo list :-)

+1

Getting off topic but while we're on the subject of configuring a proxy, is CVMFS cache (ie CVMFS_QUOTA_LIMIT=4096) still required in the case of going through a caching proxy ?
6) Message boards : ATLAS application : error on Atlas native: 195 (0x000000C3) EXIT_CHILD_FAILED (Message 44225)
Posted 29 Jan 2021 by [AF] Hydrosaure
Post:
On this test machine, configuration is now as follows:
$ cat /etc/cvmfs/default.local
CVMFS_USE_CDN=yes
# BEGIN ANSIBLE MANAGED BLOCK
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/var/lib/cvmfs
CVMFS_HTTP_PROXY=DIRECT
CVMFS_SEND_INFO_HEADER=yes
# END ANSIBLE MANAGED BLOCK


After a while I ran

$ sudo cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.5.0 1596 22 23752 14 2 1 3189293 4194304 0 65024 0 194 100 3 1 http://s1cern-cvmfs.openhtc.io/cvmfs/cvmfs-config.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.5.0 1687 22 36640 78110 2 75 3189293 4194304 559 65024 0 10889 99.8898 278 330 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat atlas-condb.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.5.0 1811 22 24372 8534 2 1 3189293 4194304 0 65024 0 13 92.3077 66 102 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas-condb.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.5.0 2186 21 24976 19547 2 5 3189293 4194304 7 65024 0 209 99.5215 3 5 http://s1cern-cvmfs.openhtc.io/cvmfs/sft.cern.ch DIRECT 1


Seems to be using openhtc.io if I read this right.

and yes installing a proxy is on my todo list :-)
7) Message boards : ATLAS application : error on Atlas native: 195 (0x000000C3) EXIT_CHILD_FAILED (Message 44218)
Posted 28 Jan 2021 by [AF] Hydrosaure
Post:
Can confirm that with a self built version of Singularity (version 3.7.1) ATLAS works correctly on a fresh Ubuntu 20.04.

A result as example: https://lhcathome.cern.ch/lhcathome/result.php?resultid=297199728
8) Questions and Answers : Unix/Linux : Getting CVMFS to work under Linux Containers (LXC) (Message 40718)
Posted 29 Nov 2019 by [AF] Hydrosaure
Post:
I have setup a Boinc VM running as a container under LXC on Proxmox.
All projects seem to work fine except for CERN and the CVMFS based apps.

I've changed the container to run with an "unconfined" profile still all tasks requiring CVMFS fail at even detecting cvmfs program

Example log:
08:11:11 (12658): wrapper (7.15.26016): starting
08:11:11 (12658): wrapper (7.15.26016): starting
08:11:11 (12658): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.29 ()
08:11:12 UTC +00:00 2019-11-29: cranky-0.0.29: [INFO] Detected Theory App
08:11:12 UTC +00:00 2019-11-29: cranky-0.0.29: [INFO] Checking CVMFS.
08:11:13 UTC +00:00 2019-11-29: cranky-0.0.29: [ERROR] 'cvmfs_config probe sft.cern.ch' failed.
08:11:13 (12658): cranky exited; CPU time 0.083059
08:11:13 (12658): app exit status: 0xce
08:11:13 (12658): called boinc_finish(195)


A "cvmfs_config probe" returns OK for all targets when ran manually.

Any help appreciated.

Host in question => https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10624114
9) Questions and Answers : Unix/Linux : ATLAS (native_mt) Python incompatibility on Linux (Message 40607)
Posted 23 Nov 2019 by [AF] Hydrosaure
Post:
Latest version of native ATALS is still using Python2; it is starting to become a pressing matter seeing the EOL of that version.
10) Message boards : ATLAS application : ATLAS native version 2.73 (Message 40600)
Posted 22 Nov 2019 by [AF] Hydrosaure
Post:
Any idea what could be the issue here ?

https://lhcathome.cern.ch/lhcathome/result.php?resultid=252658162

2019-11-22 15:36:27,522: Checking singularity with cmd:/cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
2019-11-22 15:37:43,370: Singularity isnt working: INFO:    Convert SIF file to sandbox...
INFO:    Cleaning up image...
FATAL:   container creation failed: mount ->/cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc error: can't remount /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc: invalid argument
2019-11-22 15:37:45,377: running start_atlas return value is 3
2019-11-22 15:47:45,448: parent process exit 3
2019-11-22 15:47:45,449: child process exit 3


Yet if I manually try to execute this command line

boinc@boinc-z87:~$ /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/current/bin/singularity --debug exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img hostname
VERBOSE [U=1000,P=5653]    execStarter()                 starter-suid not found, using user namespace
VERBOSE [U=1000,P=5653]    execStarter()                 User namespace requested, convert image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img to sandbox
INFO    [U=1000,P=5653]    execStarter()                 Convert SIF file to sandbox...
DEBUG   [U=1000,P=5653]    Init()                        Entering image format intializer
DEBUG   [U=1000,P=5653]    Init()                        Check for image format sif
VERBOSE [U=1000,P=5653]    print()                       Set messagelevel to: 5
DEBUG   [U=1000,P=5653]    init()                        PIPE_EXEC_FD value: 7
VERBOSE [U=1000,P=5653]    init()                        Container runtime
VERBOSE [U=1000,P=5653]    is_suid()                     Check if we are running as setuid
DEBUG   [U=1000,P=5653]    init()                        Read json configuration from pipe
DEBUG   [U=1000,P=5653]    init()                        Set child signal mask
DEBUG   [U=1000,P=5653]    init()                        Wait completion of stage1
DEBUG   [U=1000,P=15975]   set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=15975]   init()                        Spawn stage 1
VERBOSE [U=1000,P=15975]   startup()                     Execute stage 1
DEBUG   [U=1000,P=15975]   Stage()                       Entering stage 1
DEBUG   [U=1000,P=15975]   prepareFd()                   Open file descriptor for /cvmfs
DEBUG   [U=1000,P=15975]   Init()                        Entering image format intializer
DEBUG   [U=1000,P=15975]   Init()                        Check for image format sif
DEBUG   [U=1000,P=15975]   Init()                        sif format initializer returns: not a SIF file image
DEBUG   [U=1000,P=15975]   Init()                        Check for image format sandbox
DEBUG   [U=1000,P=5653]    init()                        Create socketpair for master communication channel
DEBUG   [U=1000,P=5653]    cleanup_fd()                  Check file descriptor /proc/self/fd/3 pointing to /cvmfs
DEBUG   [U=1000,P=5653]    cleanup_fd()                  Check file descriptor /proc/self/fd/4 pointing to anon_inode:[eventpoll]
DEBUG   [U=1000,P=5653]    cleanup_fd()                  Closing /proc/self/fd/4
DEBUG   [U=1000,P=5653]    cleanup_fd()                  Check file descriptor /proc/self/fd/5 pointing to /tmp/rootfs-490992350
VERBOSE [U=1000,P=5653]    user_namespace_init()         Create user namespace
DEBUG   [U=65534,P=5653]   setup_userns_mappings()       Write deny to set group file
DEBUG   [U=65534,P=5653]   setup_userns_mappings()       Write to GID map
DEBUG   [U=65534,P=5653]   setup_userns_mappings()       Write to UID map
VERBOSE [U=1000,P=5653]    create_namespace()            Create mount namespace
DEBUG   [U=1000,P=5653]    init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=5653]    init()                        Spawn master process
DEBUG   [U=1000,P=16013]   set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=16013]   create_namespace()            Create mount namespace
DEBUG   [U=1000,P=16013]   set_parent_death_signal()     Set parent death signal to 9
DEBUG   [U=1000,P=16013]   prepare_stage()               Entering in stage 2
VERBOSE [U=1000,P=16014]   init()                        Spawn RPC server
VERBOSE [U=1000,P=16014]   startup()                     Serve RPC requests
VERBOSE [U=1000,P=5653]    startup()                     Execute master process
DEBUG   [U=1000,P=5653]    setupSessionLayout()          Attempting to use underlay (enable underlay = yes)
DEBUG   [U=1000,P=5653]    setupUnderlayLayout()         Creating underlay SESSIONDIR layout
DEBUG   [U=1000,P=5653]    addRootfsMount()              Mount rootfs in read-only mode
DEBUG   [U=1000,P=5653]    addRootfsMount()              Mounting directory rootfs: /tmp/rootfs-490992350
DEBUG   [U=1000,P=5653]    addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=1000,P=5653]    addKernelMount()              Adding proc to mount list
VERBOSE [U=1000,P=5653]    addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=1000,P=5653]    addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=1000,P=5653]    addKernelMount()              Adding sysfs to mount list
VERBOSE [U=1000,P=5653]    addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=1000,P=5653]    addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=1000,P=5653]    addDevMount()                 Adding dev to mount list
VERBOSE [U=1000,P=5653]    addDevMount()                 Default mount: /dev:/dev
DEBUG   [U=1000,P=5653]    addHostMount()                Not mounting host file systems per configuration
VERBOSE [U=1000,P=5653]    addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=5653]    addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=1000,P=5653]    addHomeStagingDir()           Staging home directory (/home/boinc) at /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc
DEBUG   [U=1000,P=5653]    addHomeMount()                Adding home directory mount [/cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc:/home/boinc] to list using layer: underlay
DEBUG   [U=1000,P=5653]    isLayerEnabled()              Using Layer system: underlay
DEBUG   [U=1000,P=5653]    addUserbindsMount()           Adding /cvmfs to mount list
DEBUG   [U=1000,P=5653]    addUserbindsMount()           Checking for 'user bind control' in configuration file
DEBUG   [U=1000,P=5653]    addTmpMount()                 Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=5653]    addTmpMount()                 Default mount: /tmp:/tmp
VERBOSE [U=1000,P=5653]    addTmpMount()                 Default mount: /var/tmp:/var/tmp
DEBUG   [U=1000,P=5653]    addScratchMount()             Not mounting scratch directory: Not requested
VERBOSE [U=1000,P=5653]    addCwdMount()                 Default mount: /home/boinc: to the container
DEBUG   [U=1000,P=5653]    addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=1000,P=5653]    addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=5653]    addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=1000,P=5653]    addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=1000,P=5653]    create()                      Mount all
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting tmpfs to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /tmp/rootfs-490992350 to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/.singularity.d to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/.singularity.d
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/alrb to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/alrb
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/anaconda-post.log to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/anaconda-post.log
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/cvmfs to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/cvmfs
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/dev to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/dev
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/docker-creation-date to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/docker-creation-date
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/etc to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/etc
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/gpfs to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/gpfs
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/media to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/media
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/mnt to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/mnt
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/opt to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/opt
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/proc to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/proc
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/root to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/root
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/run to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/run
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/scratch to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/scratch
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/singularity-creation-date to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/singularity-creation-date
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/srv to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/srv
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/sys to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/sys
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/tmp to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/tmp
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/usr to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/usr
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/var to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay/var
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/underlay to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final
DEBUG   [U=1000,P=5653]    setPropagationMount()         Set RPC mount propagation flag to SLAVE
VERBOSE [U=1000,P=5653]    Passwd()                      Checking for template passwd file: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=1000,P=5653]    Passwd()                      Creating passwd content
VERBOSE [U=1000,P=5653]    Passwd()                      Creating template passwd file and appending user data: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG   [U=1000,P=5653]    addIdentityMount()            Adding /etc/passwd to mount list
VERBOSE [U=1000,P=5653]    addIdentityMount()            Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=1000,P=5653]    Group()                       Checking for template group file: /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=1000,P=5653]    Group()                       Creating group content
DEBUG   [U=1000,P=5653]    addIdentityMount()            Adding /etc/group to mount list
VERBOSE [U=1000,P=5653]    addIdentityMount()            Default mount: /etc/group:/etc/group
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /dev to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/dev
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /etc/localtime to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/usr/share/zoneinfo/UTC
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /etc/hosts to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/etc/hosts
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/etc/singularity/actions to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/.singularity.d/actions
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/.singularity.d/actions
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /proc to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/proc
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /sys to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/sys
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /home/boinc to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/home/boinc to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/home/boinc
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/home/boinc
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /tmp to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/tmp
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/tmp
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /var/tmp to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/var/tmp
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/var/tmp
DEBUG   [U=1000,P=5653]    mountGeneric()                Skipping mount /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/home/boinc, /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/home/boinc already mounted
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/etc/resolv.conf to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/etc/passwd to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/etc/passwd
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/etc/group to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/etc/group
DEBUG   [U=1000,P=5653]    mountGeneric()                Mounting /cvmfs to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/cvmfs
DEBUG   [U=1000,P=5653]    mountGeneric()                Remounting /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final/cvmfs
DEBUG   [U=1000,P=5653]    create()                      Chroot into /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final
DEBUG   [U=1000,P=16014]   Chroot()                      Change current directory to /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final
DEBUG   [U=1000,P=16014]   Chroot()                      Hold reference to host / directory
DEBUG   [U=1000,P=16014]   Chroot()                      Called pivot_root on /cvmfs/atlas.cern.ch/repo/containers/sw/singularity/x86_64-el7/3.2.1/var/singularity/mnt/session/final
DEBUG   [U=1000,P=16014]   Chroot()                      Change current directory to host / directory
DEBUG   [U=1000,P=16014]   Chroot()                      Apply slave mount propagation for host / directory
DEBUG   [U=1000,P=16014]   Chroot()                      Called unmount(/, syscall.MNT_DETACH)
DEBUG   [U=1000,P=16014]   Chroot()                      Changing directory to / to avoid getpwd issues
DEBUG   [U=1000,P=5653]    create()                      Chdir into / to avoid errors
VERBOSE [U=1000,P=16013]   startup()                     Execute stage 2
DEBUG   [U=1000,P=16013]   Stage()                       Entering stage 2
DEBUG   [U=1000,P=5653]    PostStartProcess()            Post start process
boinc-z87
DEBUG   [U=1000,P=5653]    CleanupContainer()            Cleanup container
VERBOSE [U=1000,P=5653]    CleanupContainer()            Removing image /tmp/rootfs-490992350
INFO    [U=1000,P=5653]    CleanupContainer()            Cleaning up image...
DEBUG   [U=1000,P=5653]    Master()                      Child exited with exit status 0
boinc@boinc-z87:~$
11) Message boards : ATLAS application : native_mt app on Ubuntu 18.08 (Message 36580)
Posted 28 Aug 2018 by [AF] Hydrosaure
Post:
I ran both commands; they complete correctly.
Good catch with the thread number, I've lowered it to 4 for now.

Digging some more into system logs, I've found these errors
Aug 28 18:22:12 boinc-ab350m Singularity: action-suid (U=1000,P=19539)> Home directory is not owned by calling user: /home/boinc
Aug 28 18:22:12 boinc-ab350m Singularity: action-suid (U=1000,P=19539)> Retval = 255

yet
boinc@boinc-ab350m:/var/lib/boinc/slots$ ls -ld /home/boinc/
drwxr-xr-x 8 boinc boinc 23 Aug 28 19:25 /home/boinc/

I've also tried what is listed under
http://opensciencegrid.org/docs/worker-node/install-singularity/#validating-singularity and it looks ok

boinc@boinc-ab350m:/var/lib/boinc/slots$ singularity exec --contain --ipc --pid \
>                 --home $PWD:/srv \
>                 --bind /cvmfs \
>                 /cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo:el6 \
>                 ps -ef
WARNING: Container does not have an exec helper script, calling 'ps' directly
UID        PID  PPID  C STIME TTY          TIME CMD
boinc        1     0  0 17:32 ?        00:00:00 ps -ef
12) Message boards : ATLAS application : native_mt app on Ubuntu 18.08 (Message 36571)
Posted 27 Aug 2018 by [AF] Hydrosaure
Post:
Anyone got native ATLAS app to run on Ubuntu 18.04 LTS ?

I have a Ryzen machine running this OS and am trying to get it to crunch.
cvmfs is installed and singularity 2.5.2 as well.
Still tasks fail after about 10 minutes...

Looking at the logs, it fails at this step:
check singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname

Singularity isnt working...

running start_atlas return value is 3


Yet manually running that command it seems fine:
boinc@boinc-ab350m:/var/lib/boinc/slots$ singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img host
name
boinc-ab350m
boinc@boinc-ab350m:/var/lib/boinc/slots$ echo $?
0
boinc@boinc-ab350m:/var/lib/boinc/slots$


Any idea?

Host link: https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10510464

Example of failed task: https://lhcathome.cern.ch/lhcathome/result.php?resultid=205978265



©2021 CERN