LHC@home

@all volunteers having problems with Theory native.

Some errors are caused by an installed but misconfigured CVMFS.

Before you request any task you may check if "cvmfs_config probe" returns OK like the following example that has configured all repositories required for ATLAS and Theory.

cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK


If any of those is missing, include it in /etc/cvmfs/default.local.
If any of those fails, post a message here.

FWIW, /cvmfs is completely empty. This is what's in /etc/cvmfs/default.local:

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY=DIRECT

Now what?

Edit: more info

$ cvmfs_config chksetup
Warning: CernVM-FS map is not referenced from autofs maps (/etc/auto.master)

Did you follow the hints here?
https://cernvm.cern.ch/portal/filesystem/quickstart
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html



Especially: did you run "cvmfs_config setup" once when you installed CVMFS?


In addition you may check the automounter config.
1. Check if a file /etc/auto.cvmfs exists
2. Check if it is enabled in /etc/auto.master
3. restart your automounter (or reboot), e.g. with "systemd restart autofs.service" (depending on your linux distro)
4. check if /cvmfs is mounted
5. check again "cvmfs_config probe"


<edit>
links activated
</edit>

$ cvmfs_config chksetup
Warning: CernVM-FS map is not referenced from autofs maps (/etc/auto.master)

AutoFS has not been setup for CVMFS and the file /etc/auto.master.d/cvmfs.autofs is missing. It has the following contents:

# automatically generated by CernVM-FS
/cvmfs /etc/auto.cvmfs

This file is generated when you run sudo cvmfs_config setup

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch

ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch
You may add it and run "cvmfs_config reload".
Without it some tasks may fail depending on the app you get.

Did you follow the hints here?
https://cernvm.cern.ch/portal/filesystem/quickstart
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html

Of course, I followed the instructions under the Debian/Ubuntu section.

Especially: did you run "cvmfs_config setup" once when you installed CVMFS?

Ah! I just now searched around for that string on the instructions page, and I see why I missed it. The instructions to do the "cvmfs_config setup" looks like it is in the MacOS section, so I didn't think it applied to what I was doing with Ubuntu. Maybe something can be added to the "cvmfs_config setup" section that says "ALL OSs must do the following? Or something like that?

In any case, it looks like it should work now. But I am locked out for a day, and am unable to test it.
$cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch

ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch
You may add it and run "cvmfs_config reload".
Without it some tasks may fail depending on the app you get.

Thanks. Added and reloaded.

I have attempted to install the cvmfs software on one of my systems, but have been unsuccessful at getting it to work. I have managed to get it to the point the cvmfs_config probe is successful, but it looks like the runc instance offered up from grid.cern.ch requires a newer version of some components than seems to have been back-ported to Ubuntu 14.04. Can somebody confirm if it's expected to work under an Ubuntu 14.04 LTS Operating System? Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
$

https://lhcathome.cern.ch/lhcathome/result.php?resultid=219421137

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
23:46:01 (3124): wrapper (7.15.26016): starting
23:46:01 (3124): wrapper (7.15.26016): starting
23:46:01 (3124): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.28 ()
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Detected TheoryN App
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Checking CVMFS.
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Checking runc.
/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version
04:46:01 2019-03-19: cranky-0.0.28: [ERROR] 'runc -v' failed.
23:46:02 (3124): cranky exited; CPU time 0.020000
23:46:02 (3124): app exit status: 0xce
23:46:02 (3124): called boinc_finish(195)

</stderr_txt>
]]>

Ah! I just now searched around for that string on the instructions page, and I see why I missed it. The instructions to do the "cvmfs_config setup" looks like it is in the MacOS section, so I didn't think it applied to what I was doing with Ubuntu. Maybe something can be added to the "cvmfs_config setup" section that says "ALL OSs must do the following? Or something like that?

Thanks for the feedback, I will update the page.

In any case, it looks like it should work now. But I am locked out for a day, and am unable to test it.

Unlocked. I am considering putting a manual unlock button on the host page.

ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch

Thanks for spotting that! I have added it to the file.

/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version

run "ldd /cvmfs/grid.cern.ch/vc/containers/runc" to check if all libs are installed (here: libseccomp.so.2).
If any is missing, install it from your distro's repository.

I have attempted to install the cvmfs software on one of my systems, but have been unsuccessful at getting it to work. I have managed to get it to the point the cvmfs_config probe is successful, but it looks like the runc instance offered up from grid.cern.ch requires a newer version of some components than seems to have been back-ported to Ubuntu 14.04. Can somebody confirm if it's expected to work under an Ubuntu 14.04 LTS Operating System? Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.

I configured some clients to request CMS and Theory native but without manual intervention they only get Theory tasks.
(Posted here as I guess Laurence follows this thread)

/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version

run "ldd /cvmfs/grid.cern.ch/vc/containers/runc" to check if all libs are installed (here: libseccomp.so.2).
If any is missing, install it from your distro's repository.

$ ldd -v /cvmfs/grid.cern.ch/vc/containers/runc
        linux-vdso.so.1 =>  (0x00007fff8d12a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff8e2cd8000)
        libseccomp.so.2 => /usr/lib/x86_64-linux-gnu/libseccomp.so.2 (0x00007ff8e2ab8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff8e28b0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff8e24e0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff8e3860000)

        Version information:
        /cvmfs/grid.cern.ch/vc/containers/runc:
                libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
                libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /usr/lib/x86_64-linux-gnu/libseccomp.so.2:
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libdl.so.2:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

I'll presume that 14.04 LTS doesn't fall under the category of "relatively recent kernel that supports rootless containers" :) But is 16.04 new enough, or should I go right to 18.04?

Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.

It works with 16.04.5

I configured some clients to request CMS and Theory native but without manual intervention they only get Theory tasks.

I have requested both ATLAS and Theory (with "Run test applications?" selected), but am getting only Theory.
I assumed that was just because they have assigned a higher priority to Theory for testing purposes. It is OK with me.

It's working for me now. Thanks for all the help!

Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.

It works with 16.04.5

Thanks for the confirmation Jim1348. I think I'll give 18.04.2 a quick try, but at least I've installed 16.04 once or twice before so it's nice to know at least that will work.

I did the idiot-/foolproof test. Poweroff without shutdown.
Here are the 4 affected results:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=219410585
https://lhcathome.cern.ch/lhcathome/result.php?resultid=219410572
https://lhcathome.cern.ch/lhcathome/result.php?resultid=219410568
https://lhcathome.cern.ch/lhcathome/result.php?resultid=219410213

Tried "sudo touch /var/lib/boinc/slots/0/cernvm/shared/shutdown" to gracefully kill a hung sherpa job. The command succeeded but the task didn't shutdown.
Is graceful shutdown not an option with native Theory or am I creating the file in the wrong folder?

Is graceful shutdown not an option with native Theory or am I creating the file in the wrong folder?

No, there is no graceful shutdown and no time limit of 18 hours.
Only the BOINC fpops, memory and disk bound could kill the task (or a user abort), resulting in the ERROR-state without granting credits.