Message boards : Theory Application : Issues Native Theory application
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
zombie67 [MM]
Avatar

Send message
Joined: 24 Nov 06
Posts: 76
Credit: 7,914,481
RAC: 27,114
Message 38300 - Posted: 19 Mar 2019, 14:37:57 UTC - in response to Message 38299.  
Last modified: 19 Mar 2019, 14:45:29 UTC

@all volunteers having problems with Theory native.

Some errors are caused by an installed but misconfigured CVMFS.

Before you request any task you may check if "cvmfs_config probe" returns OK like the following example that has configured all repositories required for ATLAS and Theory.

cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK


If any of those is missing, include it in /etc/cvmfs/default.local.
If any of those fails, post a message here.

FWIW, /cvmfs is completely empty. This is what's in /etc/cvmfs/default.local:

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch
CVMFS_QUOTA_LIMIT=4096
CVMFS_CACHE_BASE=/scratch/cvmfs
CVMFS_HTTP_PROXY=DIRECT

Now what?

Edit: more info

$ cvmfs_config chksetup
Warning: CernVM-FS map is not referenced from autofs maps (/etc/auto.master)
ID: 38300 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,907,232
RAC: 137,947
Message 38301 - Posted: 19 Mar 2019, 14:53:26 UTC
Last modified: 19 Mar 2019, 14:55:04 UTC

Did you follow the hints here?
https://cernvm.cern.ch/portal/filesystem/quickstart
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html



Especially: did you run "cvmfs_config setup" once when you installed CVMFS?


In addition you may check the automounter config.
1. Check if a file /etc/auto.cvmfs exists
2. Check if it is enabled in /etc/auto.master
3. restart your automounter (or reboot), e.g. with "systemd restart autofs.service" (depending on your linux distro)
4. check if /cvmfs is mounted
5. check again "cvmfs_config probe"


<edit>
links activated
</edit>
ID: 38301 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 372
Credit: 238,712
RAC: 0
Message 38302 - Posted: 19 Mar 2019, 14:58:35 UTC - in response to Message 38300.  


$ cvmfs_config chksetup
Warning: CernVM-FS map is not referenced from autofs maps (/etc/auto.master)


AutoFS has not been setup for CVMFS and the file /etc/auto.master.d/cvmfs.autofs is missing. It has the following contents:
# automatically generated by CernVM-FS
/cvmfs /etc/auto.cvmfs


This file is generated when you run sudo cvmfs_config setup
ID: 38302 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,907,232
RAC: 137,947
Message 38303 - Posted: 19 Mar 2019, 15:01:01 UTC - in response to Message 38300.  

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch

ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch
You may add it and run "cvmfs_config reload".
Without it some tasks may fail depending on the app you get.
ID: 38303 · Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 24 Nov 06
Posts: 76
Credit: 7,914,481
RAC: 27,114
Message 38304 - Posted: 19 Mar 2019, 15:07:57 UTC - in response to Message 38301.  
Last modified: 19 Mar 2019, 15:13:19 UTC

Did you follow the hints here?
https://cernvm.cern.ch/portal/filesystem/quickstart
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html

Of course, I followed the instructions under the Debian/Ubuntu section.

Especially: did you run "cvmfs_config setup" once when you installed CVMFS?


Ah! I just now searched around for that string on the instructions page, and I see why I missed it. The instructions to do the "cvmfs_config setup" looks like it is in the MacOS section, so I didn't think it applied to what I was doing with Ubuntu. Maybe something can be added to the "cvmfs_config setup" section that says "ALL OSs must do the following? Or something like that?

In any case, it looks like it should work now. But I am locked out for a day, and am unable to test it.
$cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
ID: 38304 · Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 24 Nov 06
Posts: 76
Credit: 7,914,481
RAC: 27,114
Message 38305 - Posted: 19 Mar 2019, 15:13:03 UTC - in response to Message 38303.  
Last modified: 19 Mar 2019, 15:13:28 UTC

CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch

ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch
You may add it and run "cvmfs_config reload".
Without it some tasks may fail depending on the app you get.

Thanks. Added and reloaded.
ID: 38305 · Report as offensive     Reply Quote
Mumps [MM]

Send message
Joined: 7 Apr 11
Posts: 4
Credit: 56,682,229
RAC: 0
Message 38306 - Posted: 19 Mar 2019, 15:15:03 UTC

I have attempted to install the cvmfs software on one of my systems, but have been unsuccessful at getting it to work. I have managed to get it to the point the cvmfs_config probe is successful, but it looks like the runc instance offered up from grid.cern.ch requires a newer version of some components than seems to have been back-ported to Ubuntu 14.04. Can somebody confirm if it's expected to work under an Ubuntu 14.04 LTS Operating System? Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
$

https://lhcathome.cern.ch/lhcathome/result.php?resultid=219421137

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
23:46:01 (3124): wrapper (7.15.26016): starting
23:46:01 (3124): wrapper (7.15.26016): starting
23:46:01 (3124): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.28 ()
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Detected TheoryN App
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Checking CVMFS.
04:46:01 2019-03-19: cranky-0.0.28: [INFO] Checking runc.
/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version
04:46:01 2019-03-19: cranky-0.0.28: [ERROR] 'runc -v' failed.
23:46:02 (3124): cranky exited; CPU time 0.020000
23:46:02 (3124): app exit status: 0xce
23:46:02 (3124): called boinc_finish(195)

</stderr_txt>
]]>
ID: 38306 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 372
Credit: 238,712
RAC: 0
Message 38307 - Posted: 19 Mar 2019, 15:24:43 UTC - in response to Message 38304.  
Last modified: 19 Mar 2019, 15:25:13 UTC


Ah! I just now searched around for that string on the instructions page, and I see why I missed it. The instructions to do the "cvmfs_config setup" looks like it is in the MacOS section, so I didn't think it applied to what I was doing with Ubuntu. Maybe something can be added to the "cvmfs_config setup" section that says "ALL OSs must do the following? Or something like that?

Thanks for the feedback, I will update the page.

In any case, it looks like it should work now. But I am locked out for a day, and am unable to test it.

Unlocked. I am considering putting a manual unlock button on the host page.
ID: 38307 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 372
Credit: 238,712
RAC: 0
Message 38308 - Posted: 19 Mar 2019, 15:26:43 UTC - in response to Message 38303.  


ALICE is missing.
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch,cernvm-prod.cern.ch,sft.cern.ch,alice.cern.ch

Thanks for spotting that! I have added it to the file.
ID: 38308 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,907,232
RAC: 137,947
Message 38309 - Posted: 19 Mar 2019, 15:27:09 UTC - in response to Message 38306.  

/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version

run "ldd /cvmfs/grid.cern.ch/vc/containers/runc" to check if all libs are installed (here: libseccomp.so.2).
If any is missing, install it from your distro's repository.
ID: 38309 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 372
Credit: 238,712
RAC: 0
Message 38310 - Posted: 19 Mar 2019, 15:31:04 UTC - in response to Message 38306.  

I have attempted to install the cvmfs software on one of my systems, but have been unsuccessful at getting it to work. I have managed to get it to the point the cvmfs_config probe is successful, but it looks like the runc instance offered up from grid.cern.ch requires a newer version of some components than seems to have been back-ported to Ubuntu 14.04. Can somebody confirm if it's expected to work under an Ubuntu 14.04 LTS Operating System? Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.
ID: 38310 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,907,232
RAC: 137,947
Message 38311 - Posted: 19 Mar 2019, 15:41:14 UTC
Last modified: 19 Mar 2019, 15:42:03 UTC

I configured some clients to request CMS and Theory native but without manual intervention they only get Theory tasks.
(Posted here as I guess Laurence follows this thread)
ID: 38311 · Report as offensive     Reply Quote
Mumps [MM]

Send message
Joined: 7 Apr 11
Posts: 4
Credit: 56,682,229
RAC: 0
Message 38312 - Posted: 19 Mar 2019, 15:55:45 UTC - in response to Message 38309.  
Last modified: 19 Mar 2019, 15:56:59 UTC

/cvmfs/grid.cern.ch/vc/containers/runc: symbol lookup error: /cvmfs/grid.cern.ch/vc/containers/runc: undefined symbol: seccomp_version

run "ldd /cvmfs/grid.cern.ch/vc/containers/runc" to check if all libs are installed (here: libseccomp.so.2).
If any is missing, install it from your distro's repository.

$ ldd -v /cvmfs/grid.cern.ch/vc/containers/runc
        linux-vdso.so.1 =>  (0x00007fff8d12a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff8e2cd8000)
        libseccomp.so.2 => /usr/lib/x86_64-linux-gnu/libseccomp.so.2 (0x00007ff8e2ab8000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff8e28b0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff8e24e0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff8e3860000)

        Version information:
        /cvmfs/grid.cern.ch/vc/containers/runc:
                libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
                libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /usr/lib/x86_64-linux-gnu/libseccomp.so.2:
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libdl.so.2:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2


I'll presume that 14.04 LTS doesn't fall under the category of "relatively recent kernel that supports rootless containers" :) But is 16.04 new enough, or should I go right to 18.04?
ID: 38312 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 38313 - Posted: 19 Mar 2019, 16:00:32 UTC - in response to Message 38310.  

Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.

It works with 16.04.5
ID: 38313 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 38314 - Posted: 19 Mar 2019, 16:03:30 UTC - in response to Message 38311.  

I configured some clients to request CMS and Theory native but without manual intervention they only get Theory tasks.

I have requested both ATLAS and Theory (with "Run test applications?" selected), but am getting only Theory.
I assumed that was just because they have assigned a higher priority to Theory for testing purposes. It is OK with me.
ID: 38314 · Report as offensive     Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 24 Nov 06
Posts: 76
Credit: 7,914,481
RAC: 27,114
Message 38315 - Posted: 19 Mar 2019, 16:10:36 UTC

It's working for me now. Thanks for all the help!
ID: 38315 · Report as offensive     Reply Quote
Mumps [MM]

Send message
Joined: 7 Apr 11
Posts: 4
Credit: 56,682,229
RAC: 0
Message 38317 - Posted: 19 Mar 2019, 16:34:23 UTC - in response to Message 38313.  

Or what is the oldest Ubuntu that it should work with, so I know if I should spend the time installing 16.04 or 18.04 LTS's.

It needs a relatively recent kernel that supports rootless containers.

It works with 16.04.5

Thanks for the confirmation Jim1348. I think I'll give 18.04.2 a quick try, but at least I've installed 16.04 once or twice before so it's nice to know at least that will work.
ID: 38317 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 38319 - Posted: 19 Mar 2019, 17:34:59 UTC

ID: 38319 · Report as offensive     Reply Quote
bronco

Send message
Joined: 13 Apr 18
Posts: 443
Credit: 8,438,885
RAC: 0
Message 38320 - Posted: 19 Mar 2019, 18:03:51 UTC

Tried "sudo touch /var/lib/boinc/slots/0/cernvm/shared/shutdown" to gracefully kill a hung sherpa job. The command succeeded but the task didn't shutdown.
Is graceful shutdown not an option with native Theory or am I creating the file in the wrong folder?
ID: 38320 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 38322 - Posted: 19 Mar 2019, 18:43:41 UTC - in response to Message 38320.  

Is graceful shutdown not an option with native Theory or am I creating the file in the wrong folder?
No, there is no graceful shutdown and no time limit of 18 hours.
Only the BOINC fpops, memory and disk bound could kill the task (or a user abort), resulting in the ERROR-state without granting credits.
ID: 38322 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Theory Application : Issues Native Theory application


©2024 CERN