LHC@home

Hi all,

I'm running under Linux debian 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux ...

... and I've been trying to make ATLAS native work, desperately.

I've done my best to install cvmfs and singularity and it seems okay.
I followed the instructions given at cvmfs.readthedocs.io and some similar website for singularity.

Basically, a ATLAS job is running, percentage is going up, but when I look at the properties of this job via BOINC it doesn't show any CPU time elapsed.
Moreover, the system monitor of Debian doesn't show any kind of CPU activities although I've set BOINC to use 2 CPUs and the amount of RAM used stay low.

Please, could you enlighten me on what's going on?

I really don't want to use Windows for BOINC as I'm under Linux for programming from now (mainly Oz language).

I started a SixTrack (avx) job earlier and it showed CPU time on the properties.

I am, of course, willing to give any logs necessary to fix this, willing to use another distro too for BOINC.

Best regards,

micropro

Welcome,
you have more connects for Atlas than needed.
2020-03-05 23:27:59] Probing /cvmfs/alice.cern.ch... OK
[2020-03-05 23:28:01] Probing /cvmfs/alice-ocdb.cern.ch... OK
[2020-03-05 23:28:03] Probing /cvmfs/atlas.cern.ch... OK
[2020-03-05 23:28:04] Probing /cvmfs/atlas-condb.cern.ch... OK
[2020-03-05 23:28:04] Probing /cvmfs/cms.cern.ch... OK
[2020-03-05 23:28:06] Probing /cvmfs/lhcb.cern.ch... OK
[2020-03-05 23:28:07] Probing /cvmfs/grid.cern.ch... OK
[2020-03-05 23:28:07] Probing /cvmfs/na61.cern.ch... OK
[2020-03-05 23:28:08] Probing /cvmfs/boss.cern.ch... OK
[2020-03-05 23:28:09] Probing /cvmfs/sft.cern.ch... OK
[2020-03-05 23:28:09] Probing /cvmfs/geant4.cern.ch... OK
atlas-nightlies.cern.ch and cernvm-prod.cern.ch and cvmfs-config.cern.ch are also needed.
squashfs is needed for your installation. There is a Error-message in your Atlas-Task.
In Atlas-Folder are more infos you need to run Atlas well. for Example Yeti's Checklist.

Thanks for your advice!

I've updated the default.local file and installed what I though necessary about squashfs.

However, I don't know what you mean by "Atlas-Folder"... I haven't figured out where the log files are... yet.

I've checked Yeti's Checklist, which seems much more helpfull for people running on Windows but if I understood it well, there's not definitive explanation concerning this issue.
Am I missing something about what he's saying ?

It seems that, depending on the job, the CPU time taken into account varies a lot.

I started a new ATLAS job to test after squashfs... all good.
Resuming my first ATLAS job... rather bad at first, then it began to consider the CPU time.

I won't say it's random, because there must be a reasonable explanation for that, but without knowing it... it's disturbing.

Anyway, I'll keep an eye on the numbers and probably edit my post at the end of the job.

Best regards,

micropro

A local singularity installation should be avoided since in the past there were runtime problems when the local singularity version was different from that version the ATLAS image had been created with.
If singularity is not required for other use it should completely be removed and the computer should be restarted.
ATLAS will then use the required singularity version from the CVMFS repository.


The basic CVMFS installation comes with a setup that is widely used inside the WLCG.
To use it for LHC@home some tweaks are recommended.

/etc/cvmfs/default.local

CVMFS_REPOSITORIES="atlas,atlas-condb,grid,cernvm-prod,sft,alice"
CVMFS_HTTP_PROXY="DIRECT" # only if no local proxy is used
CVMFS_QUOTA_LIMIT=6000 # slightly more than the default of 4000 MB
CVMFS_SEND_INFO_HEADER=yes

Additional repositories like atlas-nightlies will be configured and mounted automatically by the tasks.
Hence it should be avoided to configure them here.


/etc/cvmfs/domain.d/cern.ch.local

CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1unl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io/cvmfs/@fqrn@"

/etc/cvmfs/config.d/atlas-nightlies.cern.ch.local

CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@"

To activate the changes run "sudo cvmfs_config reload".
Then check with "sudo cvmfs_config probe" if you get "OKs" and with "sudo cvmfs_config stat" if your repositories use openhtc.io.
The output should look like this example:

Running /usr/bin/cvmfs_config stat grid.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.0.0 13975 20137 32932 11655 2 1 4483639 7077889 10 65024 0 7500 99.2267 189000 1238 http://s1cern-cvmfs.openhtc.io/cvmfs/grid.cern.ch DIRECT 1

Thank you very much !

I'll try that as soon as I can. Didn't know singularity should be avoided.

micropro

Here it goes...

Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... Failed!
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

How can I check I've done everything right ? Because I'm certain 100% that I followed every step advised.

Any help would be welcome.

micropro

Run "sudo cvmfs_config wipecache", then "sudo cvmfs_config reload" and "sudo cvmfs_config probe".

If you still get an error, run "sudo cvmfs_config showconfig -s cernvm-prod.cern.ch" and "sudo cvmfs_config chksetup".
If you can't solve the problem based on the output, post the output here.

Hi again, something's wrong, can't see what or why so, here's what I did and the output of it.

$ sudo cvmfs_config wipecache
Wiping out /var/lib/cvmfs/alice.cern.ch ... OK
Wiping out /var/lib/cvmfs/atlas-condb.cern.ch ... OK
Wiping out /var/lib/cvmfs/atlas.cern.ch ... OK
Wiping out /var/lib/cvmfs/cernvm-prod.cern.ch ... OK
Wiping out /var/lib/cvmfs/grid.cern.ch ... OK
Wiping out /var/lib/cvmfs/sft.cern.ch ... OK
Wiping out /var/lib/cvmfs/shared ... OK

Following by...

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/grid.cern.ch... Failed!
Probing /cvmfs/cernvm-prod.cern.ch... ^C

I stopped there as a failed was enough to investigate...

$ sudo cvmfs_config showconfig -s atlas.cern.ch

CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch    # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=DIRECT    # from /etc/cvmfs/default.local
CVMFS_IGNORE_SIGNATURE=no    # from /etc/cvmfs/default.conf
CVMFS_KEYS_DIR=/etc/cvmfs/keys    # from /etc/cvmfs/default.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_RW=no    # from /etc/cvmfs/default.conf
CVMFS_NFILES=65536    # from /etc/cvmfs/default.conf
CVMFS_NFS_SOURCE=no    # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS=http://wpad/wpad.dat    # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=6000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas,atlas-condb,grid,cernvm-prod,sft,alice    # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=yes    # from /etc/cvmfs/default.local
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1unl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=no    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf

______

$ sudo cvmfs_config chksetup
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys

Still I had to try this one...

$ sudo cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
df: .: Software caused connection abort
df: .: Transport endpoint is not connected
/usr/bin/cvmfs_config: line 912: +: syntax error: operand expected (error token is "+")

All probing return a 'Failed!'

Would it be of any help to post the content of default.conf and default.local ?
I know I must be annoying with that. I'd really appreciate some help but if tell me it's not worth it I could understand.

micropro

$ sudo cvmfs_config chksetup
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys
Error: no public keys in /etc/cvmfs/keys

Did you install the software package "cvmfs-config-default" that includes the missing keys?
It looks like you did not.

Did you run the initial "cvmfs_config setup" directly after installation?

See:
https://cernvm.cern.ch/portal/filesystem/quickstart
https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html

Did you install the software package "cvmfs-config-default" that includes the missing keys?

Actually I checked to be sure and I did install it before all this kind of mess.

Yes I runned the initial "cvmfs_config setup", I remember it well since I followed the instruction you've given in your links.
After "cvmfs_config setup" I've done "$ cat /etc/auto.master.d/cvmfs.autofs" to check if the file contained /cvmfs and /etc/auto.cvmfs

I've gone through default.config and default.local once again... and thank you for brining up the links actually it was of great help.

In default.config, there is :

CVMFS_KEYS_DIR=/etc/cvmfs/keys/

BUT what the Client Configuration Web page shows is :

CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch

and indeed it seems that a file, cern.pub, is needed.

Maybe I dit a useless thing but I added this line to default.local as well.

A little of cvmfs_config and it's working like a charm with cvmfs_config stat AND cvmfs_config probe

I'm gonna try an ATLAS job and post it the result here.

It's weird because I never touched default.conf before... only created default.local.

Anyway, thanks you for your help and your patience. I'll let you know if it works well now.

micropro


EDIT : seems it runs the same way for ATLAS, and Theory simulation (I got one job) brings a "comping error".
I'll put these problems on hold for now... really awful headache for 4 days.

*.conf must not be changed.
Strictly follow the comments in /etc/cvmfs/default.conf: "Parameter files are sourced in the following order ..."

/etc/cvmfs/default.local is not the correct file to configure CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch.
Instead if your CVMFS is correctly installed this parameter is set as default in /etc/cvmfs/domain.d/cern.ch.conf and looks like this:

CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch

"cvmfs_config showconfig -s atlas.cern.ch" should output:

CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf

You set CVMFS_SERVER_URL in /etc/cvmfs/domain.d/cern.ch.conf
That's a misconfiguration and might cause trouble if you upgrade CVMFS in the future.
Instead put the openhtc.io servers in /etc/cvmfs/domain.d/cern.ch.local.

By undoing my mistakes... it failed again (back to normal I guess).

Reading carefully the default.conf, it must not be edited ok... (well, we can do it so, "should not" is more suitable, just my opinon).

$ cvmfs_config showconfig -s altas.cern.ch
...
CVMFS_KEYS_DIR=/etc/cvmfs/keys/    # from /etc/cvmfs/default.conf
...

Actually, that makes sense when I look at my other files because none of the other contain any kind of information about CVMFS_KEYS_DIR

As far as cern.ch.conf is concerned I have this, is it correct ?

$ cat cern.ch.conf 
# Don't edit here.  Create /etc/cvmfs/domain.d/cern.ch.local.
# As a rule of thumb, overwrite only parameters you find in here.
# If you look for any other parameter, check /etc/cvmfs/default.(conf|local)
# and /etc/cvmfs/config.d/<your_repository>.(conf|local)
#
# Parameter files are sourced in the following order
# /etc/cvmfs/default.conf
# /etc/cvmfs/default.d/*
# /etc/cvmfs/default.local
# /etc/cvmfs/domain.d/<your_domain>.conf
# /etc/cvmfs/domain.d/<your_domain>.local
# /etc/cvmfs/config.d/<your_repository>.conf
# /etc/cvmfs/config.d/<your_repository>.local
#
# Use cvmfs_config showconfig to get the effective parameters.
#

CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1unl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io/cvmfs/@fqrn@"
# CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch
# CVMFS_USE_GEOAPI=yes

Just so you know, if I uncomment the CVMFS_KEYS_DIR of this file, that I wipecache, reload and probe (successfuly), then I have as I should :

$ cvmfs_config showconfig -s altas.cern.ch
...
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf
...

I must admit I'm a bit lost among all this mistakes I've made (and unmade sometimes) but still I don't understand what this line was commented.
I see a bit clearer in the way cvmfs_config is organised though. Even if I don't get cvmfs to work with this installation, I'm grateful for all this help !

micropro

http://s1unl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io/cvmfs/@fqrn@
?
You are on a good way, but it is very difficult with the parameters.
It is important to do exactly the way of the installations.
In the Atlas-Forum are more infos.

Well if I remember it well, I edited this file following the advice given in the third or fourth post of this topic.

Anyway, I've looked at Atlas's topics and followed the instructions given there.

So, I've now a cern.ch.local with the URLs correct (I guess) :

$ sudo cvmfs_config showconfig -s

Running /usr/bin/cvmfs_config -s atlas:
[...]
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.local
[...]

After a reload, the stat gives something unexpected (following Atlas topic recommandations) :

$ sudo cvmfs_config stat

Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.7.1.0 3571 3 28616 13 0 1 57631 6144001 0 65024 0 0 n/a 0 0 http://s1cern-cvmfs.openhtc.io/cvmfs/cvmfs-config.cern.ch DIRECT 1

stat atlas.cern.ch gives me :

$ sudo cvmfs_config stat atlas.cern.ch
atlas.cern.ch not mounted

I don't remember doing anything else than what was advised in the installation instructions but when you look here and there you notice there are some slightly differences.

Anyhow, after editing, wipecache, reload, probing gives me all okays.

Don't know what I am supposed to do next...

Thanks again for your help and patience (and reading my English... not my native language)

micropro

http://s1unl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io/cvmfs/@fqrn@
?
You are on a good way, but it is very difficult with the parameters.
It is important to do exactly the way of the installations.
In the Atlas-Forum are more infos.

Was my fault, had up to now not this background for changes from the openhtc.io, sorry.

This is the oldest message about openhtc.io found in the Atlas thread:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4758&postid=35857#35857

Well, it seems we have to keep our config files up to date.

When I see some configuration posted I tell myself some default.conf must have been altered... in the right way I mean, even if it is not recommended (but actually, if you have a backup file, it works well).

I still had some messed up CPU Time but whatever... could it be my hardware by the way ?

On Windows CMS keeps sending me "error computing" or similar messages... but this is for another thread I guess ;)

As far as I'm concerned, my quesions were answered and problems solved. Thank you very much !!

micropro