LHC@home

Dave Dykstra from Fermilab lately presented Cloudflare's openhtc.io at the CHEP 2018 Conference.
His presentation can be found here:
https://indico.cern.ch/event/587955/contributions/2936824/

Volunteers running ATLAS native can already use openhtc.io when they change their local CVMFS configuration as shown below.


1. cd to the local folder /etc/cvmfs/domain.d. It usually contains a couple of "*.conf" files
2. Create a corresponding "*.local" file for each domain you want to switch to openhtc.io
3. Insert the following line into each "*.local" file:

CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@'

4. Run "cvmfs_config showconfig -s" to check if the new configuration will be accepted
5. Run "cvmfs_config reload" to activate the new configuration
6. "cvmfs_config stat" should look like this example:

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.5.0.0 10512 21230 66668 37686 1 58 2705096 3538945 2100 65024 0 11203 99.8126 24924 770 http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1

Volunteers using their own local squid would see "http://<IP_of_your_local_proxy>:3128" instead if "DIRECT".


<edit>
Have been informed that my wording "Cloudflare's openhtc.io" may be not be correct.
Anybody who is interested in the correct wording may read Dave Dykstra's presentation.
</edit>

Yes our own Laurence Field is one of the co-authors

Years ago I saw that field of grass and buffalo / bison above FermiLab
Mars sure is bright tonight (depending on where you are)

nice, thanks for the infos and tipps, computezrmle!

looks like it is working here.

Thanks a lot, it also works well for me (I'm also at the CHEP 2018 conference this week).

Volunteers running ATLAS native can already use openhtc.io when they change their local CVMFS configuration as shown below.


1. cd to the local folder /etc/cvmfs/domain.d. It usually contains a couple of "*.conf" files
2. Create a corresponding "*.local" file for each domain you want to switch to openhtc.io

In that folder I see 3 *.conf:

cern.ch.conf
egi.eu.conf
opensciencegrid.org.conf

I don't know the last two though they sound interesting. I know only cern.ch and I assume that to run ATLAS native tasks it would suffice to create a *.local file for only cern.ch.conf. Correct?

In that folder I see 3 *.conf:

cern.ch.conf
egi.eu.conf
opensciencegrid.org.conf

I don't know the last two though they sound interesting. I know only cern.ch and I assume that to run ATLAS native tasks it would suffice to create a *.local file for only cern.ch.conf. Correct?

Yes, that worked for me. I just created a "cern.ch.local" in that folder as directed, and it all worked.

@Jim1348

Marvelous. It seems to be working here too or at least it passed all the tests described by computezrmie in the op. I will assume it's caching since i don't know how to test and verify. Maybe what I don't know won't hurt me :)

@ all you other Linux wizards (gyllic, computezrmie, etc)...

Thanks for all your great advice on getting native ATLAS tasks working. It's nice to be able to wiggle the mouse and login without having the swap daemon go nuts and end up with a fubar task.

I will assume it's caching since i don't know how to test and verify. Maybe what I don't know won't hurt me :)

One practical test is that in your Task details (stderr_txt file), you will see a "CPUUsage=" number. It should go up after the cache is activated. It depends on the number of cores you run per work unit, but I run two cores, and now get CPUUsage=195%. Before using the cache, I was getting around CPUUsage=189% or less. That is the payoff, increased CPU efficiency.

Thanks. I completed only 2 tasks without the caching setup so I don't have much of a baseline but it looks like caching is giving about a 6% increase in CPU usage. I consider it verified... cache is working :)

It works also for me. At the moment in one SL69.
Running /usr/bin/cvmfs_config stat cvmfs-config.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.5.0.0 8715 7 29220 8 3 1 3995592 4194304 0 65024 0 0 n/a 0 0 http://s1cern-cvmfs.openhtc.io/cvmfs/cvmfs-config.cern.ch DIRECT 1

Thanks for working out.

bronco wrote:

... I will assume it's caching since i don't know how to test and verify.

Yes, it's caching.
To test it, simply run "cvmfs_config stat" and look at the HITRATE.

Your individual cache size can be configured via CVMFS_QUOTA_LIMIT in "/etc/cvmfs/default.local".

Typical hitrates in relation to the cache size:

cache size: 2700 MB
hitrate: 95 %

cache size: 3500 MB
hitrate: 99 %

To test it, simply run "cvmfs_config stat" and look at the HITRATE.

I haven't changed the default settings, and get this:

Running /usr/bin/cvmfs_config stat atlas.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.5.0.0 2749 2416 43680 37830 2 58 3273787 4096000 559 65024 0 296062 99.8868 378017 928 http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1

Running /usr/bin/cvmfs_config stat sft.cern.ch:
VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2.5.0.0 16268 2089 23528 10053 3 3 3273787 4096000 7 65024 0 5623 99.7866 406 1 http://s1fnal-cvmfs.openhtc.io/cvmfs/sft.cern.ch DIRECT 1

So it appears that my HITRATE is over 99% if I am reading this correctly.

... So it appears that my HITRATE is over 99% if I am reading this correctly.

Yes.
Your setup uses the default value for CVMFS_QUOTA_LIMIT (4GB) which is set in "/etc/cvmfs/default.conf".
The value is shown as CACHEMAX(K) while CACHEUSE(K) shows how much is currently in use.

Be aware that (by default) all configured repositories share the same local cache and it's total size.

So it appears that my HITRATE is over 99% if I am reading this correctly.

Yes, you are reading this correctly. This hitrate shows that caching some data for ATLAS is a good idea and that using a local cache will improve efficiency. This is also one of the big benefits of the native app compared to the vbox app because the local cache of the vbox app gets deleted as soon as the VM has shut down and been removed.

EDIT: looks like computezrmle was faster so ignore this post :-)

EDIT: looks like computezrmle was faster so ignore this post :-)

I think it is a great post. I just wish that LHCb could do as well.

Be aware that (by default) all configured repositories share the same local cache and it's total size.

OK. I have been running just one native ATLAS at a time (2 cores per work unit), which probably helps the efficiency. I will increase that to three WUs at a time and see what happens. I can increase the size of the cache if necessary. I have 32 GB just sitting there mostly unused, and this would be the best use I could put it to.

Thanks again, as always.

So it appears that my HITRATE is over 99% if I am reading this correctly.

I also stuck with the 4GB default and also get over 99% HITRATE.

Thanks for all the great infos, guys.

I am attempting to run native ATLAS again with openhtc, but when I run a check I get:

$ sudo cvmfs_config chksetup
Warning: failed to use Geo-API with s1cern-cvmfs.openhtc.io
Warning: failed to access http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1bnl-cvmfs.openhtc.io
Warning: failed to use Geo-API with s1cern-cvmfs.openhtc.io
Warning: failed to access http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas-condb.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1bnl-cvmfs.openhtc.io
Warning: failed to use Geo-API with s1cern-cvmfs.openhtc.io
Warning: failed to access http://s1bnl-cvmfs.openhtc.io/cvmfs/grid.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1bnl-cvmfs.openhtc.io

Does this mean that native ATLAS won't work at all? Do I need to change something?

Please run "cvmfs_config showconfig -s atlas.cern.ch" and post the output.

$ cvmfs_config showconfig -s atlas.cern.ch
CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=DIRECT # from /etc/cvmfs/default.local
CVMFS_IGNORE_SIGNATURE=no # from /etc/cvmfs/default.conf
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_MOUNT_RW=no # from /etc/cvmfs/default.conf
CVMFS_NFILES=65536 # from /etc/cvmfs/default.conf
CVMFS_NFS_SOURCE=no # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS=http://wpad/wpad.dat # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=8192 # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=no # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.local
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf