Message boards : ATLAS application : ATLAS tasks are in calculation error
Message board moderation

To post messages, you must log in.

AuthorMessage
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44263 - Posted: 4 Feb 2021, 17:50:20 UTC

Hi,

On Linux Ubuntu 20.04 since a few days all the ATLAS tasks are in calculation error, apparently VirtualBox does not launch. I restarted the whole project to see but it's still the same ...

What's wrong ?

Sam.
Powered by Debian & Ubuntu 20.04
Boinc version 7.16.6 x86_64-pc-linux-gnu
ID: 44263 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1818
Credit: 122,901,402
RAC: 76,405
Message 44265 - Posted: 4 Feb 2021, 18:21:20 UTC - in response to Message 44263.  

On Linux you can run ATLAS and Theory either as VBox app or as native app.
To run native apps a CVMFS client must be installed on the computer and it must be correctly configured.

There are a lot of comments about that in the forum.
Good starting points would be:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4971&postid=38259
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4840&postid=36880
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594
ID: 44265 · Report as offensive     Reply Quote
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44269 - Posted: 4 Feb 2021, 23:36:56 UTC - in response to Message 44265.  

On Linux you can run ATLAS and Theory either as VBox app or as native app.
To run native apps a CVMFS client must be installed on the computer and it must be correctly configured.

There are a lot of comments about that in the forum.
Good starting points would be:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4971&postid=38259
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4840&postid=36880
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594


I did all of this but still get:

$ sudo cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/grid.cern.ch... Failed!
Probing /cvmfs/cernvm-prod.cern.ch... Failed!
Probing /cvmfs/sft.cern.ch... Failed!
Probing /cvmfs/alice.cern.ch... Failed!


The ATLAS tasks worked very well before and since the maintenance of the server it no longer works ...

Sam.
Powered by Debian & Ubuntu 20.04
Boinc version 7.16.6 x86_64-pc-linux-gnu
ID: 44269 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1818
Credit: 122,901,402
RAC: 76,405
Message 44270 - Posted: 5 Feb 2021, 9:25:06 UTC - in response to Message 44269.  

The ATLAS tasks worked very well before and since the maintenance of the server it no longer works ...

The ATLAS tasks that worked fine were VBox tasks.
Since you switched to ATLAS native they fail due to the missing local CVMFS client.
It has nothing to do with server maintenance.
Here a snippet from your logfile:
[2021-02-04 22:37:28] Checking for CVMFS
[2021-02-04 22:37:28] No cvmfs_config command found, will try listing directly



You'll have to decide whether to run ATLAS VBox which has a well configured CVMFS client inside the VM or whether you install/configure your own CVMFS client which is a must for native apps.

If you try the latter and "sudo cvmfs_config probe" prints at least a single error try the following steps:
1. ensure "atlas" or "atlas.cern.ch" is listed beside other repositories in CVMFS_REPOSITORIES=... of your /etc/cvmfs/default.local.
2. run "sudo cvmfs_config showconfig -s atlas.cern.ch" and post it's output here.

If it's not atlas that fails do the same steps with the repository that fails.
It makes no sense to start a single ATLAS/Theory native task before you solved that.
ID: 44270 · Report as offensive     Reply Quote
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44279 - Posted: 7 Feb 2021, 15:26:12 UTC - in response to Message 44270.  

Hi,

I have the cvmfs 2.8.0~1+ubuntu20.04 installed but I have problem to configure it:

samp@jupiter:~/Téléchargements/Boinc$ sudo wget https://lhcathome.cern.ch/lhcathome/download/default.local -O /etc/cvmfs/default.local
--2021-02-07 15:37:03--  https://lhcathome.cern.ch/lhcathome/download/default.local
Résolution de lhcathome.cern.ch (lhcathome.cern.ch)… 137.138.156.174, 137.138.148.247
Connexion à lhcathome.cern.ch (lhcathome.cern.ch)|137.138.156.174|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 1075 (1,0K)
Enregistre : «/etc/cvmfs/default.local»

/etc/cvmfs/default.local            100%[================================================================>]   1,05K  --.-KB/s    ds 0s      

2021-02-07 15:37:04 (77,4 MB/s) - «/etc/cvmfs/default.local» enregistré [1075/1075]

samp@jupiter:~/Téléchargements/Boinc$ sudo cvmfs_config reload
samp@jupiter:~/Téléchargements/Boinc$ cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... Failed!
Probing /cvmfs/atlas-condb.cern.ch... Failed!
Probing /cvmfs/grid.cern.ch... Failed!
Probing /cvmfs/cernvm-prod.cern.ch... Failed!
Probing /cvmfs/sft.cern.ch... Failed!
Probing /cvmfs/alice.cern.ch... Failed!
samp@jupiter:~/Téléchargements/Boinc$ 

!?
Powered by Debian & Ubuntu 20.04
Boinc version 7.16.6 x86_64-pc-linux-gnu
ID: 44279 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1818
Credit: 122,901,402
RAC: 76,405
Message 44280 - Posted: 7 Feb 2021, 16:27:37 UTC - in response to Message 44279.  

When you installed CVMFS, did you follow all required steps?

1. Install both packets: "cvmfs" and "cvmfs-config-default"
2. Replace /etc/cvmfs/default.local (according to your post I guess you did that)
3. Run "cvmfs_config setup"
4. Follow the hints given here: https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html
5. Reboot (sometimes required twice)


If "cvmfs_config probe" still fails, try "cvmfs_config chksetup".
If this shows errors try to fix them.


If you can't fix the problem, post the output of "cvmfs_config chksetup" and "cvmfs_config showconfig -s atlas.cern.ch".
ID: 44280 · Report as offensive     Reply Quote
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44282 - Posted: 7 Feb 2021, 20:14:42 UTC - in response to Message 44280.  

Hi,

Now I got this:

samp@jupiter:~$ sudo cvmfs_config probe
[sudo] Mot de passe de samp : 
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
samp@jupiter:~$ sudo cvmfs_config chksetup
Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io
Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io
Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io
Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas-condb.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io
Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io
Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/grid.cern.ch/.cvmfspublished through proxy DIRECT
Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io
samp@jupiter:~$ 


samp@jupiter:~$ sudo cvmfs_config showconfig -s atlas.cern.ch
CVMFS_REPOSITORY_NAME=atlas.cern.ch
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPO_DEFAULT_ENV=1    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DEFAULT_DOMAIN=cern.ch    # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_FALLBACK_PROXY=    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=DIRECT    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_KCACHE_TIMEOUT=2    # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=3    # from /etc/cvmfs/default.local
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat'    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=4000    # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch    # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=yes    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=yes    # from /etc/cvmfs/default.local
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
samp@jupiter:~$ 


Sam.
ID: 44282 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1818
Credit: 122,901,402
RAC: 76,405
Message 44283 - Posted: 8 Feb 2021, 6:43:43 UTC - in response to Message 44282.  

I wonder if you ever check your logfiles, e.g.:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=299453228
<core_client_version>7.4.23</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)
</message>
<stderr_txt>
04:20:55 (46158): wrapper (7.7.26015): starting
04:20:55 (46158): wrapper: running run_atlas (--nthreads 4)
awk: line 2: function strftime never defined
04:20:56 (46158): run_atlas exited; CPU time 0.000000
04:20:56 (46158): app exit status: 0xd
04:20:56 (46158): called boinc_finish(195)

</stderr_txt>
]]>

For a couple of days that computer reports all ATLAS tasks as failures - ATM 128 and the number is growing.
All of them fail due to a missing gawk.
This has already been discussed in April 2020 and can easily be found using the forum's extended search.
Makes no sense to request ATLAS native on that computer until the issue is fixed.



Regarding CVMFS on this computer:
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10679626
There was a success until this task:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=299406856
[2021-02-08 00:08:23] Probing /cvmfs/atlas.cern.ch... OK
[2021-02-08 00:08:24] Probing /cvmfs/atlas-condb.cern.ch... OK
[2021-02-08 00:08:25] Probing /cvmfs/grid.cern.ch... OK
[2021-02-08 00:08:25] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2021-02-08 00:08:25] 2.8.0.0 634080 0 24684 78657 3 1 449119 4096000 0 130560 0 0 n/a 525 202 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1
[2021-02-08 00:08:25] CVMFS is ok




All tasks you started this morning failed again:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=299459502
[2021-02-08 06:07:00] Checking for CVMFS
[2021-02-08 06:07:00] Probing /cvmfs/atlas.cern.ch... Failed!
[2021-02-08 06:07:00] Probing /cvmfs/atlas-condb.cern.ch... Failed!
[2021-02-08 06:07:00] Probing /cvmfs/grid.cern.ch... Failed!
[2021-02-08 06:07:00] Probing /cvmfs/cernvm-prod.cern.ch... Failed!
[2021-02-08 06:07:01] Probing /cvmfs/sft.cern.ch... Failed!
[2021-02-08 06:07:01] Probing /cvmfs/alice.cern.ch... Failed!
[2021-02-08 06:07:01] cvmfs_config probe failed, aborting the job

Looks like you changed the CVMFS configuration before.




Independend from CVMFS your 1st task reports another issue:
[2021-02-08 00:08:42] FATAL:   container creation failed: mount ->/var error: can't remount /var: operation not permitted

IIRC this has also been discussed a couple of times.
Might be an issue related to Ubuntu/Debian.
Since I'm not running them you would need help from the Ubuntu/Debian experts.
ID: 44283 · Report as offensive     Reply Quote
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44284 - Posted: 8 Feb 2021, 16:57:46 UTC - in response to Message 44283.  

Hi,

When I registered on this project the ATLAS tasks were not causing problems with VirtualBox without having configured anything special, what is the interest of CVMFS?

For the moment I have changed my settings to no longer receive ATLAS tasks, thank you for your help anyway.

Sam.
ID: 44284 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1818
Credit: 122,901,402
RAC: 76,405
Message 44285 - Posted: 8 Feb 2021, 17:22:25 UTC - in response to Message 44284.  

Unlike SixTrack ATLAS, Theory and CMS are very complex scientific apps that require access to large software repositories and DB snippets.
That data is distributed via CVMFS and Frontier.
To guarantee a well configured environment all of that is packed into a VirtualBox VM.

ATLAS/Theory native had bee developed later, mainly to save some RAM and - regarding CVMFS - to keep most of the files in a local cache.
So far native apps work only on Linux and a volunteer has to do more work to get them running.

If it's too much work or not compatible with your host system you may configure your computers to request only VBox apps:
https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project
ID: 44285 · Report as offensive     Reply Quote
Sam Przyswa

Send message
Joined: 26 Jan 21
Posts: 18
Credit: 1,504,701
RAC: 5,155
Message 44286 - Posted: 8 Feb 2021, 20:40:12 UTC - in response to Message 44285.  

Unlike SixTrack ATLAS, Theory and CMS are very complex scientific apps that require access to large software repositories and DB snippets.
That data is distributed via CVMFS and Frontier.
To guarantee a well configured environment all of that is packed into a VirtualBox VM.

ATLAS/Theory native had bee developed later, mainly to save some RAM and - regarding CVMFS - to keep most of the files in a local cache.
So far native apps work only on Linux and a volunteer has to do more work to get them running.

If it's too much work or not compatible with your host system you may configure your computers to request only VBox apps:
https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project


Thank you for all these explanations, I admire and respect all this work and it is a pleasure for me to participate in all these scientific projects and I thank you for all the work you do.

In fact I solved the problem I disabled the "Native" mode and now the ATLAS tasks are executed in Virtualbox I will revalidate the sending of the ATLAS tasks.

Sam.
ID: 44286 · Report as offensive     Reply Quote

Message boards : ATLAS application : ATLAS tasks are in calculation error


©2021 CERN