Message boards :
ATLAS application :
ATLAS tasks are in calculation error
Message board moderation
Author | Message |
---|---|
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
Hi, On Linux Ubuntu 20.04 since a few days all the ATLAS tasks are in calculation error, apparently VirtualBox does not launch. I restarted the whole project to see but it's still the same ... What's wrong ? Sam. Powered by Debian & Ubuntu 20.04 Boinc version 7.16.6 x86_64-pc-linux-gnu |
Send message Joined: 15 Jun 08 Posts: 2410 Credit: 225,982,745 RAC: 125,943 |
On Linux you can run ATLAS and Theory either as VBox app or as native app. To run native apps a CVMFS client must be installed on the computer and it must be correctly configured. There are a lot of comments about that in the forum. Good starting points would be: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4971&postid=38259 https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4840&postid=36880 https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594 |
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
On Linux you can run ATLAS and Theory either as VBox app or as native app. I did all of this but still get: $ sudo cvmfs_config probe Probing /cvmfs/atlas.cern.ch... Failed! Probing /cvmfs/atlas-condb.cern.ch... Failed! Probing /cvmfs/grid.cern.ch... Failed! Probing /cvmfs/cernvm-prod.cern.ch... Failed! Probing /cvmfs/sft.cern.ch... Failed! Probing /cvmfs/alice.cern.ch... Failed! The ATLAS tasks worked very well before and since the maintenance of the server it no longer works ... Sam. Powered by Debian & Ubuntu 20.04 Boinc version 7.16.6 x86_64-pc-linux-gnu |
Send message Joined: 15 Jun 08 Posts: 2410 Credit: 225,982,745 RAC: 125,943 |
The ATLAS tasks worked very well before and since the maintenance of the server it no longer works ... The ATLAS tasks that worked fine were VBox tasks. Since you switched to ATLAS native they fail due to the missing local CVMFS client. It has nothing to do with server maintenance. Here a snippet from your logfile: [2021-02-04 22:37:28] Checking for CVMFS [2021-02-04 22:37:28] No cvmfs_config command found, will try listing directly You'll have to decide whether to run ATLAS VBox which has a well configured CVMFS client inside the VM or whether you install/configure your own CVMFS client which is a must for native apps. If you try the latter and "sudo cvmfs_config probe" prints at least a single error try the following steps: 1. ensure "atlas" or "atlas.cern.ch" is listed beside other repositories in CVMFS_REPOSITORIES=... of your /etc/cvmfs/default.local. 2. run "sudo cvmfs_config showconfig -s atlas.cern.ch" and post it's output here. If it's not atlas that fails do the same steps with the repository that fails. It makes no sense to start a single ATLAS/Theory native task before you solved that. |
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
Hi, I have the cvmfs 2.8.0~1+ubuntu20.04 installed but I have problem to configure it: samp@jupiter:~/Téléchargements/Boinc$ sudo wget https://lhcathome.cern.ch/lhcathome/download/default.local -O /etc/cvmfs/default.local --2021-02-07 15:37:03-- https://lhcathome.cern.ch/lhcathome/download/default.local Résolution de lhcathome.cern.ch (lhcathome.cern.ch)… 137.138.156.174, 137.138.148.247 Connexion à lhcathome.cern.ch (lhcathome.cern.ch)|137.138.156.174|:443… connecté. requête HTTP transmise, en attente de la réponse… 200 OK Taille : 1075 (1,0K) Enregistre : «/etc/cvmfs/default.local» /etc/cvmfs/default.local 100%[================================================================>] 1,05K --.-KB/s ds 0s 2021-02-07 15:37:04 (77,4 MB/s) - «/etc/cvmfs/default.local» enregistré [1075/1075] samp@jupiter:~/Téléchargements/Boinc$ sudo cvmfs_config reload samp@jupiter:~/Téléchargements/Boinc$ cvmfs_config probe Probing /cvmfs/atlas.cern.ch... Failed! Probing /cvmfs/atlas-condb.cern.ch... Failed! Probing /cvmfs/grid.cern.ch... Failed! Probing /cvmfs/cernvm-prod.cern.ch... Failed! Probing /cvmfs/sft.cern.ch... Failed! Probing /cvmfs/alice.cern.ch... Failed! samp@jupiter:~/Téléchargements/Boinc$ !? Powered by Debian & Ubuntu 20.04 Boinc version 7.16.6 x86_64-pc-linux-gnu |
Send message Joined: 15 Jun 08 Posts: 2410 Credit: 225,982,745 RAC: 125,943 |
When you installed CVMFS, did you follow all required steps? 1. Install both packets: "cvmfs" and "cvmfs-config-default" 2. Replace /etc/cvmfs/default.local (according to your post I guess you did that) 3. Run "cvmfs_config setup" 4. Follow the hints given here: https://cvmfs.readthedocs.io/en/stable/cpt-quickstart.html 5. Reboot (sometimes required twice) If "cvmfs_config probe" still fails, try "cvmfs_config chksetup". If this shows errors try to fix them. If you can't fix the problem, post the output of "cvmfs_config chksetup" and "cvmfs_config showconfig -s atlas.cern.ch". |
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
Hi, Now I got this: samp@jupiter:~$ sudo cvmfs_config probe [sudo] Mot de passe de samp : Probing /cvmfs/atlas.cern.ch... OK Probing /cvmfs/atlas-condb.cern.ch... OK Probing /cvmfs/grid.cern.ch... OK samp@jupiter:~$ sudo cvmfs_config chksetup Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas.cern.ch/.cvmfspublished through proxy DIRECT Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas-condb.cern.ch/.cvmfspublished through proxy DIRECT Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io Warning: failed to use Geo-API with s1ral-cvmfs.openhtc.io Warning: failed to access http://s1ihep-cvmfs.openhtc.io/cvmfs/grid.cern.ch/.cvmfspublished through proxy DIRECT Warning: failed to use Geo-API with s1ihep-cvmfs.openhtc.io samp@jupiter:~$ samp@jupiter:~$ sudo cvmfs_config showconfig -s atlas.cern.ch CVMFS_REPOSITORY_NAME=atlas.cern.ch CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf CVMFS_CACHE_DIR=/var/lib/cvmfs/shared CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf CVMFS_CLIENT_PROFILE= # from /etc/cvmfs/default.conf CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf CVMFS_FALLBACK_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf CVMFS_HTTP_PROXY=DIRECT # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_KCACHE_TIMEOUT=2 # from /etc/cvmfs/default.local CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf CVMFS_MAX_RETRIES=3 # from /etc/cvmfs/default.local CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch # from /etc/cvmfs/default.local CVMFS_SEND_INFO_HEADER=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1ihep-cvmfs.openhtc.io/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local CVMFS_USE_GEOAPI=yes # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf samp@jupiter:~$ Sam. |
Send message Joined: 15 Jun 08 Posts: 2410 Credit: 225,982,745 RAC: 125,943 |
I wonder if you ever check your logfiles, e.g.: https://lhcathome.cern.ch/lhcathome/result.php?resultid=299453228 <core_client_version>7.4.23</core_client_version> <![CDATA[ <message> process exited with code 195 (0xc3, -61) </message> <stderr_txt> 04:20:55 (46158): wrapper (7.7.26015): starting 04:20:55 (46158): wrapper: running run_atlas (--nthreads 4) awk: line 2: function strftime never defined 04:20:56 (46158): run_atlas exited; CPU time 0.000000 04:20:56 (46158): app exit status: 0xd 04:20:56 (46158): called boinc_finish(195) </stderr_txt> ]]> For a couple of days that computer reports all ATLAS tasks as failures - ATM 128 and the number is growing. All of them fail due to a missing gawk. This has already been discussed in April 2020 and can easily be found using the forum's extended search. Makes no sense to request ATLAS native on that computer until the issue is fixed. Regarding CVMFS on this computer: https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10679626 There was a success until this task: https://lhcathome.cern.ch/lhcathome/result.php?resultid=299406856 [2021-02-08 00:08:23] Probing /cvmfs/atlas.cern.ch... OK [2021-02-08 00:08:24] Probing /cvmfs/atlas-condb.cern.ch... OK [2021-02-08 00:08:25] Probing /cvmfs/grid.cern.ch... OK [2021-02-08 00:08:25] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2021-02-08 00:08:25] 2.8.0.0 634080 0 24684 78657 3 1 449119 4096000 0 130560 0 0 n/a 525 202 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1 [2021-02-08 00:08:25] CVMFS is ok All tasks you started this morning failed again: https://lhcathome.cern.ch/lhcathome/result.php?resultid=299459502 [2021-02-08 06:07:00] Checking for CVMFS [2021-02-08 06:07:00] Probing /cvmfs/atlas.cern.ch... Failed! [2021-02-08 06:07:00] Probing /cvmfs/atlas-condb.cern.ch... Failed! [2021-02-08 06:07:00] Probing /cvmfs/grid.cern.ch... Failed! [2021-02-08 06:07:00] Probing /cvmfs/cernvm-prod.cern.ch... Failed! [2021-02-08 06:07:01] Probing /cvmfs/sft.cern.ch... Failed! [2021-02-08 06:07:01] Probing /cvmfs/alice.cern.ch... Failed! [2021-02-08 06:07:01] cvmfs_config probe failed, aborting the job Looks like you changed the CVMFS configuration before. Independend from CVMFS your 1st task reports another issue: [2021-02-08 00:08:42] [31mFATAL: [0m container creation failed: mount ->/var error: can't remount /var: operation not permitted IIRC this has also been discussed a couple of times. Might be an issue related to Ubuntu/Debian. Since I'm not running them you would need help from the Ubuntu/Debian experts. |
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
Hi, When I registered on this project the ATLAS tasks were not causing problems with VirtualBox without having configured anything special, what is the interest of CVMFS? For the moment I have changed my settings to no longer receive ATLAS tasks, thank you for your help anyway. Sam. |
Send message Joined: 15 Jun 08 Posts: 2410 Credit: 225,982,745 RAC: 125,943 |
Unlike SixTrack ATLAS, Theory and CMS are very complex scientific apps that require access to large software repositories and DB snippets. That data is distributed via CVMFS and Frontier. To guarantee a well configured environment all of that is packed into a VirtualBox VM. ATLAS/Theory native had bee developed later, mainly to save some RAM and - regarding CVMFS - to keep most of the files in a local cache. So far native apps work only on Linux and a volunteer has to do more work to get them running. If it's too much work or not compatible with your host system you may configure your computers to request only VBox apps: https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project |
Send message Joined: 26 Jan 21 Posts: 25 Credit: 13,070,626 RAC: 0 |
Unlike SixTrack ATLAS, Theory and CMS are very complex scientific apps that require access to large software repositories and DB snippets. Thank you for all these explanations, I admire and respect all this work and it is a pleasure for me to participate in all these scientific projects and I thank you for all the work you do. In fact I solved the problem I disabled the "Native" mode and now the ATLAS tasks are executed in Virtualbox I will revalidate the sending of the ATLAS tasks. Sam. |
©2024 CERN