log in

New ATLAS app version released for Linux hosts


Advanced search

Message boards : News : New ATLAS app version released for Linux hosts

1 · 2 · Next
Author Message
Wenjing Wu
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist
Send message
Joined: 23 Jun 14
Posts: 10
Credit: 62,271,789
RAC: 927,646
Message 31904 - Posted: 10 Aug 2017, 11:04:25 UTC
Last modified: 10 Aug 2017, 12:15:29 UTC

We released a new version of the ATLAS app today, 2.41 for the x86_64-pc-linux-gnu platform.
The new features of this version include:
1. It requires the host OS to be either Scentific Linux 6 or Cent OS 7.
2. It require CVMFS and Singularity instead of Virtualbox to run the ATLAS jobs.
3. It is more efficient, as the avoidance of using Virtualbox.
Currently, this version is set to beta version.

For people who want to try it out,we provide a script to install everything including CVMFS, singularity here,


Try it if you are interested!

computezrmle
Send message
Joined: 15 Jun 08
Posts: 347
Credit: 3,501,142
RAC: 1,838
Message 31905 - Posted: 10 Aug 2017, 11:35:17 UTC - in response to Message 31904.

We released a new version of the ATLAS app today, 2.41 for the x86_64-pc-linux-gnu platform.
The new features of this version include:
1. It requires the host OS to be either Scentific Linux 6 or Cent OS 7.
2. It require CVMFS and Singularity instead of Virtualbox to run the ATLAS jobs.
3. It is more efficient, as the avoidance of using Virtualbox.
Currently, this version is set to beta version.

For people who want to try it out,we provide a script to install everything including CVMFS, singularity here,
http://atlasathome.cern.ch/boinc_conf/install_cvmfs_sin.sh

Try it if you are interested!

Sounds nice, at least at first glance, but I have some questions:

1. Why is it hosted on the old ATLAS server and not on the dev server?
2. Is it based on the version David Cameron has tested a few months ago?
3. Why is it restricted to the mentioned distributions? Simply not tested on others or due to some specific requirements?

Wenjing Wu
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist
Send message
Joined: 23 Jun 14
Posts: 10
Credit: 62,271,789
RAC: 927,646
Message 31906 - Posted: 10 Aug 2017, 12:10:55 UTC - in response to Message 31905.



1. Why is it hosted on the old ATLAS server and not on the dev server?
2. Is it based on the version David Cameron has tested a few months ago?
3. Why is it restricted to the mentioned distributions? Simply not tested on others or due to some specific requirements?


1. Only the script file for installing cvmfs/singularity is hosted on the old ATLAS server, the app is still on the lhcathome server.
2. Yes, it is based on the native version David tested a few months ago
3. So far we only tested it on SLC6 and CentOS 7, it might work on other Linux, depending on if one can successfully install the CVMFS and Singularity there.
So please feel free to try, and we would appreciate the feedback on that.

Cheers!

gyllic
Send message
Joined: 9 Dec 14
Posts: 93
Credit: 1,093,009
RAC: 4,770
Message 31913 - Posted: 10 Aug 2017, 20:36:13 UTC - in response to Message 31906.
Last modified: 10 Aug 2017, 20:56:21 UTC

Exciting news!

3. So far we only tested it on SLC6 and CentOS 7, it might work on other Linux, depending on if one can successfully install the CVMFS and Singularity there.
So please feel free to try, and we would appreciate the feedback on that.

Im gonna try it on Debian (in a week or so). I will let you know if i was successfull.

Edit:
1. Why is it hosted on the old ATLAS server and not on the dev server?

Good point. The native app is only shown on atlasathome.cern.ch and not on lhcathome.cern.ch or the dev site.

Wenjing Wu
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist
Send message
Joined: 23 Jun 14
Posts: 10
Credit: 62,271,789
RAC: 927,646
Message 31914 - Posted: 11 Aug 2017, 1:07:04 UTC - in response to Message 31913.
Last modified: 11 Aug 2017, 1:07:30 UTC


Good point. The native app is only shown on atlasathome.cern.ch and not on lhcathome.cern.ch or the dev site.


Actually it is on lhcathome.cern.ch,as I mentioned early, it is set to be beta version, so you need to enable "test app" in the lhcathome preference in your account, otherwise, you do not receive jobs from the native app..

maeax
Send message
Joined: 2 May 07
Posts: 232
Credit: 11,991,406
RAC: 14,253
Message 31916 - Posted: 11 Aug 2017, 6:39:51 UTC

Is it multicore or single?

When multicore, the parameter for the number of Cores is missing.

http://atlasathome.cern.ch/apps.php

computezrmle
Send message
Joined: 15 Jun 08
Posts: 347
Credit: 3,501,142
RAC: 1,838
Message 31918 - Posted: 11 Aug 2017, 7:17:15 UTC - in response to Message 31916.

Is it multicore or single?

When multicore, the parameter for the number of Cores is missing.

http://atlasathome.cern.ch/apps.php

There's an app_config.xml included in the package which sets the ncores to 2 and the RAM usage to 4200 MB.

tullio
Send message
Joined: 19 Feb 08
Posts: 449
Credit: 2,076,809
RAC: 332
Message 31919 - Posted: 11 Aug 2017, 8:42:32 UTC - in response to Message 31904.

No luck on SuSE Linux 42.2 64-bit, running SixTrack.
Tullio

computezrmle
Send message
Joined: 15 Jun 08
Posts: 347
Credit: 3,501,142
RAC: 1,838
Message 31922 - Posted: 11 Aug 2017, 13:03:44 UTC

The current situation is as follows:
1. I installed CVMFS (from CERN) and singularity (from my distro's repository)
2. CVMFS connects to the ATLAS dirs
3. A BOINC client instance is prepared to be attached to the project.


After point 3 there are still open issues.

http://atlasathome.cern.ch/boinc_conf/start_boinc shows the following command:

ProDir/boinccmd --project_attach https://lhcathome.cern.ch/lhcathome/ BOINC_KEY

But there is no ATLAS beta app in the app list.


Or could it be http://atlasathome.cern.ch/ where we have the beta app 2.41 (native_mt) (beta test), but unfortunately no WU to be send.

Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester
Send message
Joined: 15 Jul 05
Posts: 114
Credit: 1,203,349
RAC: 7,020
Message 31923 - Posted: 11 Aug 2017, 14:07:50 UTC - in response to Message 31922.

The app is available as a beta app from this project, LHC@home, and not from the old ATLAS@home. The app version is 241.

I got a few tasks on my desktop, so it should work if /CVMFS is mounted and Singularity installed.

Wenjing and the ATLAS team can provide more details and help.

I suggest to continue the discussion on this app under the ATLAS application sub-forum.

Profile Olivier Fehr
Avatar
Send message
Joined: 1 Jun 17
Posts: 9
Credit: 743,649
RAC: 1,799
Message 31924 - Posted: 11 Aug 2017, 14:21:36 UTC
Last modified: 11 Aug 2017, 15:15:50 UTC

Singularity compiled successfully on Ubuntu 17.04 and CentOS 7.3 boxes. Still have to implement the CVMFS part though.
MODIFY: CVMFS installed sucessfully too on CentOS.

Juha
Send message
Joined: 22 Mar 17
Posts: 24
Credit: 231,948
RAC: 611
Message 31925 - Posted: 11 Aug 2017, 14:22:34 UTC - in response to Message 31923.

What kind of hardware requirements these task have, RAM, multi-core?

Right now all I get is:

No tasks are available for ATLAS Simulation
Message from server: VirtualBox is not installed

gyllic
Send message
Joined: 9 Dec 14
Posts: 93
Credit: 1,093,009
RAC: 4,770
Message 31929 - Posted: 11 Aug 2017, 19:22:52 UTC - in response to Message 31924.

Singularity compiled successfully on Ubuntu 17.04 and CentOS 7.3 boxes. Still have to implement the CVMFS part though.
MODIFY: CVMFS installed sucessfully too on CentOS.

according to the singularity homepage should singularity be in the repositories for debian stretch and for ubuntu since 16.10, so no need to compile it. but great if it worked.

for cvmfs: i successfully compiled it a couple of months ago on debian jessie, so you should not have to much problems compiling it on ubuntu i guess.

Profile Olivier Fehr
Avatar
Send message
Joined: 1 Jun 17
Posts: 9
Credit: 743,649
RAC: 1,799
Message 31930 - Posted: 12 Aug 2017, 7:10:17 UTC - in response to Message 31929.

Indeed, there is a Singularity package (named singluarity-container) for Ubuntu 17.04. It is version 2.2 while the latest Singularity version on the website is at 2.3.1. Since there have been 2 security related updates since, one labeled as moderate the other as high severity issue, I thought it worthwhile to compile the latest version from source.
As for the CVMFS, I'll make an attempt to compile it on Ubuntu 17.04 as soon as I can.

Profile Olivier Fehr
Avatar
Send message
Joined: 1 Jun 17
Posts: 9
Credit: 743,649
RAC: 1,799
Message 31931 - Posted: 12 Aug 2017, 8:19:28 UTC - in response to Message 31930.

That's a success. I have both Singularity and CVMFS software up and running on both CentOS 7.3 and Ubuntu 17.04. Now lets see if I can get any of the new Atlas packages...

Jim1348
Send message
Joined: 15 Nov 14
Posts: 86
Credit: 3,720,810
RAC: 13,984
Message 31932 - Posted: 12 Aug 2017, 15:11:58 UTC

It seems to be set up OK on Ubuntu 16.10. But since I don't use a Proxy, that was a bit of a puzzle until I set CVMFS_HTTP_PROXY=DIRECT in the default.local file. Now it all checks out OK, though I have no work yet.

But I am wondering about a SQUID. I have a good Internet connection (25 Mbps down, 4 Mbps up) and don't think I need one. Would it help the CERN servers? Maybe someone can give some setup guidance if so.

ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar
Send message
Joined: 29 Aug 05
Posts: 305
Credit: 2,415,598
RAC: 5,149
Message 31933 - Posted: 12 Aug 2017, 15:25:35 UTC - in response to Message 31932.

It seems to be set up OK on Ubuntu 16.10. But since I don't use a Proxy, that was a bit of a puzzle until I set CVMFS_HTTP_PROXY=DIRECT in the default.local file. Now it all checks out OK, though I have no work yet.

But I am wondering about a SQUID. I have a good Internet connection (25 Mbps down, 4 Mbps up) and don't think I need one. Would it help the CERN servers? Maybe someone can give some setup guidance if so.

You really only need a squid if you have more than one cvmfs instance, since cvmfs caches too, as well as the squid. I've been through this at work; I did eventually set up a squid because a server became available, and because I'll probably end up installing cvmfs on all our machines rather than just our Grid user-interface -- when I get enough round tuits.
____________

Jim1348
Send message
Joined: 15 Nov 14
Posts: 86
Credit: 3,720,810
RAC: 13,984
Message 31934 - Posted: 12 Aug 2017, 15:57:15 UTC - in response to Message 31933.

Very well. I can use it without worry.

computezrmle
Send message
Joined: 15 Jun 08
Posts: 347
Credit: 3,501,142
RAC: 1,838
Message 31935 - Posted: 12 Aug 2017, 20:18:21 UTC

@Ivan, Jim1348

It's a bit OT here, therefore only some very short remarks.

CERN's vbox apps can get a huge benefit from a local squid.
CMS benefits most, then ATLAS and LHCb, Theory less but sufficient.
Up to 95 % regarding the request hits and 50 % regarding the byte hits.
The setup needs some special tuning compared to the standard setup and should get some assist from iptables.

An "old" computer with a core 2 duo, 1-2 GB RAM (250-350 MB for squid), 30-80 GB free disk space is far enough.
And there is also a benefit for fast DSL lines as the squid avoids latency.

Jim1348
Send message
Joined: 15 Nov 14
Posts: 86
Credit: 3,720,810
RAC: 13,984
Message 31936 - Posted: 12 Aug 2017, 21:11:18 UTC - in response to Message 31935.

CERN's vbox apps can get a huge benefit from a local squid.
CMS benefits most, then ATLAS and LHCb, Theory less but sufficient.
Up to 95 % regarding the request hits and 50 % regarding the byte hits.
The setup needs some special tuning compared to the standard setup and should get some assist from iptables.

I don't know if it helps much, but I use a large write-cache anyway on all my Ubuntu machines. They have 32 GB of memory, and I devote half of it to the cache. That would not be as sophisticated as a squid, but it does not require any special tuning either. I might give squid a try later.

But maybe all the other apps can move to CVMFS instead? That would be interesting.

1 · 2 · Next

Message boards : News : New ATLAS app version released for Linux hosts