Message boards : News : New ATLAS app version released for Linux hosts
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
AGLT2
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist

Send message
Joined: 23 Jun 14
Posts: 12
Credit: 6,309,108,646
RAC: 1,130,436
Message 31904 - Posted: 10 Aug 2017, 11:04:25 UTC
Last modified: 10 Aug 2017, 12:15:29 UTC

We released a new version of the ATLAS app today, 2.41 for the x86_64-pc-linux-gnu platform.
The new features of this version include:
1. It requires the host OS to be either Scentific Linux 6 or Cent OS 7.
2. It require CVMFS and Singularity instead of Virtualbox to run the ATLAS jobs.
3. It is more efficient, as the avoidance of using Virtualbox.
Currently, this version is set to beta version.

For people who want to try it out,we provide a script to install everything including CVMFS, singularity here,


Try it if you are interested!
ID: 31904 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2528
Credit: 253,722,201
RAC: 56,522
Message 31905 - Posted: 10 Aug 2017, 11:35:17 UTC - in response to Message 31904.  

We released a new version of the ATLAS app today, 2.41 for the x86_64-pc-linux-gnu platform.
The new features of this version include:
1. It requires the host OS to be either Scentific Linux 6 or Cent OS 7.
2. It require CVMFS and Singularity instead of Virtualbox to run the ATLAS jobs.
3. It is more efficient, as the avoidance of using Virtualbox.
Currently, this version is set to beta version.

For people who want to try it out,we provide a script to install everything including CVMFS, singularity here,
http://atlasathome.cern.ch/boinc_conf/install_cvmfs_sin.sh

Try it if you are interested!

Sounds nice, at least at first glance, but I have some questions:

1. Why is it hosted on the old ATLAS server and not on the dev server?
2. Is it based on the version David Cameron has tested a few months ago?
3. Why is it restricted to the mentioned distributions? Simply not tested on others or due to some specific requirements?
ID: 31905 · Report as offensive     Reply Quote
AGLT2
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist

Send message
Joined: 23 Jun 14
Posts: 12
Credit: 6,309,108,646
RAC: 1,130,436
Message 31906 - Posted: 10 Aug 2017, 12:10:55 UTC - in response to Message 31905.  



1. Why is it hosted on the old ATLAS server and not on the dev server?
2. Is it based on the version David Cameron has tested a few months ago?
3. Why is it restricted to the mentioned distributions? Simply not tested on others or due to some specific requirements?


1. Only the script file for installing cvmfs/singularity is hosted on the old ATLAS server, the app is still on the lhcathome server.
2. Yes, it is based on the native version David tested a few months ago
3. So far we only tested it on SLC6 and CentOS 7, it might work on other Linux, depending on if one can successfully install the CVMFS and Singularity there.
So please feel free to try, and we would appreciate the feedback on that.

Cheers!
ID: 31906 · Report as offensive     Reply Quote
gyllic

Send message
Joined: 9 Dec 14
Posts: 202
Credit: 2,533,875
RAC: 0
Message 31913 - Posted: 10 Aug 2017, 20:36:13 UTC - in response to Message 31906.  
Last modified: 10 Aug 2017, 20:56:21 UTC

Exciting news!

3. So far we only tested it on SLC6 and CentOS 7, it might work on other Linux, depending on if one can successfully install the CVMFS and Singularity there.
So please feel free to try, and we would appreciate the feedback on that.

Im gonna try it on Debian (in a week or so). I will let you know if i was successfull.

Edit:
1. Why is it hosted on the old ATLAS server and not on the dev server?

Good point. The native app is only shown on atlasathome.cern.ch and not on lhcathome.cern.ch or the dev site.
ID: 31913 · Report as offensive     Reply Quote
AGLT2
Volunteer moderator
Project administrator
Project developer
Project tester
Volunteer developer
Project scientist

Send message
Joined: 23 Jun 14
Posts: 12
Credit: 6,309,108,646
RAC: 1,130,436
Message 31914 - Posted: 11 Aug 2017, 1:07:04 UTC - in response to Message 31913.  
Last modified: 11 Aug 2017, 1:07:30 UTC


Good point. The native app is only shown on atlasathome.cern.ch and not on lhcathome.cern.ch or the dev site.


Actually it is on lhcathome.cern.ch,as I mentioned early, it is set to be beta version, so you need to enable "test app" in the lhcathome preference in your account, otherwise, you do not receive jobs from the native app..
ID: 31914 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2240
Credit: 173,894,884
RAC: 3,092
Message 31916 - Posted: 11 Aug 2017, 6:39:51 UTC

Is it multicore or single?

When multicore, the parameter for the number of Cores is missing.

http://atlasathome.cern.ch/apps.php
ID: 31916 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2528
Credit: 253,722,201
RAC: 56,522
Message 31918 - Posted: 11 Aug 2017, 7:17:15 UTC - in response to Message 31916.  

Is it multicore or single?

When multicore, the parameter for the number of Cores is missing.

http://atlasathome.cern.ch/apps.php

There's an app_config.xml included in the package which sets the ncores to 2 and the RAM usage to 4200 MB.
ID: 31918 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 708
Credit: 4,336,250
RAC: 0
Message 31919 - Posted: 11 Aug 2017, 8:42:32 UTC - in response to Message 31904.  

No luck on SuSE Linux 42.2 64-bit, running SixTrack.
Tullio
ID: 31919 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2528
Credit: 253,722,201
RAC: 56,522
Message 31922 - Posted: 11 Aug 2017, 13:03:44 UTC

The current situation is as follows:
1. I installed CVMFS (from CERN) and singularity (from my distro's repository)
2. CVMFS connects to the ATLAS dirs
3. A BOINC client instance is prepared to be attached to the project.


After point 3 there are still open issues.

http://atlasathome.cern.ch/boinc_conf/start_boinc shows the following command:
ProDir/boinccmd --project_attach https://lhcathome.cern.ch/lhcathome/ BOINC_KEY

But there is no ATLAS beta app in the app list.


Or could it be http://atlasathome.cern.ch/ where we have the beta app 2.41 (native_mt) (beta test), but unfortunately no WU to be send.
ID: 31922 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 248
Credit: 5,974,599
RAC: 0
Message 31923 - Posted: 11 Aug 2017, 14:07:50 UTC - in response to Message 31922.  

The app is available as a beta app from this project, LHC@home, and not from the old ATLAS@home. The app version is 241.

I got a few tasks on my desktop, so it should work if /CVMFS is mounted and Singularity installed.

Wenjing and the ATLAS team can provide more details and help.

I suggest to continue the discussion on this app under the ATLAS application sub-forum.
ID: 31923 · Report as offensive     Reply Quote
Profile Olivier Fehr
Avatar

Send message
Joined: 1 Jun 17
Posts: 9
Credit: 964,242
RAC: 0
Message 31924 - Posted: 11 Aug 2017, 14:21:36 UTC
Last modified: 11 Aug 2017, 15:15:50 UTC

Singularity compiled successfully on Ubuntu 17.04 and CentOS 7.3 boxes. Still have to implement the CVMFS part though.
MODIFY: CVMFS installed sucessfully too on CentOS.
ID: 31924 · Report as offensive     Reply Quote
Juha

Send message
Joined: 22 Mar 17
Posts: 30
Credit: 360,676
RAC: 0
Message 31925 - Posted: 11 Aug 2017, 14:22:34 UTC - in response to Message 31923.  

What kind of hardware requirements these task have, RAM, multi-core?

Right now all I get is:

No tasks are available for ATLAS Simulation
Message from server: VirtualBox is not installed
ID: 31925 · Report as offensive     Reply Quote
gyllic

Send message
Joined: 9 Dec 14
Posts: 202
Credit: 2,533,875
RAC: 0
Message 31929 - Posted: 11 Aug 2017, 19:22:52 UTC - in response to Message 31924.  

Singularity compiled successfully on Ubuntu 17.04 and CentOS 7.3 boxes. Still have to implement the CVMFS part though.
MODIFY: CVMFS installed sucessfully too on CentOS.

according to the singularity homepage should singularity be in the repositories for debian stretch and for ubuntu since 16.10, so no need to compile it. but great if it worked.

for cvmfs: i successfully compiled it a couple of months ago on debian jessie, so you should not have to much problems compiling it on ubuntu i guess.
ID: 31929 · Report as offensive     Reply Quote
Profile Olivier Fehr
Avatar

Send message
Joined: 1 Jun 17
Posts: 9
Credit: 964,242
RAC: 0
Message 31930 - Posted: 12 Aug 2017, 7:10:17 UTC - in response to Message 31929.  

Indeed, there is a Singularity package (named singluarity-container) for Ubuntu 17.04. It is version 2.2 while the latest Singularity version on the website is at 2.3.1. Since there have been 2 security related updates since, one labeled as moderate the other as high severity issue, I thought it worthwhile to compile the latest version from source.
As for the CVMFS, I'll make an attempt to compile it on Ubuntu 17.04 as soon as I can.
ID: 31930 · Report as offensive     Reply Quote
Profile Olivier Fehr
Avatar

Send message
Joined: 1 Jun 17
Posts: 9
Credit: 964,242
RAC: 0
Message 31931 - Posted: 12 Aug 2017, 8:19:28 UTC - in response to Message 31930.  

That's a success. I have both Singularity and CVMFS software up and running on both CentOS 7.3 and Ubuntu 17.04. Now lets see if I can get any of the new Atlas packages...
ID: 31931 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 31932 - Posted: 12 Aug 2017, 15:11:58 UTC

It seems to be set up OK on Ubuntu 16.10. But since I don't use a Proxy, that was a bit of a puzzle until I set CVMFS_HTTP_PROXY=DIRECT in the default.local file. Now it all checks out OK, though I have no work yet.

But I am wondering about a SQUID. I have a good Internet connection (25 Mbps down, 4 Mbps up) and don't think I need one. Would it help the CERN servers? Maybe someone can give some setup guidance if so.
ID: 31932 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1060
Credit: 7,737,438
RAC: 2,649
Message 31933 - Posted: 12 Aug 2017, 15:25:35 UTC - in response to Message 31932.  

It seems to be set up OK on Ubuntu 16.10. But since I don't use a Proxy, that was a bit of a puzzle until I set CVMFS_HTTP_PROXY=DIRECT in the default.local file. Now it all checks out OK, though I have no work yet.

But I am wondering about a SQUID. I have a good Internet connection (25 Mbps down, 4 Mbps up) and don't think I need one. Would it help the CERN servers? Maybe someone can give some setup guidance if so.

You really only need a squid if you have more than one cvmfs instance, since cvmfs caches too, as well as the squid. I've been through this at work; I did eventually set up a squid because a server became available, and because I'll probably end up installing cvmfs on all our machines rather than just our Grid user-interface -- when I get enough round tuits.
ID: 31933 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 31934 - Posted: 12 Aug 2017, 15:57:15 UTC - in response to Message 31933.  

Very well. I can use it without worry.
ID: 31934 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2528
Credit: 253,722,201
RAC: 56,522
Message 31935 - Posted: 12 Aug 2017, 20:18:21 UTC

@Ivan, Jim1348

It's a bit OT here, therefore only some very short remarks.

CERN's vbox apps can get a huge benefit from a local squid.
CMS benefits most, then ATLAS and LHCb, Theory less but sufficient.
Up to 95 % regarding the request hits and 50 % regarding the byte hits.
The setup needs some special tuning compared to the standard setup and should get some assist from iptables.

An "old" computer with a core 2 duo, 1-2 GB RAM (250-350 MB for squid), 30-80 GB free disk space is far enough.
And there is also a benefit for fast DSL lines as the squid avoids latency.
ID: 31935 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 31936 - Posted: 12 Aug 2017, 21:11:18 UTC - in response to Message 31935.  

CERN's vbox apps can get a huge benefit from a local squid.
CMS benefits most, then ATLAS and LHCb, Theory less but sufficient.
Up to 95 % regarding the request hits and 50 % regarding the byte hits.
The setup needs some special tuning compared to the standard setup and should get some assist from iptables.

I don't know if it helps much, but I use a large write-cache anyway on all my Ubuntu machines. They have 32 GB of memory, and I devote half of it to the cache. That would not be as sophisticated as a squid, but it does not require any special tuning either. I might give squid a try later.

But maybe all the other apps can move to CVMFS instead? That would be interesting.
ID: 31936 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : News : New ATLAS app version released for Linux hosts


©2024 CERN