Message boards :
ATLAS application :
Guide for building everything from sources to run native ATLAS on Debian 9 (Stretch) Version 2
Message board moderation
Author | Message |
---|---|
Send message Joined: 9 Dec 14 Posts: 202 Credit: 2,533,875 RAC: 0 ![]() ![]() |
Hi Guys! This is Version 2 of a short guide for building every program you need to run native ATLAS tasks on Debian 9 (Stretch) from sources. For Version 1 and all the corresponding discussions look here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4703. To get an “overview†of the native Linux ATLAS app, see here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4617. Why Version 2 of this guide? Since singularity (container software that is needed) has been completely rewritten in Go, the build process works differently compared to that one described in Version 1. Additionally, it is now shown how to build the BOINC manager (optional) from source. Also, more comments and more detailed instructions, especially for the CVMFS setup, have been added as well as some additional informations. Why use the native ATLAS app? There are mainly two reason why you should consider running the native ATLAS app: higher efficiency and lower hardware requirements compared to the Virtual Box ATLAS app. This is achieved by a couple of measures/properties of the native ATLAS app: • No start up and shutdown phase of the virtual box instance is needed. • You don’t lose CPU power for virtualization or emulation of an OS (which you do when using the Virtual Box app) if you use the native app • The native app needs much less RAM compared to the virtual box app. • You can use a local CVMFS cache that is used for every new task (if you use the virtual box app this cache gets deleted once the calculation of one task has finished and has to be filled up again for the next task) which saves internet bandwidth and leads to a faster start of the actual calculations. • You can configure CVMFS to use openhtc.io, which uses cloudflare’s CDN, and therefore benefits from all the advantages of a CDN (see the “build and install cvmfs†section on how to set it up) What do you need to run the native ATLAS app? - Linux (Debian is used for this guide, but others are possible) https://debian.org - CVMFS (cern virtual machine file system) https://cernvm.cern.ch/portal/filesystem - singularity (container) https://www.sylabs.io/ - BOINC https://boinc.berkeley.edu/ At the time this is written, Debian is in version 9.5, BOINC in version 7.13.0, CVMFS in version 2.6.0 and singularity in version 3.0.0-beta1. Unfortunately it is not possible to run ONLY native ATLAS and other projects that use a virtualbox approach at the same time with the same boinc instance. The setup described here will work, but it is more or less random if you get native ATLAS tasks or virtual box ATLAS tasks. There are three ways to force to get only native ATLAS tasks: • The most elegant way would be to install two boinc instances on the same machine. One instance will be used to crunch virtualbox tasks only, while the second boinc instance will crunch native ATLAS only (thanks computezrmle for the info). You probably will find some guides in the internet on how to setup two or more boinc instances on the same machine. • Deinstall/remove virtualbox completely from your PC (of course, all virtualbox based tasks won’t function any more), • or tell the BOINC client to ignore it (of course, all virtualbox based tasks won’t function any more). Look here for more informations: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4617&postid=34426. You also have to accept to run beta apps in your LHC@home settings here https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project. You can install most of the programs from repositories, but this guide shows you how to compile and set up everything from source. With compiling the programs from sources, you get the latest available code (and not the sometimes rather old packages from the repositories). Build environment: I have set up a new PC with a Debian 9.5 net installation and have installed Debian Stretch with the GNOME desktop environment. To run native ATLAS you don’t need a desktop environment, so a “server configuration†should be sufficient and this guide should work exactly the same. However, it was only tested on the mentioned configuration. If you run in “server modeâ€, you only have to compile the boinc client (more details in the “Compile boinc†section). If you don’t use a desktop environment building the boinc manger will not work (at least not that way how it is described here) and would not make a lot of sense. Additionally, I have installed sudo and added my user to the sudo group. You need to have root rights on your PC. The same process may also work on other Linux OS's (especially on Debian derivates like Ubuntu), but it was only tested on Debian 9 (Stretch). Let's start to compile and install boinc from sources. Open a terminal and type the following: 1. update packages and install all new required packages: sudo apt updateTo build only the boinc client install these packages: sudo apt install git build-essential pkg-config libsqlite3-dev libssl-dev libcurl4-openssl-dev m4 dh-autoreconf zlib1g-devIf you want to build the boinc client and the boinc manager, install these packages (a desktop environment has already to be installed in order to be able to build the boinc manager): sudo apt install git build-essential pkg-config libsqlite3-dev libssl-dev libcurl4-openssl-dev m4 dh-autoreconf zlib1g-dev freeglut3-dev libwxgtk3.0-dev libwxgtk-webview3.0-dev libjpeg-dev libnotify-dev libxmu-dev libxi-dev libgtk2.0-dev 2. Add user boinc (no root rights required, and I chose no password): sudo adduser boinc 3. Change to user boinc, make a new directory and clone the git repository into it: su su boinc cd mkdir boinc_source git clone https://github.com/BOINC/boinc.git boinc_source cd boinc_sourceIf you want to compile the newest available code, continue with point 4. If you want to compile a specific version, e.g. 7.12.0, do the following: To get an overview of all tagged code versions, type git tagTo checkout version 7.12.0 for example, type git checkout tags/client_release/7.12/7.12.0 4. To compile and install boinc, type ./_autosetupTo compile only the client, use ./configure --disable-server --disable-manager --enable-optimizeTo compile client and manager type (building the boinc manager will probably only work if you have a desktop environment installed) ./configure --disable-server --enable-optimizeFinally, compile everything (will take a while) and install it make su make install 5. To generate needed files, start and stop the client: /usr/local/etc/init.d/boinc-client start /usr/local/etc/init.d/boinc-client stop exit cd 6. To be able to manage this boinc client with the boinc manager from the same machine or from another PC, you have to edit the "gui_rpc_auth.cfg" file which is located in the boinc (user) home directory. A random password is already written in this file. Write a new password of your choice in the file if you want, save and close it. I have chosen no password. On a multi-user computer, this file should be protected against access by other users. Additionally, the file "remote_hosts.cfg" has to be created in the boinc (user) home directory. Write all IPs in this file from which you want to access this boinc client, e.g.: 127.0.0.1 192.168.0.2 192.168.0.3The IP 127.0.0.1 is needed if you want to control the client with the compiled boinc manager if the manager and the client are on the same machine. 7. Start the boinc client again (as root). Now you should be able to connect to your client with boinc manager, which can be started by typing boincmgr into the terminal (if your boinc manager can’t connect with the desired client, try to reboot the PC where the client you want to connect to is installed). If the boinc client does not start after a PC restart automatically, use the above commands to start it. Next, add LHC@home and adjust the settings as needed (you maybe have to increase the allowed memory, disk space, ...). Boinc should now be good to go. Build and install cvmfs: 1. Change to a different (e.g. your) user, e.g. testing su testing 2. Install all required packages: sudo apt install cmake uuid-dev libfuse-dev python-dev libcap-dev attr autofsThese packages have to be installed. All other libraries etc. that are necessary to build CVMFS will be build automatically from the externals directory within the CVMFS sources. To get an overview of all of these needed packages, you can look here: https://github.com/cvmfs/cvmfs/tree/devel/externals. Probably another way would be to install these libraries etc. from Debian’s repositories (most of them should be available). Here, we use the provided sources within the CVMFS source code. 3. Make a directory, clone the git repository and create the needed directory structure (again, if you don’t want to use the latest available code and build a specific version, use the git tag and git checkout commands as shown in the boinc section): cd mkdir cvmfs_source git clone https://github.com/cvmfs/cvmfs.git cvmfs_source cd cvmfs_source mkdir -p build cd build 4. Build (this will take a while) and install cvmfs: cmake ../ make sudo make install 5. Make a directory that will be used as CVMFS cache. Here in this example the cache location will be /scratch/cvmfs (but you can use other locations as well): sudo mkdir -p /scratch/cvmfs 6. Now we have to configure CVMFS. To do so, add the file default.local into /etc/cvmfs/ sudo nano /etc/cvmfs/default.localand add the following to it: CVMFS_REPOSITORIES=atlas.cern.ch,atlas-condb.cern.ch,grid.cern.ch CVMFS_CACHE_BASE=/scratch/cvmfs CVMFS_QUOTA_LIMIT=4096 CVMFS_HTTP_PROXY=DIRECTThe CVMFS_REPOSITORIES variable defines all repositories CVMFS will use. The shown above are needed in order to run the native ATLAS app. The CVMFS_CACHE_BASE variable defines where CVMFS will place/search your local cache directory. If you have chosen a different location than the one shown in step 5, you will need to write your location here instead. The CVMFS_QUOTA_LIMIT variable defines the size of the local cache directory in Megabytes (in this example 4GB will be used). It is important to know that this is a soft quota. Using a cache size of 4GB will result in CVMFS hitrates of about 99%. If you have to (or want to) use a proxy server for CVMFS, you should change the CVMFS_HTTP_PROXY variable (e.g. CVMFS_HTTP_PROXY="http://192.168.1.15:3128†if your proxy server is at 192.168.1.15 and listen on port 3128). If you don’t want to use a proxy, use the DIRECT input as shown in the example above. 7. Now set up CVMFS for the first time: sudo cvmfs_config setup 8. Test if CVMFS is working (you should get all "Ok"). cvmfs_config probeIf it fails, try "sudo service autofs restart" and try to probe again. 9. If you want to use openhtc.io for your CVMFS setup, follow this excellent guide from computezrmle https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4758. Build and install singularity: This section is completely different compared to version 1 of this guide since singularity has been completely rewritten in Go. At the time this is written, singularity 3 is in version 3.0.0 Beta 1. Since it is still in beta, it is possible that the build process will change in near future (although rather unlikely). To get the latest process, look here https://github.com/sylabs/singularity/blob/master/INSTALL.md. The following is basically the same as it is written in the linked file. 1. First, install all required packages: sudo apt-get install build-essential libssl-dev uuid-dev libgpgme11-dev 2. To build version 3.x, follow this guide. If you (whyever) want to build singularity 2.x, use the same git tag and git checkout commands as shown above and go to version 1 of this guide where the process is described. To build version 3.x do the following (since golang can be a little bit finicky to setup, it is probably a good idea to follow exactly this guide): 3. Install golang Go to the golang homepage https://golang.org/dl/ and choose the version you want and download it. Here we use version 1.11 for linux 64 bit. cd wget https://dl.google.com/go/go1.11.linux-amd64.tar.gzExtract the archive to /usr/local sudo tar -C /usr/local -xzf go1.11.linux-amd64.tar.gzSet up the environment for go (this will adapt your .bashrc file (add go bin paths), and the GOPATH variable will point to a directory called “go†which will be located within your home directory). The following shown commands will only work if you extracted the go tar file into /usr/local.: echo 'export GOPATH=${HOME}/go' >> ~/.bashrc echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc source ~/.bashrcNow clone the singularity repository mkdir -p $GOPATH/src/github.com/sylabs cd $GOPATH/src/github.com/sylabs git clone https://github.com/sylabs/singularity.git cd singularityand finally install the golang dependencies: go get -u -v github.com/golang/dep/cmd/dep 4. Finally compile and install singularity (needed dependencies will be downloaded automatically): cd $GOPATH/src/github.com/sylabs/singularity ./mconfig cd ./builddir make sudo make install 3. Test if it works (you should get an output): singularity --version If everything worked well, you should now have everything you need to run native ATLAS tasks on Debian 9. Again, feel free to report mistakes, improvements, ask questions, make any suggestions, etc. It would be much appreciated! Gyllic |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 ![]() ![]() |
Unfortunately it is not possible to run ONLY native ATLAS and other projects that use a virtualbox approach at the same time with the same boinc instance. The setup described here will work, but it is more or less random if you get native ATLAS tasks or virtual box ATLAS tasks. There are three ways to force to get only native ATLAS tasks: The advice at that link didn't work for me. After adding the recommended line to cc_config.xml and rebooting the client continued fetching ATLAS VBox tasks . I also had to edit client_state.xml and client_state_prev.xml and change "<p_vm_extensions_disabled>0</p_vm_extensions_disabled>" to "<p_vm_extensions_disabled>1</p_vm_extensions_disabled>". |
Send message Joined: 13 May 14 Posts: 387 Credit: 15,314,184 RAC: 0 ![]() ![]() |
Singularity version 3 was officially released yesterday: https://www.sylabs.io/2018/10/sylabs-announces-new-release-of-singularity-to-expand-current-ecosystem-of-resources-that-tackles-todays-most-demanding-workloads/ Can anyone confirm if ATLAS tasks work with this version? From gyllic's post it looks like the beta version had no problems but would be good to know if we can give the all clear for the official release. |
Send message Joined: 9 Dec 14 Posts: 202 Credit: 2,533,875 RAC: 0 ![]() ![]() |
Can anyone confirm if ATLAS tasks work with this version?I cloned the current git repository, checked out tag v3.0.0, compiled singularity and ran an native ATLAS task. It worked without any problems. The output of "singularity --version" is: "singularity version v3.0.0" The corresponding task that was crunched with this singularity version is: https://lhcathome.cern.ch/lhcathome/result.php?resultid=207507345 So yes, looks like it is working. |
Send message Joined: 13 May 14 Posts: 387 Credit: 15,314,184 RAC: 0 ![]() ![]() |
Great, thanks for testing this out! |
![]() Send message Joined: 30 Aug 14 Posts: 145 Credit: 10,847,070 RAC: 0 ![]() ![]() |
Hi! At first let me thank you for taking the time to create this guide! My question: What are the current RAM requirements for native Atlas tasks? Thanks and regards, djoser. Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us |
![]() Send message Joined: 15 Jun 08 Posts: 2473 Credit: 244,948,365 RAC: 171,287 ![]() ![]() ![]() |
My question: What are the current RAM requirements for native Atlas tasks? You may follow the discussion here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4875 |
![]() Send message Joined: 30 Aug 14 Posts: 145 Credit: 10,847,070 RAC: 0 ![]() ![]() |
I was aware of that discussion, but the guys over there are still "fighting" about the correct formula. One says it's 2000 + 2000 MB * nCores, antother says it's 100 + 2000 MB * nCores. So i tried my luck over here :-) Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us |
![]() Send message Joined: 15 Jun 08 Posts: 2473 Credit: 244,948,365 RAC: 171,287 ![]() ![]() ![]() |
Why do you think you would get other numbers here? OK, no answer necessary as it would be OT. Just the following points: 1. Your computers (4 cores, 8 GB RAM) are most likely able to saturate all cores with ATLAS native. 2. I may be able to run a test series in 1 or 2 weeks to answer the question over there. |
![]() Send message Joined: 30 Aug 14 Posts: 145 Credit: 10,847,070 RAC: 0 ![]() ![]() |
Sorry for making you get annoyed... I'm currently setting up new machine (Ryzen 5 2400G 4c/8t with 16GB RAM) and wanted to run Atlas native on it. That's why i was asking... Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! https://gridcoin.us |
Send message Joined: 19 Feb 08 Posts: 708 Credit: 4,336,250 RAC: 0 ![]() ![]() |
I have a new Windows 10 HP PC with a Ryzen 5 1400 and a GTX 1060. Unfortunately, all my Atlas@home tasks fail because AMD-V is not enabled in BIOS. I have asked the HP Support to explain to me how I change the BIOS but so far no answer. SETI@home tasks, both GPU and CPU, idem Einstein and GPUGRID all run 8 CPU tasks and one GPU task successfully. The CPU has 4 cores but runs eight CPU tasks. Tullio |
![]() Send message Joined: 15 Jun 08 Posts: 2473 Credit: 244,948,365 RAC: 171,287 ![]() ![]() ![]() |
I have a new Windows 10 HP PC with a Ryzen 5 1400 and a GTX 1060. Unfortunately, all my Atlas@home tasks fail because AMD-V is not enabled in BIOS. I have asked the HP Support to explain to me how I change the BIOS but so far no answer. Is it this what you are looking for? https://support.hp.com/us-en/document/c04760917 On Ryzen-Systems it may be called "SVM Mode" or similar. |
Send message Joined: 19 Feb 08 Posts: 708 Credit: 4,336,250 RAC: 0 ![]() ![]() |
Cannot find any virtualization option in the Ryzen BIOS, upgraded. They should have warned, I would not have bought a Ryzen PC had I known it. I am running Atlas@home on my old Opteron 1210, vintage 2008. Tullio |
Send message Joined: 19 Feb 08 Posts: 708 Credit: 4,336,250 RAC: 0 ![]() ![]() |
Maybe HP, which produces Omen as a gaming rig, does not want it to be used for scientific computing and forbids modification to its BIOS to admit virtualization. Tullio |
Send message Joined: 26 Oct 18 Posts: 93 Credit: 4,188,598 RAC: 0 ![]() ![]() |
AMD-V is not enabled in BIOS Where do you actually see that? Where does it read? This page... https://linustechtips.com/main/topic/783613-virtualization-in-ryzen-5-1600/ A: "All I could suggest is installing Hyper-V from within Windows features, reboot, disable it, then reboot again." B: "I have it enabled in Windows features, I have SVM enabled, check if virtualization is on, and then still the same. I also check if Hyper-V Virtual Hardware management is running and it is." A: "Yes, but you have to disable it for VirtualBox or VMWare to work properly (allow 64 bit guests)." Then, https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v "Requirements... Windows 10 Enterprise, Pro, or Education" "The Hyper-V role cannot be installed on Windows 10 Home" I see your machine is running Windows 10 Core Edition, which is pretty much the same as Windows 10 Home, if I remember right. Maybe that could be a problem? Could it require a 'larger' version of Windows 10 for Ryzen to support virtualization. |
Send message Joined: 19 Feb 08 Posts: 708 Credit: 4,336,250 RAC: 0 ![]() ![]() |
I think I found the right menu in F10. Unfortunately, the menus are in Italian and they don't correspond to the online manuals, which are English. Lost in translation. I have a Theory Simulation running on 8 cores, and more Atlas@home tasks in a waiting state. Tullio |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 ![]() ![]() |
I tried building CVMFS today on Lubuntu 18.10 and got errors in step 3. All the required packages mentioned in step 2 installed error free but in step 3 the command "cmake ../" ended with: --> Extracting MaxMind-DB-Reader-python-1.4.0.tar.gz MaxMind-DB-Reader-python-1.4.0/ MaxMind-DB-Reader-python-1.4.0/.gitignore MaxMind-DB-Reader-python-1.4.0/.gitmodules MaxMind-DB-Reader-python-1.4.0/.pylintrc MaxMind-DB-Reader-python-1.4.0/.travis-yapf.sh MaxMind-DB-Reader-python-1.4.0/.travis.yml MaxMind-DB-Reader-python-1.4.0/.uncrustify.cfg MaxMind-DB-Reader-python-1.4.0/HISTORY.rst MaxMind-DB-Reader-python-1.4.0/LICENSE MaxMind-DB-Reader-python-1.4.0/MANIFEST.in MaxMind-DB-Reader-python-1.4.0/README.rst MaxMind-DB-Reader-python-1.4.0/dev-bin/ MaxMind-DB-Reader-python-1.4.0/dev-bin/release.sh MaxMind-DB-Reader-python-1.4.0/docs/ MaxMind-DB-Reader-python-1.4.0/docs/Makefile MaxMind-DB-Reader-python-1.4.0/docs/conf.py MaxMind-DB-Reader-python-1.4.0/docs/index.rst MaxMind-DB-Reader-python-1.4.0/examples/ MaxMind-DB-Reader-python-1.4.0/examples/benchmark.py MaxMind-DB-Reader-python-1.4.0/maxminddb/ MaxMind-DB-Reader-python-1.4.0/maxminddb/__init__.py MaxMind-DB-Reader-python-1.4.0/maxminddb/compat.py MaxMind-DB-Reader-python-1.4.0/maxminddb/const.py MaxMind-DB-Reader-python-1.4.0/maxminddb/decoder.py MaxMind-DB-Reader-python-1.4.0/maxminddb/errors.py MaxMind-DB-Reader-python-1.4.0/maxminddb/extension/ MaxMind-DB-Reader-python-1.4.0/maxminddb/extension/maxminddb.c MaxMind-DB-Reader-python-1.4.0/maxminddb/file.py MaxMind-DB-Reader-python-1.4.0/maxminddb/reader.py MaxMind-DB-Reader-python-1.4.0/setup.cfg MaxMind-DB-Reader-python-1.4.0/setup.py MaxMind-DB-Reader-python-1.4.0/tests/ MaxMind-DB-Reader-python-1.4.0/tests/data/ MaxMind-DB-Reader-python-1.4.0/tests/decoder_test.py MaxMind-DB-Reader-python-1.4.0/tests/reader_test.py --> Building maxminddb Traceback (most recent call last): File "setup.py", line 12, in <module> from setuptools import setup, Extension, Feature ImportError: No module named setuptools CMake Error at CMakeLists.txt:189 (message): bootstrapping failed -- Configuring incomplete, errors occurred! See also "/home/bronco/cvmfs_source/build/CMakeFiles/CMakeOutput.log". Nothing in /home/bronco/cvmfs_source/build/CMakeFiles/CMakeOutput.log seemed related to the above errors but I kept a copy of it anyway. |
Send message Joined: 9 Dec 14 Posts: 202 Credit: 2,533,875 RAC: 0 ![]() ![]() |
Looks like you are missing the python setuptools package. To fix your problem, type: sudo apt install python-setuptoolsand execute the cmake ../command again. Then follow the shown procedure. |
Send message Joined: 13 Apr 18 Posts: 443 Credit: 8,438,885 RAC: 0 ![]() ![]() |
Thanks, gyllic, that got cvmfs built and working. The host is https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10576391 Now a problem with singularity. It built and installed OK and "singularity --version" reports singularity version 3.0.1-145.gc9822fec. Stderr output from 2 failed tasks have This is not SLC6, need to run with Singularity.... Checking Singularity... Singularity is installed copy /var/lib/boinc/slots/0/shared/start_atlas.sh copy /var/lib/boinc/slots/0/shared/input.tar.gz copy /var/lib/boinc/slots/0/shared/RTE.tar.gz copy /var/lib/boinc/slots/0/shared/ATLAS.root_0 start atlas job with PandaID=4164466230 Testing the function of Singularity... check singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname Singularity isnt working... running start_atlas return value is 3 tar czvf shared/result.tar.gz tar: Cowardly refusing to create an empty archive Try 'tar --help' or 'tar --usage' for more information. |
Send message Joined: 9 Dec 14 Posts: 202 Credit: 2,533,875 RAC: 0 ![]() ![]() |
Now a problem with singularity. It built and installed OK and "singularity --version" reports To get more information, please do the following: 1. probe cvmfs: cvmfs_config probe 2. Post the output of the command: singularity --debug exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname |
©2024 CERN