LHC@home

BOINC never worked that way.
The client always tries to use all available resources except if a limit is set by the user.

Just a simple example with project A and project B, both CPU projects.
The project settings on the servers A an B allow you to set a relative weight (default 100).

Your client ensures that long term the runtime usage 1) is given 100/100 to each project.
In case project A doesn't send out work for a while your computer will run only project B.
Once project A starts sending work again your client will automatically prefer project A for a while.

The local work buffer is filled based on expected runtime, not based on number of tasks.


If you plan to spend a fix number of cores, e.g. 4, to only 1 (sub)project (ATLAS) the best solution would be to set up an additional client that is configured to run nothing but ATLAS.


1) simply spoken.
In reality it's much more complex.

UGGGH!!!! (Frustration) (Beating head on desk)
I give up! If its more complex then I just face the reality of the inability of BOINC to work in a way that fits what I want to do.
So then its 4 cores on a single task and roll the dice on when ATLAS will be able to run again. Crazy.
I moved LHC to 200 and WCG to 80 to try and push more towards ATLAS.

Hi, are you running SixTrack, virtually no I/O? Thanks. Eric

CMS - not tweaked? How can one tweak them and what will result?

You can tweak some parameters using an app_config.xml.
See this page for details:
https://boinc.berkeley.edu/wiki/Client_configuration

Your own app_config.xml must strictly follow the template shown there.
Default values can be found in client_state.xml

Mostly used for LHC@home tweaking:
<max_concurrent>n</max_concurrent>
<project_max_concurrent>N</project_max_concurrent>
<avg_ncpus>x</avg_ncpus> # 1)

The VM's RAM size can be tweaked using
<cmdline>--memory_size_mb 2048</cmdline> # 2)


1) The manual explains: "...(possibly fractional) ..." but this makes no sense here since it also tells vboxwrapper how many cores it should configure for the VM. The latter only accepts integer values, hence use "x" or "x.0".

2) 2048 is the default for CMS and doesn't need to be specified here.
Setting it higher would be waste of RAM since a VM never returns allocated RAM to the OS.
Setting it a bit lower will slow down the VM.
Setting it much lower will cause the scientific app not to run since it checks if enough RAM is available.

native Theory:
usually 600-800 MB per task.
BUT! occasionally there will be special tasks (madgraph) that allocate >6.5 GB! plus a 2nd core.

CMS
If not tweaked each task will set up a 2 GB VM.
+ some MB for vboxwrapper

I don't get many stuck VMs anymore.

ATLAS is the most tricky to run, if you want to allow unlimited WUs then it tries to use 10GB of memory per WU, you can of course tweek this but then its hard to keep the memory usage on track manually. CMS runs smoothly.

I don't think there is much disk activity in general, peak total transfers are 38% peak writes are about 40 and 100 for reads.

A squid proxy will reduce the load on the CERN servers and your internet usage.

I use 90% of total thread to give some breathing room, for OS overhead running 42 CMS at once on a 48 thread system this however is 96 GB of ram usage and 100% CPU load.

e.g. on one computer now, I have 14 ATLAS, 17 CMS and 5 theory this is using 98% CPU and 156 GB of memory.

I have twelve machines, spaced to provide heating in the winter.
In the summer, I turn off all except five in the basement, where I do not need to air condition them.

There is not much point in spending money both for running the machines and then for the cost of removing the heat by air conditioning. That is a waste of energy for me.

You are quite welcome. Nothing is easy with native, especially singularity.
It all depends on which version of Linux you have, and what libraries it contains.

I am glad it worked this time.

sudo apt-get install singularity

Even though it says "Singularity works", I haven't had that version of singularity work in years, on Ubuntu 20.04.2 and earlier.

This is what worked for me about six months ago, though it is a fairly safe bet that things have changed by now.
But it might get you started. (First uninstall the version you have, with sudo apt remove singularity, though it probably won't find it anyway).

First: install Dependencies:
sudo apt-get update && sudo apt-get install -y \
build-essential \
libssl-dev \
uuid-dev \
libgpgme11-dev \
squashfs-tools \
libseccomp-dev \
wget \
pkg-config \
git \
cryptsetup

To correct broken packages (examples):
sudo apt install libseccomp2=2.4.3-1ubuntu1
sudo apt install libssl1.1=1.1.1f-1ubuntu2

Second: install GO
sudo snap install go --classic
Result: go 1.16.2 from Michael Hudson-Doyle (mwhudson) installed

go get -u github.com/golang/dep/cmd/dep
(wait for the download!)

Check version: go version

-----------------------------------------------------------------------------------
Install Singularity (check latest version at https://github.com/hpcng/singularity/releases):
See: https://sylabs.io/guides/3.7/user-guide/quick_start.html

go get -d github.com/sylabs/singularity

export VERSION=3.7.2 && \
wget https://github.com/hpcng/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
tar -xzf singularity-${VERSION}.tar.gz && \
cd singularity

./mconfig && \
make -C ./builddir && \
sudo make -C ./builddir install

Check Version: singularity --version

Thank you. I did sudo apt-get install singularity, so we'll see what happens.

I think you get an old version that way, though I have not tried it recently. It might work.

You can get the latetest version by following these instructions:
https://sylabs.io/guides/3.0/user-guide/installation.html
Note that you have to install GO first.

The following works for me on Ubuntu 18.04/20.04.
Not all versions of GO and Singularity work for me, but these do.
You can find the latest versions as indicated.

Install GO (check latest version at https://pkg.go.dev/golang.org/dl#pkg-subdirectories):

export VERSION=1.15.2 OS=linux ARCH=amd64 && \
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
    rm go$VERSION.$OS-$ARCH.tar.gz

echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
    source ~/.bashrc

go get -u github.com/golang/dep/cmd/dep

-----------------------------------------------------------------------------------

Install Singularity (check latest version at https://github.com/hpcng/singularity/releases):

export VERSION=3.6.4 && # adjust this as necessary \
    mkdir -p $GOPATH/src/github.com/sylabs && \
    cd $GOPATH/src/github.com/sylabs && \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
    tar -xzf singularity-${VERSION}.tar.gz && \
    cd ./singularity && \
    ./mconfig

./mconfig && \
    make -C ./builddir && \
    sudo make -C ./builddir install

Check Version: singularity --version

Thank you. I did sudo apt-get install singularity, so we'll see what happens.

I think you get an old version that way, though I have not tried it recently. It might work.

You can get the latetest version by following these instructions:
https://sylabs.io/guides/3.0/user-guide/installation.html
Note that you have to install GO first.

The following works for me on Ubuntu 18.04/20.04.
Not all versions of GO and Singularity work for me, but these do.
You can find the latest versions as indicated.

Install GO (check latest version at https://pkg.go.dev/golang.org/dl#pkg-subdirectories):

export VERSION=1.15.2 OS=linux ARCH=amd64 && \
    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
    rm go$VERSION.$OS-$ARCH.tar.gz

echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
    source ~/.bashrc

go get -u github.com/golang/dep/cmd/dep

-----------------------------------------------------------------------------------

Install Singularity (check latest version at https://github.com/hpcng/singularity/releases):

export VERSION=3.6.4 && # adjust this as necessary \
    mkdir -p $GOPATH/src/github.com/sylabs && \
    cd $GOPATH/src/github.com/sylabs && \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
    tar -xzf singularity-${VERSION}.tar.gz && \
    cd ./singularity && \
    ./mconfig

./mconfig && \
    make -C ./builddir && \
    sudo make -C ./builddir install

Check Version: singularity --version

have the same info from CentOS8.
After installing singularity from OS - Tasks are running well.
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5613

CMS tasks take about 2500 MB memory if I remember correctly. But they always use only 1 CPU core so not much you can do there. Their run time (if everything is working properly) is 12...18 hours. They run so that when 12 hours is full, they look for point where the currently running job is finished and finish the task there. If the task is not finished at 18 hours they will be terminated by Boinc.

Perhaps that file could be updated with the minimum needed configuration ...

The file on the server is already up to date.
Be aware that it includes 2 optional settings (with proxy/without proxy) and one of them has to be activated by the user.

In general:
Native apps require more settings to be done by the user.
This is easier, faster and more reliable than to guess certain values.
In addition some steps require to be done by root.

although I'm still unclear how one can actually optimize their configuration

The simple answer

Cache as much as possible as close as possible to the point were it is used.
To avoid less efficient effort focus on the major bottlenecks first.


More LHC@home specific

CVMFS is heavily used but has it's own cache - one cache instance per machine.
A machine can't share it's CVMFS cache with other machines.
Each VM counts as individual machine.
Outdated or missing data is requested from the project servers.

Frontier is heavily used by ATLAS and CMS. It has no own local cache.
Each app sends all Frontier requests to the project servers.


Cloudflare's openhtc.io infrastructure helps to distribute CVMFS and Frontier data.
They run a very fast worldwide network and one of their proxy caches will most likely be located much closer to your clients than any project server.

VBox apps use openhtc.io by default but users running native apps have to set "CVMFS_USE_CDN=yes" in their CVMFS configuration.
This is disabled in the default configuration because lots of computers in various datacenters use special connections and require this to be set "OFF".


A local HTTP proxy closes the gap between openhtc.io and the local clients.
It can cache data for all local CVMFS and Frontier clients as well as offload openhtc.io and the project servers.