LHC@home

Thanks for looking into this. There's a `/cvmfs/grid.cern.ch/vc/containers/runc.new` that also seems to work fine.

Just to be sure, from what I can find, none of the LHC@Home application source code is open, right?

$ runc -v
runc version 1.1.0-0ubuntu1.1
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3

$ /cvmfs/grid.cern.ch/vc/containers/runc -v
runc version spec: 1.0.0

Most distros use cgroup v2 and this should not have been taken out of beta with only v1 support. I have nearly 100 failed tasks now just from this application.

... for newer kernels, cgroup has changed from V1 to V2.
This ends with "\"mountpoint for cgroup not found\" for Theory native ...

This affects all Linux systems using cgroups v2.
Theory's suspend/resume does only support cgroups v1 (freezer).
ATM there's no solution available as it would mean "somebody" would have to write the code to support cgroups v2.

... while Atlas native runs o.k.

Unlike Theory ATLAS (native)
- uses Singularity instead of Runc
- does not support suspend/resume.

Found the guide here : https://apptainer.org/docs/admin/main/installation.html

WU is running a lot longer than it did before so I think I am good now, oddly the CPU and memory usage is minimum, is that normal?

[Service]
Type=simple
ProtectHome=true
ProtectSystem=full
ProtectControlGroups=true
ReadWritePaths=-/var/lib/boinc -/etc/boinc-client
Nice=10
User=boinc
WorkingDirectory=/var/lib/boinc
ExecStart=/usr/bin/boinc
ExecStop=/usr/bin/boinccmd --quit
ExecReload=/usr/bin/boinccmd --read_cc_config
ExecStopPost=/bin/rm -f lockfile
IOSchedulingClass=idle
# The following options prevent setuid root as they imply NoNewPrivileges=true
# Since Atlas requires setuid root, they break Atlas
# In order to improve security, if you're not using Atlas,
# Add these options to the [Service] section of an override file using
# sudo systemctl edit boinc-client.service
#NoNewPrivileges=true
#ProtectKernelModules=true
#ProtectKernelTunables=true
#RestrictRealtime=true
#RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
#RestrictNamespaces=true
#PrivateUsers=true
#CapabilityBoundingSet=
#MemoryDenyWriteExecute=true
#PrivateTmp=true  #Block X11 idle detection

Did you check whether your squid correctly rejects requests initiated from outside your LAN?
I suspect you intercept all HTTP traffic from inside your LAN directly at the router and force it through squid, right?
This means at least traffic to destination port 80.
Are you aware that some CVMFS/Frontier servers use ports 8000 or 8080.
It would be worth to also route them through squid.

You may try out the tuning options from my HowTo and reduce this to 256 MB.
This would leave more RAM for other use on the squid box, e.g. for disk cache.
4-10 GB is suggested to be the CVMFS disk cache size.

If you run Theory vbox each VM will set up it's own CVMFS cache (meanwhile old and degraded).
Hence, each task will send out lots of update requests.
They get all lost when the VM shuts down.
Your squid should cover most of them but its more efficient to run Theory native and keep the data in the local CVMFS cache on the crunching box.

Large downloads happen from time to time although 150 GB within 1 h is very unusual.
CVMFS acts like a cache and tries to use as much data as possible from it's local store.

Do you have the same problem of disconnection with the Virtualbox 6.1.12 from the Boinc-Homepage instead of the 6.1.16?

2020-11-24 14:49:26 (13272): VM state change detected. (old = 'PoweredOff', new = 'Running')

Finally finished: https://lhcathome.cern.ch/lhcathome/result.php?resultid=259641514

===> [runRivet] Mon Jan 20 15:24:51 UTC 2020 [boinc pp jets 8000 800 - sherpa 1.4.1 default 100000 16]

Run time 4 days 12 hours 57 min 5 sec
CPU time 4 days 12 hours 21 min 56 sec

The BOINC data directory must be mounted inside the container, and with a default installation this is /var/lib/boinc-client/slots. If there are problems mounting /var you could try a different data directory or install BOINC in a different place. For example on my desktop I run boinc-client from my home directory because the root partition is too small.