LHC@home

Try this:

[sudo] watch -n10 "find /var/lib/boinc-client/slots -name \"log.EVNTtoHITS\" |sort |xargs -I {} -n1 sh -c \"echo "{}"; grep -Po 'INFO.*Run:Event.*\K\(.*' {} |tail -n4; echo\""

...

I modified the command line to monitor ATLAS native 3.01:
In this example, I used 2 CPUs per task hence the tail -n2.

sudo watch -n10 "find /var/lib/boinc-client/slots/ \( -name \"log.EVNTtoHITS\" -o -name \"AthenaMP.log\" \) |sort |xargs -I {} -n1 sh -c \"egrep 'INFO.*Run:Event ' {} |tail -n2\"|sort -k 7,7"

An example of output:

17:32:48 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       390     0    INFO          Run:Event 450000:20848791       (200th event for this worker) took 82.67 s. New average 93.67 +- 3.91
17:32:44 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       391     1    INFO          Run:Event 450000:20848792       (192th event for this worker) took 45.15 s. New average 98.49 +- 3.622
17:32:03 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       362     0    INFO          Run:Event 450000:22570763       (186th event for this worker) took 40.7 s. New average 96.78 +- 3.699
17:32:53 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       363     1    INFO          Run:Event 450000:22570764       (178th event for this worker) took 128.6 s. New average 102.1 +- 4.028
17:33:07 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       312     1    INFO          Run:Event 450000:22644313       (159th event for this worker) took 209.2 s. New average 95.61 +- 3.997
17:33:01 ISF_Kernel_FullG4MT_QS.ISF_LongLivedGeant4Tool       313     0    INFO          Run:Event 450000:22644314       (155th event for this worker) took 152.8 s. New average 99 +- 4.297

Important is the work we do with this Tasks

and not the size of the input file.

But I still can't find any words that claim they are useless for science.

a scientist added a batch to the ATLAS-queue not meant for BOINC

Where did you get that idea from?

https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5976&postid=47891

Probably a scientist added a batch to the ATLAS-queue not meant for BOINC, cause the root-file to download for 1 task is 1110MB.

In case of the 1.996,63 credit points, stderr under "max FLOPS of device" shows 37,11 GFLOPS,
in case of the 211.93 credit points, it shows 4,65 GFLOPS.

How can this come about? As said, we are talking about the same machine and the same environment.
Further, the indicated GFLOPS are never ever in alignment with the runtims / CPu times of the tasks.
So, something seems to be rather wrong with the various calculation in the ATLAS code :-(

The line:
[2022-08-30 06:15:06] Singularity works
is a hardcoded String that the script prints to the log.

It could also print "Wuppdibragglkennsdmined works" and it would still run Apptainer.

In Production is singularity active and multiattach.
Only in -dev is apptainer in use, but there is no new work (Holiday?).

and check with "cvmfs_config stat" if "DIRECT" changes to the local proxy's IP.

cvmfs_config stat atlas

cvmfs_config showconfig -s atlas |grep -E 'CVMFS_(HTTP_PROXY|SERVER_URL|USE_CDN)'

Your Theory task reported a success.

Which Squid did you use?

The one you used in the past or a new one (on the Linux VM)?