1) Questions and Answers : Getting started : Cron of CERNVM sends lots of e-mail messages to root@localhost on failure/error (Message 48747)
Posted 4 Oct 2023 by alverb
Post:
I use Squid but not in such exact manner but in more complex way. That's why I'm just fine tuning the running configuration by adding ACLs to get work better with LHC@home and others.

Are you aware that none of the addresses below are host or domain names?
Instead they are CVMFS repository names that just look like FQDNs.

Yes, I found that the "addresses" are not host nor domain names but part of URL (eg "http://s1cern-cvmfs.openhtc.io/cvmfs/atlas-nightlies.cern.ch/"). That's why I'm using "url_regex" but not "dstdomain".

The only ACLs I've missed up are in the not caching part. Even without that part all request were direct.

Regarding "mail issue" - till now there are not bulk messages but still all hosts are receiving only "ATLAS Simulation" tasks.

Best Regards!
2) Questions and Answers : Getting started : Cron of CERNVM sends lots of e-mail messages to root@localhost on failure/error (Message 48735)
Posted 3 Oct 2023 by alverb
Post:
I confirm that I've found cron tasks inside the VDI files of LHC@home CernVms that are causing this behavior.
In another Linux VM I've examined two copies of CernVMs .VDI files from different projects (Theory Simulation and ATLAS Simulation) which led me to these conclusions.

The mails were slipping away from the Windows based PCs running LHC@home and received by our Linux based mail server (where by default is local "root" account hence <root@localhost> and alias "postmaster" hence <postmaster@localhost> pointing to "root"). I don't have Linux hosts running LHC@home so I can't confirm that they behave the same way. I think they will do so, as @PekkaH confirmed, because the applications are based on same VDI images.

All this was before doing the steps suggested by @computezrmle.
So I've done the following on all machines running LHC@home (one with 8 GB and one with 16 GB of RAM):
- set LHC@home to "No New Tasks";
- waited all tasks to be reported;
- had done project reset;
- had resumed the work fetch;

- set LHC@home to use no more than 50 % of the available cores.

Since then both hosts had completed several ATLAS Simulation tasks without sending bulk e-mail messages. Till now there are no new tasks from the other LHC@home applications and I can't confirm if they are looking good too.

Just for the test, today I've set back the usage of CPU cores to 100% again.

Concerning connectivity, I have http proxy server in the network and although both PCs are allowed to connect directly to Internet the BOINC client somehow doesn't communicate correctly without explicit proxy settings.
To exclude any rejects, on the proxy server I've set direct connections to hosts and URL patterns containing following Cern hosts:
alice.cern.ch
atlas.cern.ch
atlas-condb.cern.ch
atlas-nightlies.cern.ch
cernvm-prod.cern.ch
cvmfs-config.cern.ch
grid.cern.ch
lhcathome.cern.ch
lhcathome-upload.cern.ch
sft.cern.ch
sft-nightlies.cern.ch
unpacked.cern.ch

I know that would be easier with the whole domain "cern.ch" but I have my considerations not to do so.

I'll try to keep you informed if there are or there aren't any issues.

@computezrmle thank you for your help!
@PekkaH thank you for confirming that I'm not the only one with such issues!
3) Questions and Answers : Getting started : Cron of CERNVM sends lots of e-mail messages to root@localhost on failure/error (Message 48701)
Posted 30 Sep 2023 by alverb
Post:
Sorry, I didn't mention that I've replaced all sensitive data with common ones to protect the real ones. Like "mydomain.tld" with "example.com" etc.
4) Questions and Answers : Getting started : Cron of CERNVM sends lots of e-mail messages to root@localhost on failure/error (Message 48693)
Posted 29 Sep 2023 by alverb
Post:
For a closer look you should
- make your computers visible for other volunteers (https://lhcathome.cern.ch/lhcathome/prefs.php?subset=project)
- post a link to the computer where you see got those logs from
- post a link to an example task that computer has already reported
- describe how/where you got the snippets from


- Computer: https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10641093
- Task: https://lhcathome.cern.ch/lhcathome/result.php?resultid=399846006

I have local mail server which receives the messages in it's root mailbox.

Here is a sample message source:

Return-Path: <root@localhost>
Delivered-To: admin@example.com
Received: from localhost (localhost [127.0.0.1])
by mail.example.com (mail) with ESMTP id ID
for <postmaster@localhost>; Fri, 29 Sep 2023 13:33:43 +0300 (EEST)
X-Virus-Scanned: amavis
Received: from mail.example.com ([127.0.0.1])
by localhost (mail.example.com [127.0.0.1]) (amavis, port 10024)
with ESMTP id ID for <postmaster@localhost>;
Fri, 29 Sep 2023 13:33:43 +0300 (EEST)
Received: from localhost (unknown [1.2.3.4])
by mail.example.com (mail) with SMTP id ID
for <postmaster@localhost>; Fri, 29 Sep 2023 13:33:42 +0300 (EEST)
Received: by localhost (sSMTP sendmail emulation); Fri, 29 Sep 2023 13:33:01 +0300
From: "root" <root@localhost>
Date: Fri, 29 Sep 2023 13:33:01 +0300
To: root
Subject: Cron <root@localhost> rsync -au --delete /home/boinc/cernvm/shared/html/job/ /var/www/html/job/
Content-Type: text/plain; charset=ANSI_X3.4-1968
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=108>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0>
X-Cron-Env: <LANG=C>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

rsync: change_dir "/home/boinc/cernvm/shared/html/job" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]


In the .VDI file of the task I've found "/persistent/etc/crond/sync-plots" cron tab causing this behavior:
* * * * * root rsync -au --delete /home/boinc/cernvm/shared/html/job/ /var/www/html/job/
5) Questions and Answers : Getting started : Cron of CERNVM sends lots of e-mail messages to root@localhost on failure/error (Message 48688)
Posted 29 Sep 2023 by alverb
Post:
On failure/error of scheduled task execution the crond of cernvm sends lots of e-mail messages (about 150 per hour) to root@localhost.
Here are some sample messages:


From: "root" <root@localhost>
To: root
Subject: Cron <root@localhost> rsync -au --delete /home/boinc/cernvm/shared/html/job/ /var/www/html/job/

rsync: change_dir "/home/boinc/cernvm/shared/html/job" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]



From: "root" <root@localhost>
To: root
Subject: Anacron job 'cron.daily' on localhost

/etc/cron.daily/cernvm-update-notification:

Failed to initialize root file catalog (16 - file catalog failure)


Is it possible to stop sending such messages?



©2024 CERN