LHC@home

Those entries are useful to identify a misconfigured proxy setting.
It does no harm to have it in the log.


This topic has been discussed many times, even in this thread:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5474&postid=44230

Within the Squid package there should be a "cachemgr.cgi".
If you run your own webserver, configure it to use that cgi.

See:
https://wiki.squid-cache.org/Features/CacheManager


Without a webserver "squidclient" can be used to get the same information.

squidclient mgr:info

The output from this example includes a statistic section like this:

Resource usage for squid:
        UP Time:        24673.527 seconds
        CPU Time:       2094.236 seconds
        CPU Usage:      8.49%
        CPU Usage, 5 minute avg:        0.35%
        CPU Usage, 60 minute avg:       9.49%
        Maximum Resident Size: 834032 KB
        Page faults with physical i/o: 0

On a multi CPU computer 'CPU Usage = 100%' would mean 1 core since Squid is a singlecore app.

This make me think you have same troubles.
Here my stats with squid atlas-uploads:

Resource usage for squid:
UP Time: 6801.577 seconds
CPU Time: 1154.614 seconds
CPU Usage: 16.98%
CPU Usage, 5 minute avg: 0.14%
CPU Usage, 60 minute avg: 20.90%
Maximum Resident Size: 1064416 KB
Page faults with physical i/o: 5

With direct atlas-uploads, squid average CPU usage is lower than 0.5%:
Resource usage for squid:
UP Time: 3845.516 seconds
CPU Time: 5.017 seconds
CPU Usage: 0.13%
CPU Usage, 5 minute avg: 0.01%
CPU Usage, 60 minute avg: 0.14%
Maximum Resident Size: 182528 KB
Page faults with physical i/o: 0

While uploading in progress CPU Time rises >90%

Those entries are useful to identify a misconfigured proxy setting.

This is NOT the answer for my question!

...you have same troubles.

What trouble?


I restarted my squid box last night after a kernel update.
Hence, the UP Time (24673 s) from the statistic page was a bit too short for a "good quality" answer.
It will be more relevant after a week or so.

Nonetheless, the average (total) CPU Usage of 8.49 % needs to be divided by the number of concurrently running tasks attached to this squid.
If you assume (based on your observation) that this is mainly caused by uploads, my numbers would be:
25 ATLAS
44 CMS (I force them through Squid just to have the requests in the statistics tool)

Total: 69
Now divide 8.49% by 69 and you get 0.12 %
This is the average singlecore CPU usage squid spends per task.

Peak performance for uploads only matters if squid can't saturate the upload direction of the internet line any more. I never noticed this.

No matter what the real name is.
Feel free to add an alias to your hosts file (or your name resolution service) like:

10.116.178.201 proxy.home.arpa

Then tell BOINC to use the proxy "proxy.home.arpa" and finally that name will appear in the logs.

"home.arpa" is a standard domain for public use similar to the standard IP ranges (10.a.b.c, 192.168...).
It can be used by everyone and guarantees there won't be conflicts with existing domains.
Don't us domains like "example.com" for this.

My answer is NO Squid, solution only from Cern-IT!
http://203.0.113.94:7228

Now divide 8.49% by 69 and you get 0.12 %
This is the average singlecore CPU usage squid spends per task.

Thanks for that clarification.
30 concurrently running ATLAS tasks on my side.

Peak performance for uploads only matters if squid can't saturate the upload direction of the internet line any more. I never noticed this.

No way; my upload line is limited to 1Mb/s by long VDSL-lines.
What I see by system monitor is 100% of one core as long as an upload is in progress.
The main problem is, this slows down all other requests to Squid.

Guess some kind of misconfiguration, but haven't found it yet.

The main problem is, this slows down all other requests to Squid.

How do you monitor this slowdown?
I also notice a 100 % load on 1 core while an upload is in progress but other transfers seem to be unaffected.

This is a typical ATLAS upload from my access.log:

[14/Nov/2022:02:57:46 +0100] "POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler HTTP/1.1" 200 186172617 "-" "BOINC client (x86_64-suse-linux-gnu 7.21.0)" TCP_MISS:HIER_DIRECT

size: 186 MB
typical upload speed: 4 MB/s
time used: ~45 s

Between the timestamps 02:57:00 (estimated start) and 02:57:46 (logged end) my log lists roughly 60 other transfers - various clients, various servers, various methods.

Guess some kind of misconfiguration

As for Squid I'm not aware of an option that could be responsible for this.

My answer is NO Squid, solution only from Cern-IT!
http://203.0.113.94:7228

Example from the night five Atlas (3 with 6 hours and two with 3 hours doing nothing, because of starting and get no right CVMFS connection!
It's time to find a solution for this!
Computer ID 10795955
Laufzeit 5 Stunden 41 min. 10 sek.
CPU Zeit 11 sek.
LAN is completly 1Gbit/s.

How do you monitor this slowdown?

I did a tail -f to see access.log in real time.
What I noticed is:
At the end of a ATLAS WU there are a lot of get requests to atlascern-frontier.openhtc.io.
While uploading the screen scrolls slower then normal.
And yes: This may have other reasons.

typical upload speed: 4 MB/s
time used: ~45 s

My time per WU: 3 minutes
For 120WUs a day this means 6h a day.

btw I hate software that takes full CPU doing about nothing.

Don't blame me. Switched back to direct upload.
So Squid only serves CVMFS and web browsing.
But surely I will have a look at next releases of Squid.

Knowing not enough details for a squid, but
HPC (Highperformance_Computing) need some more experience and details for us to optimize this work for Cern.
Have TByte every month over ISP.
btw, my squid is running on my fastest PC.
CernVM is a idea to let CVMFS on the local network, without Squid.
WLCG (http://wlcg.web.cern.ch)

I did a tail -f to see access.log in real time.
...
While uploading the screen scrolls slower then normal.

You may try to prefix the tail command with "stdbuf -oL " to get a line buffered output

At the end of a ATLAS WU there are a lot of get requests to atlascern-frontier.openhtc.io.

They are also at the beginning of each ATLAS task.
The huge number of requests like those are among the reasons why a local proxy is recommended.
Unlike CVMFS Frontier does not have it's own local cache.

I hate software that takes full CPU doing about nothing.

Well, it processes the data transfer.
That's far away from doing nothing.
:-)

Without a Squid those CPU seconds would be included in other processes' accounting, e.g. BOINC, Vbox or ATLAS itself.

Large downloads are also affected.
Even the internet router's monitoring shows a significant increase in CPU load.
It's mostly not at 100% but I would guess this is caused by the sampling interval plus the built in averaging method.

squid-settings for squid, no frontier-squid:
https://twiki.cern.ch/twiki/bin/view/Frontier/MyOwnSquid

An important option for squid-2.6 and later is:
collapsed_forwarding on

This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.

Have it installed in Squid 5.5 yesterday, no problems so long.

access_log none
This is meaningful, when you don't want so many requests (1GByte per day).
You can activate access.log, when you delete this line in squid.conf.
Be sure, atm you can not control this access.log, when there are problems in squid you want to watch!

I'm using Windows 10.
In BOINC "squid.conf" provided, there is a guide for setting the DNS server that squid uses -

# A MUST on Windows.
# If unsure try the the LAN IP of your internet router.
# Avoid using external DNS here.

dns_nameservers [list of IP addresses]

The documentation page for "Squid configuration directive dns_nameservers" http://www.squid-cache.org/Doc/config/dns_nameservers/ says:

On Windows platforms, if no value is specified here or in
the /etc/resolv.conf file, the list of DNS name servers are
taken from the Windows registry, both static and dynamic DHCP
configurations are supported.

I've tried my Default Gateway IP (LAN IP of the internet router) with no success.
I've tried leaving it out and blank, with no success.
Against the advice I tried an external DNS IP and it worked.

Is there another solution?
Thanks.

---

On your host open a cmd window and type "nslookup".
This returns the IP of your DNS server.

A local DNS server should be preferred in squid.conf since using an external one will be slower and less efficient.
If you expect a DNS server (forwarder) to be present on your router but it doesn't respond, check whether it is accidentally disabled.

Thank you!
nslookup returned an IPv6 address and it works.

PS.
Is there a recommended DNS server to install just for this local BOINC Squid?
Thanks.

---

Could you be so nice and explain what this squid is good for?

Is it to reduce the stress on CERN servers and make something like a decentralized cache storage?