Message boards : Number crunching : Setting up a local Squid to work with LHC@home - Comments and Questions
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6

AuthorMessage
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47525 - Posted: 13 Nov 2022, 6:56:40 UTC - in response to Message 47521.  

Those entries are useful to identify a misconfigured proxy setting.
It does no harm to have it in the log.


This topic has been discussed many times, even in this thread:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5474&postid=44230
ID: 47525 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47526 - Posted: 13 Nov 2022, 7:16:59 UTC - in response to Message 47524.  

Within the Squid package there should be a "cachemgr.cgi".
If you run your own webserver, configure it to use that cgi.

See:
https://wiki.squid-cache.org/Features/CacheManager


Without a webserver "squidclient" can be used to get the same information.
squidclient mgr:info

The output from this example includes a statistic section like this:
Resource usage for squid:
        UP Time:        24673.527 seconds
        CPU Time:       2094.236 seconds
        CPU Usage:      8.49%
        CPU Usage, 5 minute avg:        0.35%
        CPU Usage, 60 minute avg:       9.49%
        Maximum Resident Size: 834032 KB
        Page faults with physical i/o: 0

On a multi CPU computer 'CPU Usage = 100%' would mean 1 core since Squid is a singlecore app.
ID: 47526 · Report as offensive     Reply Quote
Saturn911

Send message
Joined: 3 Nov 12
Posts: 10
Credit: 75,635,299
RAC: 126,163
Message 47527 - Posted: 13 Nov 2022, 14:55:56 UTC - in response to Message 47526.  

This make me think you have same troubles.
Here my stats with squid atlas-uploads:

Resource usage for squid:
UP Time: 6801.577 seconds
CPU Time: 1154.614 seconds
CPU Usage: 16.98%
CPU Usage, 5 minute avg: 0.14%
CPU Usage, 60 minute avg: 20.90%
Maximum Resident Size: 1064416 KB
Page faults with physical i/o: 5

With direct atlas-uploads, squid average CPU usage is lower than 0.5%:
Resource usage for squid:
UP Time: 3845.516 seconds
CPU Time: 5.017 seconds
CPU Usage: 0.13%
CPU Usage, 5 minute avg: 0.01%
CPU Usage, 60 minute avg: 0.14%
Maximum Resident Size: 182528 KB
Page faults with physical i/o: 0

While uploading in progress CPU Time rises >90%
ID: 47527 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47528 - Posted: 13 Nov 2022, 15:56:22 UTC - in response to Message 47525.  

Those entries are useful to identify a misconfigured proxy setting.

This is NOT the answer for my question!
ID: 47528 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47529 - Posted: 13 Nov 2022, 17:25:54 UTC - in response to Message 47527.  

...you have same troubles.

What trouble?


I restarted my squid box last night after a kernel update.
Hence, the UP Time (24673 s) from the statistic page was a bit too short for a "good quality" answer.
It will be more relevant after a week or so.

Nonetheless, the average (total) CPU Usage of 8.49 % needs to be divided by the number of concurrently running tasks attached to this squid.
If you assume (based on your observation) that this is mainly caused by uploads, my numbers would be:
25 ATLAS
44 CMS (I force them through Squid just to have the requests in the statistics tool)

Total: 69
Now divide 8.49% by 69 and you get 0.12 %
This is the average singlecore CPU usage squid spends per task.

Peak performance for uploads only matters if squid can't saturate the upload direction of the internet line any more. I never noticed this.
ID: 47529 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47530 - Posted: 13 Nov 2022, 17:37:54 UTC - in response to Message 47528.  

No matter what the real name is.
Feel free to add an alias to your hosts file (or your name resolution service) like:
10.116.178.201 proxy.home.arpa


Then tell BOINC to use the proxy "proxy.home.arpa" and finally that name will appear in the logs.

"home.arpa" is a standard domain for public use similar to the standard IP ranges (10.a.b.c, 192.168...).
It can be used by everyone and guarantees there won't be conflicts with existing domains.
Don't us domains like "example.com" for this.
ID: 47530 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47532 - Posted: 13 Nov 2022, 21:27:10 UTC - in response to Message 47530.  
Last modified: 13 Nov 2022, 22:05:06 UTC

My answer is NO Squid, solution only from Cern-IT!
http://203.0.113.94:7228
ID: 47532 · Report as offensive     Reply Quote
Saturn911

Send message
Joined: 3 Nov 12
Posts: 10
Credit: 75,635,299
RAC: 126,163
Message 47533 - Posted: 13 Nov 2022, 22:07:13 UTC - in response to Message 47529.  
Last modified: 13 Nov 2022, 22:13:27 UTC

Now divide 8.49% by 69 and you get 0.12 %
This is the average singlecore CPU usage squid spends per task.

Thanks for that clarification.
30 concurrently running ATLAS tasks on my side.

Peak performance for uploads only matters if squid can't saturate the upload direction of the internet line any more. I never noticed this.

No way; my upload line is limited to 1Mb/s by long VDSL-lines.
What I see by system monitor is 100% of one core as long as an upload is in progress.
The main problem is, this slows down all other requests to Squid.

Guess some kind of misconfiguration, but haven't found it yet.
ID: 47533 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47535 - Posted: 14 Nov 2022, 9:04:26 UTC - in response to Message 47533.  

The main problem is, this slows down all other requests to Squid.

How do you monitor this slowdown?
I also notice a 100 % load on 1 core while an upload is in progress but other transfers seem to be unaffected.

This is a typical ATLAS upload from my access.log:
[14/Nov/2022:02:57:46 +0100] "POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler HTTP/1.1" 200 186172617 "-" "BOINC client (x86_64-suse-linux-gnu 7.21.0)" TCP_MISS:HIER_DIRECT

size: 186 MB
typical upload speed: 4 MB/s
time used: ~45 s

Between the timestamps 02:57:00 (estimated start) and 02:57:46 (logged end) my log lists roughly 60 other transfers - various clients, various servers, various methods.

Guess some kind of misconfiguration

As for Squid I'm not aware of an option that could be responsible for this.
ID: 47535 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47536 - Posted: 15 Nov 2022, 5:00:48 UTC - in response to Message 47532.  
Last modified: 15 Nov 2022, 5:40:52 UTC

My answer is NO Squid, solution only from Cern-IT!
http://203.0.113.94:7228

Example from the night five Atlas (3 with 6 hours and two with 3 hours doing nothing, because of starting and get no right CVMFS connection!
It's time to find a solution for this!
Computer ID 10795955
Laufzeit 5 Stunden 41 min. 10 sek.
CPU Zeit 11 sek.
LAN is completly 1Gbit/s.
ID: 47536 · Report as offensive     Reply Quote
Saturn911

Send message
Joined: 3 Nov 12
Posts: 10
Credit: 75,635,299
RAC: 126,163
Message 47547 - Posted: 19 Nov 2022, 9:38:07 UTC - in response to Message 47535.  

How do you monitor this slowdown?

I did a tail -f to see access.log in real time.
What I noticed is:
At the end of a ATLAS WU there are a lot of get requests to atlascern-frontier.openhtc.io.
While uploading the screen scrolls slower then normal.
And yes: This may have other reasons.

typical upload speed: 4 MB/s
time used: ~45 s

My time per WU: 3 minutes
For 120WUs a day this means 6h a day.

btw I hate software that takes full CPU doing about nothing.

Don't blame me. Switched back to direct upload.
So Squid only serves CVMFS and web browsing.
But surely I will have a look at next releases of Squid.
ID: 47547 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47548 - Posted: 19 Nov 2022, 9:51:58 UTC - in response to Message 47547.  
Last modified: 19 Nov 2022, 10:40:39 UTC

Knowing not enough details for a squid, but
HPC (Highperformance_Computing) need some more experience and details for us to optimize this work for Cern.
Have TByte every month over ISP.
btw, my squid is running on my fastest PC.
CernVM is a idea to let CVMFS on the local network, without Squid.
WLCG (http://wlcg.web.cern.ch)
ID: 47548 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47549 - Posted: 19 Nov 2022, 13:29:18 UTC - in response to Message 47547.  

I did a tail -f to see access.log in real time.
...
While uploading the screen scrolls slower then normal.

You may try to prefix the tail command with "stdbuf -oL " to get a line buffered output



At the end of a ATLAS WU there are a lot of get requests to atlascern-frontier.openhtc.io.

They are also at the beginning of each ATLAS task.
The huge number of requests like those are among the reasons why a local proxy is recommended.
Unlike CVMFS Frontier does not have it's own local cache.



I hate software that takes full CPU doing about nothing.

Well, it processes the data transfer.
That's far away from doing nothing.
:-)

Without a Squid those CPU seconds would be included in other processes' accounting, e.g. BOINC, Vbox or ATLAS itself.

Large downloads are also affected.
Even the internet router's monitoring shows a significant increase in CPU load.
It's mostly not at 100% but I would guess this is caused by the sampling interval plus the built in averaging method.
ID: 47549 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47556 - Posted: 22 Nov 2022, 11:12:24 UTC - in response to Message 47548.  

squid-settings for squid, no frontier-squid:
https://twiki.cern.ch/twiki/bin/view/Frontier/MyOwnSquid
ID: 47556 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47559 - Posted: 24 Nov 2022, 6:23:05 UTC - in response to Message 47556.  

An important option for squid-2.6 and later is:
collapsed_forwarding on

This option combines requests agressively so that a file is retrieved only once from the origin server. This is a very good idea for computer farms so make sure it is on.

Have it installed in Squid 5.5 yesterday, no problems so long.
ID: 47559 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1692
Credit: 112,545,781
RAC: 310,298
Message 47574 - Posted: 5 Dec 2022, 11:01:00 UTC - in response to Message 47559.  

access_log none
This is meaningful, when you don't want so many requests (1GByte per day).
You can activate access.log, when you delete this line in squid.conf.
Be sure, atm you can not control this access.log, when there are problems in squid you want to watch!
ID: 47574 · Report as offensive     Reply Quote
Profile Guy
Avatar

Send message
Joined: 9 Feb 08
Posts: 10
Credit: 73,297
RAC: 198
Message 47719 - Posted: 22 Jan 2023, 15:32:21 UTC

I'm using Windows 10.
In BOINC "squid.conf" provided, there is a guide for setting the DNS server that squid uses -

# A MUST on Windows.
# If unsure try the the LAN IP of your internet router.
# Avoid using external DNS here.

dns_nameservers [list of IP addresses]


The documentation page for "Squid configuration directive dns_nameservers" http://www.squid-cache.org/Doc/config/dns_nameservers/ says:

On Windows platforms, if no value is specified here or in
the /etc/resolv.conf file, the list of DNS name servers are
taken from the Windows registry, both static and dynamic DHCP
configurations are supported.


I've tried my Default Gateway IP (LAN IP of the internet router) with no success.
I've tried leaving it out and blank, with no success.
Against the advice I tried an external DNS IP and it worked.

Is there another solution?
Thanks.
ID: 47719 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,096,635
RAC: 115,419
Message 47720 - Posted: 22 Jan 2023, 16:04:43 UTC - in response to Message 47719.  

On your host open a cmd window and type "nslookup".
This returns the IP of your DNS server.

A local DNS server should be preferred in squid.conf since using an external one will be slower and less efficient.
If you expect a DNS server (forwarder) to be present on your router but it doesn't respond, check whether it is accidentally disabled.
ID: 47720 · Report as offensive     Reply Quote
Profile Guy
Avatar

Send message
Joined: 9 Feb 08
Posts: 10
Credit: 73,297
RAC: 198
Message 47721 - Posted: 22 Jan 2023, 17:05:52 UTC - in response to Message 47720.  
Last modified: 22 Jan 2023, 17:14:10 UTC

Thank you!
nslookup returned an IPv6 address and it works.

PS.
Is there a recommended DNS server to install just for this local BOINC Squid?
Thanks.
ID: 47721 · Report as offensive     Reply Quote
Previous · 1 . . . 3 · 4 · 5 · 6

Message boards : Number crunching : Setting up a local Squid to work with LHC@home - Comments and Questions


©2023 CERN