Message boards :
Number crunching :
Setting up a local Squid to work with LHC@home - Comments and Questions
Message board moderation
Previous · 1 . . . 4 · 5 · 6 · 7 · 8 · Next
Author | Message |
---|---|
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
It keeps most objects used by a series of tasks close to the computer(s) where they are used. Hence, it helps both, the server side as well as the client side. See: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5473 To get an impression how many requests are served by just 1 (out of hundreds) of Squids at CERN/WLCG within 1 day, see: http://wlcg-squid-monitor.cern.ch/awstats/bin/awstats.pl?config=atlasfrontier.cern.ch&databasebreak=day&day=01 And many of the clients getting objects from that Squid are also Squids sending requests to their parent only if an object is not (or not fresh) in their local cache. |
Send message Joined: 8 Nov 22 Posts: 10 Credit: 1,610,722 RAC: 0 |
Okay thanks but is it a cache for everyone or is it only local for my computers? |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
The configuration suggested in this forum is intended to be used only for your local network. Be aware that the example squid.conf does not allow any client to access the internet. You will have to add your computer(s)/network first to grant permission. |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
Hi, After installing squid with apt (ubuntu server 22.04), running 'squid -z' and starting squid, 'sudo systemctl status squid.service' gives: ... Active: active (running) since Wed 2023-05-24 17:24:54 UTC; 9s ago ... ERROR: listen(..., 1024) system call failed: (98) Address already in use listening port: 3128 I guess this is wrong. What should I do? Kind regards, Nejc my /etc/squid/squid.conf is: acl crunchers src 192.168.0.145 acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT follow_x_forwarded_for allow localhost follow_x_forwarded_for deny all acl wcg_nocache dstdomain .worldcommunitygrid.org cache deny wcg_nocache acl cvmfs_geoapi urlpath_regex -i ^/+cvmfs/+[0-9a-z._~-]+/+api/+[0-9a-z._~-]+/+geo/+[0-9a-z._~-]+/+[0-9a-z.,_~-]+ cache deny cvmfs_geoapi acl boinc_nocache urlpath_regex -i /download[0-9a-z._~-]*/+[0-9a-z._~-]+/+.+ cache deny boinc_nocache acl PragmaNoCache req_header Pragma no-cache cache deny PragmaNoCache acl Purge method PURGE http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access deny to_localhost http_access allow crunchers http_access allow localhost http_access deny all http_port 3128 dns_nameservers 192.168.0.1 max_filedescriptors 4096 client_dst_passthru off cache_mem 256 MB maximum_object_size_in_memory 24 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA maximum_object_size 6144 MB cache_dir aufs /var/cache/squid 20000 16 64 min-size=7937 logfile_rotate 10 logformat my_awstats %>A %lp %ui %un [%tl] "%rm %>ru HTTP/%rv" %>Hs %st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh access_log stdio:/var/log/squid/access.log logformat=my_awstats strip_query_terms off coredump_dir none ftp_user anonymous@ max_stale 37 days refresh_pattern . 0 0% 0 store_avg_object_size 1800 KB shutdown_lifetime 0 seconds collapsed_forwarding on client_persistent_connections on server_persistent_connections on log_icp_queries off dns_defnames on forwarded_for transparent |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
I forgot to mention that squid -k reconfigure gave no errors. |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
According to this lines your IPs are taken from the private ipv4 range: acl crunchers src 192.168.0.145 dns_nameservers 192.168.0.1 To check whether the networks in squid.conf match your local settings post the output of this command: ip -f inet addr show Simplify your squid.conf. If you use 192.168.0.x there's no need to enable other networks in squid.conf. Disable them (red #): acl crunchers src 192.168.0.145 #acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) #acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) #acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) #acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines #acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) #acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) #acl localnet src fc00::/7 # RFC 4193 local private network range #acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines Check for errors here After each modification of squid.conf run: squid -k parse After a Squid reload/restart check /var/log/squid/cache.log As for this error post the output of: ps -AF |grep " $(lsof -ti:3128) " ERROR: listen(..., 1024) system call failed: (98) Address already in use |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
I commented the mentioned lines. I didn't mention it before - I installed squid on the same machine that is crunching - is this a problem? Would it be better if I make a VM on one of my other servers and use that one for squid? Since I installed and configured squid none of my project can connect (Can't create HTTP response output file notices/ wcg, rosetta, sidock, gpugrid) and the other two projects (lhc and einstein) are not even visible in the boinc-client/boinctui. Before that everything was running normal - cvmfs was installed per guide that makes it possible to run cms and other tasks natively. So, after commenting the lines you mentioned: squid -k parse: 2023/05/25 12:18:40| Startup: Initializing Authentication Schemes ... 2023/05/25 12:18:40| Startup: Initialized Authentication Scheme 'basic' 2023/05/25 12:18:40| Startup: Initialized Authentication Scheme 'digest' 2023/05/25 12:18:40| Startup: Initialized Authentication Scheme 'negotiate' 2023/05/25 12:18:40| Startup: Initialized Authentication Scheme 'ntlm' 2023/05/25 12:18:40| Startup: Initialized Authentication. 2023/05/25 12:18:40| Processing Configuration File: /etc/squid/squid.conf (depth 0) 2023/05/25 12:18:40| Processing: acl crunchers src 192.168.0.145 2023/05/25 12:18:40| Processing: acl SSL_ports port 443 2023/05/25 12:18:40| Processing: acl Safe_ports port 80 # http 2023/05/25 12:18:40| Processing: acl Safe_ports port 21 # ftp 2023/05/25 12:18:40| Processing: acl Safe_ports port 443 # https 2023/05/25 12:18:40| Processing: acl Safe_ports port 70 # gopher 2023/05/25 12:18:40| Processing: acl Safe_ports port 210 # wais 2023/05/25 12:18:40| Processing: acl Safe_ports port 1025-65535 # unregistered ports 2023/05/25 12:18:40| Processing: acl Safe_ports port 280 # http-mgmt 2023/05/25 12:18:40| Processing: acl Safe_ports port 488 # gss-http 2023/05/25 12:18:40| Processing: acl Safe_ports port 591 # filemaker 2023/05/25 12:18:40| Processing: acl Safe_ports port 777 # multiling http 2023/05/25 12:18:40| Processing: acl CONNECT method CONNECT 2023/05/25 12:18:40| Processing: follow_x_forwarded_for allow localhost 2023/05/25 12:18:40| Processing: follow_x_forwarded_for deny all 2023/05/25 12:18:40| Processing: acl wcg_nocache dstdomain .worldcommunitygrid.org 2023/05/25 12:18:40| Processing: cache deny wcg_nocache 2023/05/25 12:18:40| Processing: acl cvmfs_geoapi urlpath_regex -i ^/+cvmfs/+[0-9a-z._~-]+/+api/+[0-9a-z._~-]+/+geo/+[0-9a-z._~-]+/+[0-9a-z.,_~-]+ 2023/05/25 12:18:40| Processing: cache deny cvmfs_geoapi 2023/05/25 12:18:40| Processing: acl boinc_nocache urlpath_regex -i /download[0-9a-z._~-]*/+[0-9a-z._~-]+/+.+ 2023/05/25 12:18:40| Processing: cache deny boinc_nocache 2023/05/25 12:18:40| Processing: acl PragmaNoCache req_header Pragma no-cache 2023/05/25 12:18:40| Processing: cache deny PragmaNoCache 2023/05/25 12:18:40| Processing: acl Purge method PURGE 2023/05/25 12:18:40| Processing: http_access deny !Safe_ports 2023/05/25 12:18:40| Processing: http_access deny CONNECT !SSL_ports 2023/05/25 12:18:40| Processing: http_access allow localhost manager 2023/05/25 12:18:40| Processing: http_access deny manager 2023/05/25 12:18:40| Processing: http_access deny to_localhost 2023/05/25 12:18:40| Processing: include /etc/squid/conf.d/*.conf 2023/05/25 12:18:40| Processing Configuration File: /etc/squid/conf.d/debian.conf (depth 1) 2023/05/25 12:18:40| Processing: logfile_rotate 0 2023/05/25 12:18:40| Processing: http_access allow crunchers 2023/05/25 12:18:40| Processing: http_access allow localhost 2023/05/25 12:18:40| Processing: http_access deny all 2023/05/25 12:18:40| Processing: http_port localhost:3128 2023/05/25 12:18:40| Processing: http_port 3128 2023/05/25 12:18:40| Processing: dns_nameservers 192.168.0.1 2023/05/25 12:18:40| Processing: max_filedescriptors 4096 2023/05/25 12:18:40| Processing: client_dst_passthru off 2023/05/25 12:18:40| Processing: cache_mem 256 MB 2023/05/25 12:18:40| Processing: maximum_object_size_in_memory 24 KB 2023/05/25 12:18:40| Processing: memory_replacement_policy heap GDSF 2023/05/25 12:18:40| Processing: cache_replacement_policy heap LFUDA 2023/05/25 12:18:40| Processing: maximum_object_size 6144 MB 2023/05/25 12:18:40| Processing: cache_dir aufs /var/cache/squid 20000 16 64 min-size=7937 2023/05/25 12:18:40| Processing: logfile_rotate 10 2023/05/25 12:18:40| Processing: logformat my_awstats %>A %lp %ui %un [%tl] "%rm %>ru HTTP/%rv" %>Hs %st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh 2023/05/25 12:18:40| Processing: access_log stdio:/var/log/squid/access.log logformat=my_awstats 2023/05/25 12:18:40| Processing: strip_query_terms off 2023/05/25 12:18:40| Processing: coredump_dir none 2023/05/25 12:18:40| Processing: ftp_user anonymous@ 2023/05/25 12:18:40| Processing: max_stale 37 days 2023/05/25 12:18:40| Processing: refresh_pattern . 0 0% 0 2023/05/25 12:18:40| Processing: store_avg_object_size 1800 KB 2023/05/25 12:18:40| Processing: shutdown_lifetime 0 seconds 2023/05/25 12:18:40| Processing: collapsed_forwarding on 2023/05/25 12:18:40| Processing: client_persistent_connections on 2023/05/25 12:18:40| Processing: server_persistent_connections on 2023/05/25 12:18:40| Processing: log_icp_queries off 2023/05/25 12:18:40| Processing: dns_defnames on 2023/05/25 12:18:40| Processing: forwarded_for transparent 2023/05/25 12:18:40| Initializing https:// proxy context ip -f inet addr show: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 192.168.0.145/24 metric 100 brd 192.168.0.255 scope global dynamic bond0 valid_lft 496sec preferred_lft 496sec ps -AF |grep " $(lsof -ti:3128) " - is REALLy long - should I post the whole of it here? Thank you for helping. |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
Since you run a couple of computers, is it this one? https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10830136 or this one? https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10821458 I installed squid on the same machine that is crunching - is this a problem? No Would it be better if I make a VM on one of my other servers and use that one for squid? No. You may run Squid on a VM for testing (until you are familiar with the process) but at the end it should run on a real box for better performance. Since I installed and configured squid none of my project can connect (Can't create HTTP response output file notices/ wcg, rosetta, sidock, gpugrid) and the other two projects (lhc and einstein) are not even visible in the boinc-client/boinctui. This doesn't make sense. If you just install and start Squid it binds to port 3128. Up to this point Squid does not modify anything in connection with BOINC. The HowTo clearly states to test Squid first, e.g. using a browser. If this succeeds BOINC can be told to run that Squid but even this setting does not modify BOINC's project list or the way it generates requests. BOINC just sends them to Squid instead of the original destination. cvmfs was installed per guide that makes it possible to run cms... CMS doesn't have a native app that would use a local CVMFS. 5: bond0 ... Looks like you have more than the 2 network interfaces that those you posted. May be one of them is not configured correctly. To ensure Squid binds only to 127.0.0.1 and 192.168.0.145 replace http_port localhost:3128 http_port 3128 with http_port localhost:3128 http_port 192.168.0.145:3128 tcp_outgoing_address 192.168.0.145 ps -AF |grep " $(lsof -ti:3128) " - is REALLy long - should I post the whole of it here? No. A long list points out "(lsof ...)" was empty. This happens if Squid is not running. Instead, start your Squid service and check /var/log/squid/cache.log for recent errors. |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
So on the Can't create HTTP response output file notices... as expected - mea culpa. I ran boinc as service not daemon (got mixed up on which machine I am working :) ). Will start with squid from the begining slowly and taking into account all that you wrote. Hopefully I willl post back with success. Great thanks computezrmle |
Send message Joined: 2 May 07 Posts: 2242 Credit: 173,902,375 RAC: 2,454 |
Have running Squid in a CentOS9-VM. You don't need a separate Hardware for use it. |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
After correcting squid.conf as mentioned systemctl status gives no errors. Does it mean that squid is running correctly? How can I check (no GUI)? If i now do: boinccmd --set_proxy_settings squid_hostname_or_IP 3128 '' '' '' '' '' '' '' (as metnioned at top of the thread) will all my projects be connecting through proxy correctly or do I need to add some more stuff to squid.conf? |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
After correcting squid.conf as mentioned systemctl status gives no errors. If "systemctl status ..." doesn't print an error this does not mean your clients are permitted to use Squid. Either test this with a browser on a computer you want to connect via Squid or configure BOINC to use Squid. Worst case would be that the computer is not (yet) permitted and BOINC fails to connect to the internet. Then, add a line like this to squid.conf for each IPv4 you want to allow to use Squid. Example: acl crunchers src 192.168.0.146 acl crunchers src 192.168.0.147 acl crunchers src 192.168.0.148 Then run (as root) "squid -k reconfigure" If a browser on a computer in your LAN can contact internet sites, BOINC on the same computer will also be permitted. In addition you will get log entries from that computer in /var/log/squid/access.log To follow that file you may try this command: tail -F /var/log/squid/access.log Be aware that your access.log may grow very quickly since you have a large number of worker nodes. Hence, the logfile should be rotated regularly (suggestion: once a day) either by a cronjob or via systemd running "squid -k rotate". Make yourself familiar with those basic methods since they are often required - not only by Squid. WCG is (better: was) known to be problematic in connection with the local cache, hence this is switched off in squid.conf and only Squid's proxy function is used. Other projects should work fine. If Squid related problems happen this needs to be investigated. |
Send message Joined: 17 Dec 16 Posts: 6 Credit: 32,135,519 RAC: 133,868 |
Thank you for all the help. I believe it is working now. So how can I actually see if the traffic of boinc or IP address in squid? And how can I check what squid is doing "for my clients;from it's RAM-Cache" as Yeti mentioned? |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
So how can I actually see if the traffic of boinc or IP address in squid? As already mentioned, run this (as root) on the squid box: tail -F /var/log/squid/access.log Or try this commands (also on the squid box): squidclient mgr:info squidclient mgr:refresh squidclient mgr:client_list |
Send message Joined: 12 Aug 06 Posts: 429 Credit: 10,580,644 RAC: 2,282 |
Is there an easier way to do this? I'm surprised I can't find a simple windows program to cache internet access from other computers on the local network. I've tried squid twice. The first time was successfull for a time. I followed the instructions in here and it worked for a couple of months, then decided not to. I tried repairing it which failed, so I tried uninstalling it, and it refused. I deleted the squid folder and tried to reinstall and it still didn't work. So I tried a second time today on a different computer which has never seen squid, I got stuck at the first instruction: "Open "Squid Terminal" as Administrator and run: squid -k shutdown" I did this and it said "0 [main] squid 228 cygwin_exception::open_stackdumpfile: Dumping stack trace to squid.exe.stackdump" Without following any further instructions, squid is not functioning out of the box for a browser on another computer directed to it as I would expect. Is there an alternative cache which isn't a badly ported linux program? |
Send message Joined: 12 Aug 06 Posts: 429 Credit: 10,580,644 RAC: 2,282 |
I'm not entirely sure why we have to do this anyway, at least within one computer. On Windows anyway, LHC runs every single task in it's own virtualbox. And those VBs don't seem to have access outside themselves, to the local disk in Windows. If they did, anything they download could be kept for another task running on the same computer. As it is, one computer is downloading the same files multiple times. Is this some kind of Boinc limitation? I just tried putting LHC onto a Linux machine without VB installed, and it's not working, I assumed LHC didn't use VB if in Linux already? Am I wrong? They're coming up as "native theory". |
Send message Joined: 2 May 07 Posts: 2242 Credit: 173,902,375 RAC: 2,454 |
Have only one PC (64-Core) running with a CentOS9-VM for Squid. The Transfer of Data is 2 TByte inside the PC for Theory-Tasks in a few days. |
Send message Joined: 12 Aug 06 Posts: 429 Credit: 10,580,644 RAC: 2,282 |
Have only one PC (64-Core) running with a CentOS9-VM for Squid.There's something terribly wrong with the design of the Boinc/LHC programs if it's downloading the same file more than once on the same PC. It should be stored in the Boinc folders for later use. |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 34,439 |
The design is fine. It's you who refuse to understand it. |
Send message Joined: 12 Aug 06 Posts: 429 Credit: 10,580,644 RAC: 2,282 |
The design is fine.How is it fine to download huge data file x 24 times for 24 tasks running on the same PC? |
©2024 CERN