Message boards :
Number crunching :
Setting up a local Squid to work with LHC@home - Comments and Questions
Message board moderation
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 8 · Next
Author | Message |
---|---|
![]() ![]() Send message Joined: 2 Sep 04 Posts: 455 Credit: 204,374,033 RAC: 137,484 ![]() ![]() ![]() |
TCP_Tunnel is because of https - connection [quote]squid TCP_Tunnel ![]() Supporting BOINC, a great concept ! |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
This is not the proxy from the own PC: [2020-12-19 05:41:25] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2020-12-19 05:41:25] 2.7.5.0 6547 2481 49536 75615 0 65 3410004 3670017 1606 65024 0 178595 97.0805 804596 1293 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch http://128.142.168.202:3126 1 [2020-12-19 05:41:25] CVMFS is ok [2020-12-19 05:41:25] Using singularity image /cvmfs/atlas.cern.ch/repo/containers/images/singularity/x86_64-centos7.img https://lhcathome.cern.ch/lhcathome/result.php?resultid=292390311 |
Send message Joined: 27 Sep 08 Posts: 854 Credit: 699,046,002 RAC: 180,454 ![]() ![]() ![]() |
This is a CERN internal server, I assume the image isn't in the cache so it goes to cern for data? |
Send message Joined: 27 Sep 08 Posts: 854 Credit: 699,046,002 RAC: 180,454 ![]() ![]() ![]() |
My misses are going down after 10 days: Downloads served by the proxy TCP_MEM_HIT 19769853 requests 87.6 GB TCP_HIT 2344592 requests 2.5 TB TCP_REFRESH_UNMODIFIED 354531 requests 1.6 GB Downloads requested from lhc@home TCP_MISS 84787 requests 16.9 GB TCP_REFRESH_MODIFIED 266538 requests 7.6 GB Result uploads to lhc@home TCP_MISS__UPLOAD 16102 requests 125.7 GB |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
It's a local CVMFS client used for Theory/ATLAS native. Due to a change at CERN it configures a CERN backup squid if no local proxy is used. I'm working on a suggestion but can't promise getting it ready before X-mas. |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
Squid for Windows show the local IP-Adress of the proxy in clearname in the finished Task!. |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
Squid for Windows show the local IP-Adress of the proxy in clearname in the finished Task!. It's not Squid, it's the CVMFS client that shows the IP of the proxy currently in use if you run "cvmfs_config stat". This is useful to see whether CVMFS is correctly configured. You complained lots of times via PM that a proxy IP like 192.168.a.b would violate data protection laws and would be a security risk. Neither is true! Regarding data protection: Data protection laws should ensure that a distinct person can't be identified or tracked without permission. This IP range is officially reserved for private use by everybody and indeed it is in use within an uncountable number of LANs around the world. Hence there's never a relationship to a distinct person. Regarding security: This IP range MUST NOT be forwarded outside your own LAN. Even if you would misconfigure your own internet router, your ISP would block all packets to/from that IP. Your ISP MUST block this to avoid crashing his own networks on a technical level. See: https://tools.ietf.org/html/rfc1918 |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
The name of my network is PeaceonEarth (and not since yesterday). The public free WLAN-IP's are using 192.168... In Linux-VM the Proxy-Adress is not shown! |
![]() Send message Joined: 29 Aug 05 Posts: 1066 Credit: 8,242,312 RAC: 9,543 ![]() |
Warning regarding a squid vulnerability (excerpted from an EGI SVG advisory): Affected software and risk ========================== HIGH risk vulnerability concerning Squid Package : Squid, including Frontier Squid [R 3] before version 4.15 The Squid project has publicly announced [R 1] new vulnerabilities, one of which is deemed HIGH risk, viz. CVE-2020-25097 [R 2], because it may allow services to be exposed that are not directly accessible from the client host. The other ones only concern potential denial of service and hence are deemed low risk. [R 1] http://lists.squid-cache.org/pipermail/squid-announce/2021-May/000127.html [R 2] https://github.com/squid-cache/squid/security/advisories/GHSA-jvf6-h9gj-pmj6 [R 3] https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid Fixed versions (squid-3.5.20-17.el7_9.6) are available for RHEL 7, CentOS 7, SL 7. Mitigation ========== For sites that cannot upgrade in a timely manner, temporary workarounds for the high-risk vulnerability are provided here. If frontier-squid is used, update customize.sh with the following line and either reload or restart frontier-squid: setoption("uri_whitespace", "deny") If a plain squid is used instead, set the "uri_whitespace" directive in squid.conf to either: uri_whitespace deny or uri_whitespace encode and restart the squid service. ![]() |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
Proxy-using for us Volunteers need a new strategy from Cern-IT in the future. For example WCG don't allowed a local proxy. Thank you Ivan, for this Info! |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
To fix the vulnerability use the workaround Ivan mentioned at the end of his post. There's just one line to be added to squid.conf plus a squid reload. BTW: That workaround is officially suggested by the Squid developers. For example WCG don't allowed a local proxy. They produce much less HTTP traffic. Hence, they simply don't need a proxy and that's why they don't enable their systems to use one. Proxy-using for us Volunteers need a new strategy from Cern-IT in the future. What kind of new strategy? What issues should it solve? |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
Have a Atlas native without using squid and this info in EventtoHits File: 09:56:06 warn [frontier.c:1114]: Trying next proxy db-atlas-squid.ndgf.org[153.5.68.11] with same server atlasfrontier-ai.cern.ch 09:56:06 warn [frontier.c:1014]: Request 3 on chan 24 failed at Wed Jun 9 09:56:06 2021: -7 [fn-htclient.c:445]: bad response (HTTP/1.1 403 Forbidden) proxy=db-atlas-squid.ndgf.org[153.5.68.11] server=atlasfrontier-ai.cern.ch09:56:06 warn [frontier.c:1114]: Trying next proxy db-atlas-squid.ndgf.org[153.5.68.11] with same server atlasfrontier-ai.cern.ch 09:56:06 warn [frontier.c:1014]: Request 3 on chan 24 failed at Wed Jun 9 09:56:06 2021: -7 [fn-htclient.c:445]: bad response (HTTP/1.1 403 Forbidden) proxy=db-atlas-squid.ndgf.org[153.5.68.11] server=atlasfrontier-ai.cern.ch |
Send message Joined: 2 May 07 Posts: 2257 Credit: 174,413,466 RAC: 23,149 ![]() ![]() ![]() |
For example WCG don't allowed a local proxy. NO ---- WCG is using HAPROXY!!!! https://HAPROXY.COM Proxy-using for us Volunteers need a new strategy from Cern-IT in the future. |
Send message Joined: 27 Sep 08 Posts: 854 Credit: 699,046,002 RAC: 180,454 ![]() ![]() ![]() |
can I just install the new version of Squid over old one? |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
You are thinking about v4.14 for Windows, right? I didn't yet test this version but there may be a few issues. 1. The installer may overwrite your squid.conf => backup your's before you upgrade 2. Like the 3.5 installer the new one may start Squid directly after the installation. This may cause issues with the creation of the disk cache directories. => Ensure no Squid instance is running when you create the disk cache directories or restore the squid.conf The necessary steps are already explained in the HowTo. The configuration parameters used in the HowTo's squid.conf work for both v3.5 and v4.14. As Ivan posted a while ago the following line should be added to squid.conf: uri_whitespace deny |
Send message Joined: 27 Sep 08 Posts: 854 Credit: 699,046,002 RAC: 180,454 ![]() ![]() ![]() |
Yes, since this is the stable version. The installer doesn't seem to let you install the new version over the old one, you have to uninstall and reinstall. It deletes the config file so, yes backup is needed :) Yes, it auto starts so you have to shutdown install the config then restart. |
Send message Joined: 27 Sep 08 Posts: 854 Credit: 699,046,002 RAC: 180,454 ![]() ![]() ![]() |
Seems to be problem free. |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
+1 Thanks for sharing your experience. |
Send message Joined: 8 Dec 19 Posts: 37 Credit: 7,587,438 RAC: 0 ![]() ![]() |
Would like a clarification on whether to install local Squid. I've read that it's recommended for 5+ worker nodes but not recommended to install on VMs. If I'm running native Theory & Atlas tasks with 5+ worker nodes on Hyper-V Ubuntu, is it recommended to install local Squid? Thanks. |
![]() Send message Joined: 15 Jun 08 Posts: 2571 Credit: 258,782,992 RAC: 119,060 ![]() ![]() |
Your computer list shows 5 computers with a total of 62 processors (that's BOINC terminology). "Worker node" is datacenter terminology. In this context processors, worker nodes, CPUs, threads, whatever are equivalent. The important thing is 62 which shows your maximum computing capability. Each 1-core task (Theory, CMS) counts a 1. Each n-core (ATLAS) task counts as n. Sum up the cores you expect to be used by all concurrently running tasks. It doesn't matter whether they run on bare metal or inside a VM as all of them generate lots of HTTP traffic. Most important is to connect Squid with a fast network. 1-Gbit LAN cable would be fine. If all your computers are interconnected with that fast network, 1 Squid instance would be enough. It's possible but not recommended to run that Squid on a VM: - Squid's performance is better on bare metal. - Squid would be unavailable if you shut down the host the Squid VM is running on. |
©2025 CERN