Message boards : Number crunching : Setting up a local Squid to work with LHC@home - Comments and Questions
Message board moderation
Previous · 1 . . . 5 · 6 · 7 · 8
| Author | Message |
|---|---|
|
Send message Joined: 21 Feb 11 Posts: 86 Credit: 586,107 RAC: 1 |
Append CVMFS_QUOTA_LIMIT=10000 to /etc/cvmfs/default.local to make local cache 10000 mb. You can even set it to 100000. |
zepingouinSend message Joined: 7 Jan 07 Posts: 41 Credit: 16,123,537 RAC: 22 |
Is there an easier way to do this? I'm surprised I can't find a simple windows program to cache internet access from other computers on the local network. I compiled Squid for use with Windows 10 and Cygwin, you may find binaries and instruction on this thread. Latest is Squid-6.2 and works fine. |
|
Send message Joined: 15 Jun 08 Posts: 2755 Credit: 304,271,457 RAC: 116,232 |
CVMFS_QUOTA_LIMIT is a config option for CVMFS. It has nothing to do with Squid. You can even set it to 100000 100 GB - far too much for the LHC@home use case. Of course you can, but this would just waste disk space. A quota between 4-10 GB results in a hitrate between 95-99 %. Let the CVMFS client run a couple of days (may be a week). Then check the hitrate of heavily used repos like cernvm-prod.cern.ch, sft.cern.ch (both Theory) or atlas.cern.ch (ATLAS - once it is back) Run "cvmfs_config stat [repo_name]" to get some stats including the hitrate. |
|
Send message Joined: 27 Sep 08 Posts: 946 Credit: 784,373,554 RAC: 158,973 |
ATLAS seem to have stopped uplaoding, is there something that I can check in the config files? |
|
Send message Joined: 12 Aug 06 Posts: 470 Credit: 15,295,644 RAC: 9,181 |
ATLAS seem to have stopped uplaoding, is there something that I can check in the config files?I uploaded one 10 minutes after your post. |
|
Send message Joined: 27 Sep 08 Posts: 946 Credit: 784,373,554 RAC: 158,973 |
I found it the config was needed to be updated https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5249 |
|
Send message Joined: 21 Feb 11 Posts: 86 Credit: 586,107 RAC: 1 |
Someone needs to create a step by step guide for installing and configuring cvmfs, boinc and apptainer, without building from source. |
|
Send message Joined: 12 Aug 06 Posts: 470 Credit: 15,295,644 RAC: 9,181 |
Someone needs to create a step by step guide for installing and configuring cvmfs, boinc and apptainer, without building from source.I keep saying this, but they don't like it. Apparently it's already simple? |
|
Send message Joined: 15 Jun 08 Posts: 2755 Credit: 304,271,457 RAC: 116,232 |
In most cases there's no need to compile any of them from the sources. Recent Linux distributions provide BOINC, VirtualBox, Runc, Apptainer, Squid. CERN provides precompiled CVMFS packets as described here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594&postid=48539 Most packets just need to be installed, except CVMFS and Squid which need to be configured as described here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594&postid=48539 https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5473 If this is too complex or the OS can't run native apps (Windows) install VirtualBox and let BOINC run vbox apps. |
|
Send message Joined: 21 Feb 11 Posts: 86 Credit: 586,107 RAC: 1 |
Then someone needs to unpin this thread https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4840 |
|
Send message Joined: 21 Feb 11 Posts: 86 Credit: 586,107 RAC: 1 |
You should also add instructions for disabling cgroups v2 so Theory tasks don't crash. |
|
Send message Joined: 30 Jan 22 Posts: 1 Credit: 190,659 RAC: 4,254 |
Where’s the best place to install the Squid application when running Windows 11 with the BOINC Docker app? Right now, it’s installed on my Windows system, but I’m getting a lot of TCP_MISS messages. 1775632134.196 63742 127.0.0.1 TCP_MISS/200 327 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.184.89.143 text/plain 1775632199.383 63929 127.0.0.1 TCP_MISS/200 298 POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler - HIER_DIRECT/188.185.81.88 text/plain 1775632271.721 71270 127.0.0.1 TCP_TUNNEL/200 14010 CONNECT lhcathome.cern.ch:443 - HIER_DIRECT/188.184.93.91 - |
|
Send message Joined: 12 Aug 06 Posts: 470 Credit: 15,295,644 RAC: 9,181 |
I spotted they miss on upload, that is to be expected. It can't remember the result you created, because it's new and unique. Gemini explains the rest better than me! Ideally, you should run Squid as a **Docker container** on the same network as your BOINC Docker app, rather than as a standalone Windows service. While having it on the Windows host works, running it in Docker allows for better networking performance via the **WSL 2 backend** and makes it easier to manage the internal routing between the BOINC client and the proxy. ### Why you are seeing TCP_MISS The TCP_MISS/200 entries in your log are actually expected for the specific traffic you're showing. 1. **POST Requests:** Your logs show POST requests to a file_upload_handler. By default, Squid (and most proxies) **does not cache POST requests** because they are intended to send data to the server, not retrieve it. 2. **HTTPS Tunneling:** The TCP_TUNNEL/200 for port 443 indicates an encrypted connection. Squid cannot "see" or cache the content of encrypted traffic unless you set up **SSL Bump** (SSL MITM), which is complex and usually unnecessary for BOINC. ### Recommended Setup: Squid in Docker To get the most out of your setup on Windows 11, follow these steps to move Squid into the Docker environment: #### 1. Create a Docker Network This allows your BOINC container and Squid container to communicate using internal container names. ```bash docker network create boinc-net ``` #### 2. Run Squid in a Container You can use a lightweight image like ubuntu/squid. Ensure you mount a volume for the cache so it persists when the container restarts. ```bash docker run -d --name squid-proxy \ --network boinc-net \ -v squid-cache:/var/spool/squid \ -p 3128:3128 \ ubuntu/squid ``` #### 3. Optimise squid.conf for BOINC If you want to cache the large **download** files (the actual work units/VM images) from CERN, you need to adjust the maximum object size in your squid.conf, as BOINC files are often larger than the default 4MB limit: ```bash # Allow larger files to be cached (e.g., 500MB) maximum_object_size 500 MB cache_dir ufs /var/spool/squid 5000 16 256 ``` ### Tips for LHC@home * **Ignore Upload "Misses":** Don't worry about TCP_MISS on POST requests to file_upload_handler. These are your finished results leaving your system; there is no benefit to caching them. * **Focus on Downloads:** The real benefit of Squid for BOINC is caching the .vdi or .iso files when you have multiple tasks or machines. You will see TCP_HIT when the client requests a file it has downloaded previously. * **Client Configuration:** In the BOINC Manager, ensure your proxy settings point to the Docker Desktop gateway (usually 172.17.0.1 or the specific container name if using a custom network). Since you are using a **Google Pixel 6 Pro**, you can actually monitor your BOINC status and these logs on the go using the **BOINC Monitor** app or similar third-party tools, provided you've opened the remote desktop ports in your Windows firewall. |
|
Send message Joined: 15 Jun 08 Posts: 2755 Credit: 304,271,457 RAC: 116,232 |
In reply to Raythean Buggy Cam's message of 8 Apr 2026: Where’s the best place to install the Squid application when running Windows 11 with the BOINC Docker app? Right now, it’s installed on my Windows system, but I’m getting a lot of TCP_MISS messages. TCP_MISS means that the data from the given URL is not in your local cache. In case of a POST (=upload) you will always get a TCP_MISS since it makes no sense to cache uploads. |
©2026 CERN