How do I set up a local HTTP proxy?

Author	Message
Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 47714 - Posted: 20 Jan 2023, 21:45:10 UTC Hello, How do I set up a local HTTP proxy, please? The VirtualBox shows this - --:--:--.------ VMMDev: Guest Log: [INFO] Could not find a local HTTP proxy --:--:--.------ VMMDev: Guest Log: [INFO] CVMFS and Frontier will have to use DIRECT connections --:--:--.------ VMMDev: Guest Log: [INFO] This makes the application less efficient --:--:--.------ VMMDev: Guest Log: [INFO] It also puts higher load on the project servers --:--:--.------ VMMDev: Guest Log: [INFO] Setting up a local HTTP proxy is highly recommended --:--:--.------ VMMDev: Guest Log: [INFO] Advice can be found in the project forum Thanks. ID: 47714 · Reply Quote

Harri Liljeroos Send message Joined: 28 Sep 04 Posts: 736 Credit: 49,865,667 RAC: 35,184	Message 47715 - Posted: 20 Jan 2023, 22:53:10 UTC - in response to Message 47714. Last modified: 20 Jan 2023, 22:54:49 UTC https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5473#42987 https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5474#42988 ID: 47715 · Reply Quote

computezrmle Volunteer moderator Volunteer developer Volunteer tester Help desk expert Send message Joined: 15 Jun 08 Posts: 2549 Credit: 255,373,399 RAC: 62,870	Message 47716 - Posted: 21 Jan 2023, 6:42:03 UTC - in response to Message 47714. Don't worry about an HTTP proxy ATM. Since you run only 1 computer with 8 cores and only 16 GB RAM it's very likely that you will not permanently run lots of those heavy tasks concurrently that make a proxy a recommendation. You may instead focus on running few but stable VM tasks from LHC@home. Start with not more than 1 or 2 running concurrently and only increase that number step by step if that setup was stable for a couple of days. ID: 47716 · Reply Quote

maeax Send message Joined: 2 May 07 Posts: 2245 Credit: 174,006,243 RAC: 8,727	Message 47717 - Posted: 21 Jan 2023, 9:26:26 UTC - in response to Message 47714. How do I set up a local HTTP proxy, please? If you want to make some experiences, Harri have send some links with Instructions to learn how to install a Squid. ID: 47717 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51247 - Posted: 5 Dec 2024, 17:19:51 UTC Yes, Thank you. I have set up my LHC@home to run just 1 CMS task at a time with this app_config.xml 1 multi-threaded CMS task at a time is all my 8 core CPU can manage comfortably - a minimum of 4 cores for each individual task is a project requirement. So theoretically I could run 2 CMS tasks at a time. I have actually. But using all 8 cores of my CPU for crunching leaves no room for background OS tasks to run and overall the system slows down. Anyhoo. I was using squid successfully on Windows 10 a few months ago before I migrated my BOINC projects to Linux. Because of that I feel that squid helps. In fact it's really cool - if you can set it up properly. I read the very helpful HowTo set up a local Squid and had little more to do than enter my ip address where it said. Now it's not working on my linux box (Tumbleweed 15.6). It caches what it's supposed to and passes through everything else. But nothing uploads. Not a major problem because you can just stop your BOINC client using the squid http proxy in the "Options -> Other options" -> HTTP Proxy tab and it resumes communicating - as if it wasn't there in the first place with no problems at all. I'd like to get it to work though. There are a lot of changes posted in the new comments thread. I've tried to implement them. I've even had a look at firewalls. I'm not sure what the right thing to do is. And I'm uncertain which LHC@home jobs use the proxy now. Any insights, help would be welcome. Thanks ID: 51247 · Reply Quote

computezrmle Volunteer moderator Volunteer developer Volunteer tester Help desk expert Send message Joined: 15 Jun 08 Posts: 2549 Credit: 255,373,399 RAC: 62,870	Message 51248 - Posted: 5 Dec 2024, 18:43:53 UTC - in response to Message 51247. Now it's not working on my linux box (Tumbleweed 15.6). It caches what it's supposed to and passes through everything else. But nothing uploads. Not sure what you mean. Are there issues with LHC@home or issues affecting other projects? If the latter, which projects and what do the logs report? There shouldn't be any upload issues once you can connect to a project. Squid handles uploads transparently. As for "Tumbleweed 15.6": This doesn't exist. You either run Tumbleweed or Leap 15.6. And I'm uncertain which LHC@home jobs use the proxy now. All except SixTrack. ID: 51248 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51250 - Posted: 5 Dec 2024, 22:35:50 UTC - in response to Message 51248. Last modified: 5 Dec 2024, 22:58:31 UTC I initially installed Leap years ago. Now occasionally it thinks it's Tumbleweed for unknown reasons... It depends on how it's updating or what it's connected to - sometimes it identifies as one thing and other-times the other. I've got my repos crossed somewhere. As long as it works I don't pay too much attention to it. Sometimes I'm suspicious but it only pays to be as suspicious as you are apt. I'm really only a hobbyist user. And I'm uncertain which LHC@home jobs use the proxy now. All except SixTrack. Ah well my /etc/squid/squid.conf is definitely at fault for that. I've made a mistake thinking that Atlas doesn't use squid any more and commented out "extra section 2". I must have missed something in that 3-page thread. The trouble is that I first used the squid.conf as posted with just my ip inserted where needed. It didn't work. So I dug around the new messages thread trying to fix it! Basically - Help! I'm lost. Here are some current logs - OK I have Einstein@home and LHC@home. There is no LHC work on my PC at the moment. I reinstated the squid at 20:21 GMT to get some logging. This is my BOINC client Event log during communication: ... Thu 05 Dec 2024 20:21:54 GMT \| \| Using proxy info from GUI Thu 05 Dec 2024 20:21:54 GMT \| \| Using HTTP proxy 192.168.etc.etc:3128 Thu 05 Dec 2024 20:32:41 GMT \| Einstein@Home \| Computation for task LATeah2111F_1304.0_94864_0.0_0 finished Thu 05 Dec 2024 20:32:41 GMT \| Einstein@Home \| Starting task LATeah2111F_1304.0_1007072_0.0_0 Thu 05 Dec 2024 20:32:41 GMT \| Einstein@Home \| Sending scheduler request: To fetch work. Thu 05 Dec 2024 20:32:41 GMT \| Einstein@Home \| Requesting new tasks for CPU Thu 05 Dec 2024 20:32:43 GMT \| Einstein@Home \| Started upload of LATeah2111F_1304.0_94864_0.0_0_0 Thu 05 Dec 2024 20:32:43 GMT \| Einstein@Home \| Started upload of LATeah2111F_1304.0_94864_0.0_0_1 Thu 05 Dec 2024 20:32:43 GMT \| Einstein@Home \| Scheduler request completed: got 1 new tasks Thu 05 Dec 2024 20:32:43 GMT \| Einstein@Home \| Project requested delay of 60 seconds Thu 05 Dec 2024 20:32:45 GMT \| Einstein@Home \| Finished upload of LATeah2111F_1304.0_94864_0.0_0_0 (342 bytes) Thu 05 Dec 2024 20:32:45 GMT \| Einstein@Home \| Finished upload of LATeah2111F_1304.0_94864_0.0_0_1 (338 bytes) ... It's behaving itself - It uploaded! /var/log/squid/* shows: 192.168.etc.etc 3128 - - [05/Dec/2024:20:27:06 +0000] "CONNECT einsteinathome.org:443 HTTP/1.1" 200 8590 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:20:27:09 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 48936 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:20:32:44 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1458 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:20:32:44 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1462 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:20:32:47 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 33682 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT I'm aware that TCP_HIT/MISS are whether or not the requested data was found in the squid cache. And that HIER_DIRECT means the data was pulled from the target url on the remote server. And 200 means continue (or OK). I guess that TCP_TUNNEL:HIER_DIRECT means that it goes straight through or... Is it a firewall message? My knowledge of TCP/IP is very limited. And therefore firewalls - nope. I've had a look and don't know where to start. I'm too scared to open ports or play with network security in any way - without an official trustworthy okey dokey. The squid ran for about 24 hours before I switched it off. The access log is 4.7 MB plus. The only LHC@home units at the moment are empty CMS jobs. I don't want to download those because they'll use four cores pointlessly for 20 minutes while my other project is putting them to use. But I made a note of some of the errant activity from last week, here - The BOINC Manager's Event Log: ... Thu 28 Nov 2024 22:00:12 GMT \| Einstein@Home \| Computation for task LATeah2111F_504.0_63492_0.0_0 finished Thu 28 Nov 2024 22:00:14 GMT \| Einstein@Home \| Started upload of LATeah2111F_504.0_63492_0.0_0_0 Thu 28 Nov 2024 22:00:14 GMT \| Einstein@Home \| Started upload of LATeah2111F_504.0_63492_0.0_0_1 Thu 28 Nov 2024 22:00:15 GMT \| Einstein@Home \| Finished upload of LATeah2111F_504.0_63492_0.0_0_0 (355 bytes) Thu 28 Nov 2024 22:00:15 GMT \| Einstein@Home \| Finished upload of LATeah2111F_504.0_63492_0.0_0_1 (346 bytes) ... Thu 28 Nov 2024 22:01:50 GMT \| Einstein@Home \| Computation for task h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0 finished Thu 28 Nov 2024 22:01:51 GMT \| Einstein@Home \| Starting task h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51814_0 Thu 28 Nov 2024 22:01:53 GMT \| Einstein@Home \| Started upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_0 Thu 28 Nov 2024 22:01:53 GMT \| Einstein@Home \| Started upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_1 Thu 28 Nov 2024 22:01:55 GMT \| \| Project communication failed: attempting access to reference site Thu 28 Nov 2024 22:01:55 GMT \| Einstein@Home \| Temporarily failed upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_0: transient HTTP error Thu 28 Nov 2024 22:01:55 GMT \| Einstein@Home \| Backing off 00:02:21 on upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_0 Thu 28 Nov 2024 22:01:55 GMT \| Einstein@Home \| Temporarily failed upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_1: transient HTTP error Thu 28 Nov 2024 22:01:55 GMT \| Einstein@Home \| Backing off 00:03:44 on upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_1 Thu 28 Nov 2024 22:01:55 GMT \| Einstein@Home \| Started upload of h1_1679.60_O3aC01Cl1In0__O3ASHF1d_1680.00Hz_51816_0_2 Thu 28 Nov 2024 22:01:56 GMT \| \| Internet access OK - project servers may be temporarily down. ... Thu 28 Nov 2024 22:04:27 GMT \| Einstein@Home \| Started download of Ter5_1_dns_cfbf00084_segment_5_dms_200_168.binary Thu 28 Nov 2024 22:04:30 GMT \| Einstein@Home \| Finished download of Ter5_1_dns_cfbf00084_segment_5_dms_200_168.binary (9402308 bytes) ... ... Fri 29 Nov 2024 01:12:24 GMT \| LHC@home \| Started upload of fpyKDm1Cza6nsSi4ap6QjLDmwznN0nGgGQJmq4hLDmaXSLDmbNiYgn_0_r2141493864_ATLAS_result Fri 29 Nov 2024 01:12:28 GMT \| \| Project communication failed: attempting access to reference site Fri 29 Nov 2024 01:12:28 GMT \| LHC@home \| Temporarily failed upload of fpyKDm1Cza6nsSi4ap6QjLDmwznN0nGgGQJmq4hLDmaXSLDmbNiYgn_0_r2141493864_ATLAS_result: transient HTTP error Fri 29 Nov 2024 01:12:28 GMT \| LHC@home \| Backing off 00:55:36 on upload of fpyKDm1Cza6nsSi4ap6QjLDmwznN0nGgGQJmq4hLDmaXSLDmbNiYgn_0_r2141493864_ATLAS_result Fri 29 Nov 2024 01:12:30 GMT \| \| Internet access OK - project servers may be temporarily down. ... And the simultaneous /var/log/squid/access.log: ... 192.168.etc.etc 3128 - - [28/Nov/2024:22:00:14 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1474 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:00:14 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1465 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT ... 192.168.etc.etc 3128 - - [28/Nov/2024:22:01:53 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 1006 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:01:53 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 1006 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:01:54 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 100 189250 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:01:54 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 100 189250 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:01:55 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 3750 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT ... 192.168.etc.etc 3128 - - [28/Nov/2024:22:04:30 +0000] "GET http://einstein8.aei.uni-hannover.de/EinsteinAtHome/download/2b4/Ter5_1_dns_cfbf00084_segment_5_dms_200_168.binary HTTP/1.1" 200 9402992 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [28/Nov/2024:22:04:30 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 102962 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT ... ... 192.168.etc.etc 3128 - - [29/Nov/2024:01:12:27 +0000] "POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler HTTP/1.1" 0 196933 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT 192.168.etc.etc 3128 - - [29/Nov/2024:01:12:27 +0000] "POST http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler HTTP/1.1" 0 196933 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT ... I think a squid is a great idea. I just can't understand the security of TCP/IP. Any help with this would be so welcome. Thanks. Append: the next post has squid logs showing it working and starting to fail. ID: 51250 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51251 - Posted: 5 Dec 2024, 22:37:29 UTC Here it is in the process of failing at 21:47 with Einstein@home jobs only - BOINC Manager Event log: Thu 05 Dec 2024 21:45:06 GMT \| Einstein@Home \| Computation for task LATeah2111F_1304.0_973852_0.0_0 finished Thu 05 Dec 2024 21:45:06 GMT \| Einstein@Home \| Starting task LATeah2111F_1336.0_453046_0.0_0 Thu 05 Dec 2024 21:45:08 GMT \| Einstein@Home \| Started upload of LATeah2111F_1304.0_973852_0.0_0_0 Thu 05 Dec 2024 21:45:08 GMT \| Einstein@Home \| Started upload of LATeah2111F_1304.0_973852_0.0_0_1 Thu 05 Dec 2024 21:45:10 GMT \| Einstein@Home \| Finished upload of LATeah2111F_1304.0_973852_0.0_0_0 (356 bytes) Thu 05 Dec 2024 21:45:10 GMT \| Einstein@Home \| Finished upload of LATeah2111F_1304.0_973852_0.0_0_1 (339 bytes) Thu 05 Dec 2024 21:47:23 GMT \| Einstein@Home \| Started upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40584_1_0 Thu 05 Dec 2024 21:47:27 GMT \| \| Project communication failed: attempting access to reference site Thu 05 Dec 2024 21:47:27 GMT \| Einstein@Home \| Temporarily failed upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40584_1_0: transient HTTP error Thu 05 Dec 2024 21:47:27 GMT \| Einstein@Home \| Backing off 00:44:47 on upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40584_1_0 Thu 05 Dec 2024 21:47:29 GMT \| \| Internet access OK - project servers may be temporarily down. ... Thu 05 Dec 2024 21:55:21 GMT \| Einstein@Home \| Started upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40546_0_0 Thu 05 Dec 2024 21:55:25 GMT \| \| Project communication failed: attempting access to reference site Thu 05 Dec 2024 21:55:25 GMT \| Einstein@Home \| Temporarily failed upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40546_0_0: transient HTTP error Thu 05 Dec 2024 21:55:25 GMT \| Einstein@Home \| Backing off 00:05:46 on upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_40546_0_0 Thu 05 Dec 2024 21:55:27 GMT \| \| Internet access OK - project servers may be temporarily down. /var/log/squid/access.log: 192.168.etc.etc 3128 - - [05/Dec/2024:21:45:10 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1477 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:21:45:10 +0000] "POST http://einstein4.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_medium HTTP/1.1" 200 1460 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:21:47:24 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 1004 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:21:47:26 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 100 266609 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT ... 192.168.etc.etc 3128 - - [05/Dec/2024:21:55:22 +0000] "CONNECT www.google.com:443 HTTP/1.1" 200 15708 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:21:55:22 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 1004 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:21:55:24 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 100 189248 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS_ABORTED:HIER_DIRECT ID: 51251 · Reply Quote

computezrmle Volunteer moderator Volunteer developer Volunteer tester Help desk expert Send message Joined: 15 Jun 08 Posts: 2549 Credit: 255,373,399 RAC: 62,870	Message 51253 - Posted: 6 Dec 2024, 7:01:45 UTC - in response to Message 51250. Did you add "client_request_buffer_max_size" to squid.conf as suggested here? Setting it to "100 MB" instead of "512 MB" should be enough. This makes larger uploads work: myhost.example.com 4128 - - [06/Dec/2024:06:39:26 +0100] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 4893496 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.0)" TCP_MISS:HIER_DIRECT 192.168.etc.etc 3128 - - [05/Dec/2024:20:32:47 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 33682 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT I guess that TCP_TUNNEL:HIER_DIRECT means that it goes straight through or This indicates an encrypted connection (via HTTPS port 443). Those connections are forwarded by Squid without inspecting the packets - like any router between client and server does. ID: 51253 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51254 - Posted: 6 Dec 2024, 16:26:36 UTC - in response to Message 51253. Last modified: 6 Dec 2024, 16:40:55 UTC Did you add "client_request_buffer_max_size" to squid.conf as suggested No. But I have now, for sure. client_request_buffer_max_size 100 MB It does work. The connection is now handling file_upload_handler_large where previously not. I have no work from LHC@home to test with. This was an Einstein@home GPU task (O3 1.15 (GW-cuda)) - BOINC Manager - Event log ... Fri 06 Dec 2024 13:00:16 GMT \| \| Using proxy info from GUI Fri 06 Dec 2024 13:00:16 GMT \| \| Using HTTP proxy 192.168.##.##:3128 ... Fri 06 Dec 2024 13:27:18 GMT \| Einstein@Home \| Computation for task h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32148_1 finished Fri 06 Dec 2024 13:27:18 GMT \| Einstein@Home \| Starting task h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32013_1 Fri 06 Dec 2024 13:27:20 GMT \| Einstein@Home \| Started upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32148_1_0 Fri 06 Dec 2024 13:27:20 GMT \| Einstein@Home \| Started upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32148_1_1 Fri 06 Dec 2024 13:27:21 GMT \| Einstein@Home \| Finished upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32148_1_1 (2557 bytes) Fri 06 Dec 2024 13:27:25 GMT \| Einstein@Home \| Finished upload of h1_0161.80_O3aLC01Cl1In0__O3ASBu_162.00Hz_32148_1_0 (4889766 bytes) Fri 06 Dec 2024 13:27:25 GMT \| Einstein@Home \| Sending scheduler request: To report completed tasks. Fri 06 Dec 2024 13:27:25 GMT \| Einstein@Home \| Reporting 1 completed tasks Fri 06 Dec 2024 13:27:25 GMT \| Einstein@Home \| Not requesting tasks: "no new tasks" requested via Manager Fri 06 Dec 2024 13:27:26 GMT \| Einstein@Home \| Scheduler request completed (Einstein@home downloads are working too) /var/log/squid/access.log ... 192.168.##.## 3128 - - [06/Dec/2024:13:27:21 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 1004 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.##.## 3128 - - [06/Dec/2024:13:27:21 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 3697 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.##.## 3128 - - [06/Dec/2024:13:27:24 +0000] "POST http://einstein3.aei.uni-hannover.de/EinsteinAtHome/cgi-bin/file_upload_handler_large HTTP/1.1" 200 4890936 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_MISS:HIER_DIRECT 192.168.##.## 3128 - - [06/Dec/2024:13:27:31 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 46729 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT Thanks very much for that. Question(s): (please, forgive my absence of and/or diabolical abuse of network syntax. I'm not testing you - I'm just a TCP novice asking irritating questions!) How does client_request_buffer_max_size 100 MB work? Is it merely adjusting a local squid pass-through filter? Or manipulating an established TCP connection? Or is this talking to the remote server target directly? Because adjusting this has enabled a connection that, whatever the default setting was, wasn't allowing. With all due respect, I wish someone would please post a visual TCP Tree this Christmas! Thank you. It seems the best practice with TCP is to keep it simple. I just don't know what (are the most important things) to keep simple! And finally - 192.168.##.## 3128 - - [05/Dec/2024:20:32:47 +0000] "CONNECT scheduler.einsteinathome.org:443 HTTP/1.1" 200 33682 "-" "BOINC client (x86_64-suse-linux-gnu 8.0.4)" TCP_TUNNEL:HIER_DIRECT This indicates an encrypted connection (via HTTPS port 443). Those connections are forwarded by Squid without inspecting the packets - like any router between client and server does. Is this "tunnelling"/forwarding a default squid response to any connection request that isn't described in squid.conf? I mean - is this squid access.log line describing itself or what something has told it? Thank you. ID: 51254 · Reply Quote

Toby Broom Volunteer moderator Send message Joined: 27 Sep 08 Posts: 852 Credit: 694,396,957 RAC: 120,888	Message 51255 - Posted: 6 Dec 2024, 18:33:19 UTC Last modified: 6 Dec 2024, 18:34:20 UTC I just nocahched einstien. acl nocache dstdomain .worldcommunitygrid.org radioactiveathome.org einsteinathome.org for the what the buffer does you can see the squid documents. https://www.squid-cache.org/Doc/config/client_request_buffer_max_size/ ID: 51255 · Reply Quote

computezrmle Volunteer moderator Volunteer developer Volunteer tester Help desk expert Send message Joined: 15 Jun 08 Posts: 2549 Credit: 255,373,399 RAC: 62,870	Message 51256 - Posted: 6 Dec 2024, 19:49:30 UTC - in response to Message 51254. Setting "client_request_buffer_max_size" is a workaround to mitigate a Squid bug reported for v5 in May 2022: https://bugs.squid-cache.org/show_bug.cgi?id=5214 According to the recent comments it may be solved in v7 (but this needs to be tested!): https://bugs.squid-cache.org/show_bug.cgi?id=5214#c13 https://github.com/squid-cache/squid/pull/1887 Is this "tunnelling"/forwarding a default squid response to any connection request that isn't described in squid.conf? I mean - is this squid access.log line describing itself or what something has told it? It is not a default. Instead the required options how to handle those connections are part of the proposed minimum squid.conf that is part of the Squid packet since the invention of the wheel. For more information please read the Squid manual (Toby already posted a link). some hints: acl SSL_ports port 443 acl CONNECT method CONNECT http_access deny CONNECT !SSL_ports ID: 51256 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51257 - Posted: 6 Dec 2024, 20:53:52 UTC Well thanks both. Food for thought. ID: 51257 · Reply Quote

Guy Send message Joined: 9 Feb 08 Posts: 53 Credit: 1,256,579 RAC: 881	Message 51260 - Posted: 9 Dec 2024, 8:23:03 UTC Hello, Squid is working. Aside - Is this cache refresh - 192.eg.eg.eg 3128 - - [09/Dec/2024:07:51:03 +0000] "GET http://s1ral-cvmfs.openhtc.io/cvmfs/cernvm-prod.cern.ch/.cvmfspublished HTTP/1.1" 200 1995 "-" "cvmfs Fuse 2.5.2 11ac1fe9-e05a-44b5-b29e-ca3d2370db11" TCP_REFRESH_MODIFIED:HIER_DIRECT 192.eg.eg.eg 3128 - - [09/Dec/2024:07:51:06 +0000] "GET http://s1ral-cvmfs.openhtc.io/cvmfs/sft.cern.ch/.cvmfspublished HTTP/1.1" 200 2000 "-" "cvmfs Fuse 2.5.2 11ac1fe9-e05a-44b5-b29e-ca3d2370db11" TCP_REFRESH_MODIFIED:HIER_DIRECT supposed to happen - every 4 minutes? My concern is that this frequent communication with the data servers seems to negate the point of cacheing in my machine. ID: 51260 · Reply Quote

computezrmle Volunteer moderator Volunteer developer Volunteer tester Help desk expert Send message Joined: 15 Jun 08 Posts: 2549 Credit: 255,373,399 RAC: 62,870	Message 51261 - Posted: 9 Dec 2024, 9:16:42 UTC - in response to Message 51260. Those requests are created by the CVMFS client running inside your VMs (or directly on your host if you run native apps). Without a local Squid you just don't notice them (sometimes you may since .cvmfspublished can occasionally be larger than 100 MB). The update frequency is set by the CVMFS repository manager and since .cvmfspublished is a main repository definition file you wouldn't want to use an outdated version on your side. The benefit using a Squid increases per active CVMFS instance you are running. Example: Your computer is able to run more than 1 Theory VMs. Each of them will start it's own CVMFS client. When 1 CVMFS refreshes .cvmfspublished subsequent requests will be served from the Squid cache until it needs to be refreshed again 4 min later. ID: 51261 · Reply Quote

LHC@home