Questions and Answers : Unix/Linux : Best/Preferred Proxy
Message board moderation

To post messages, you must log in.

AuthorMessage
TribbleRED
Avatar

Send message
Joined: 9 May 10
Posts: 14
Credit: 3,104,448
RAC: 0
Message 44784 - Posted: 22 Apr 2021, 7:31:33 UTC

Hello All,

This is for both Win and Linux but I opted to post here instead of there...

I am looking for a proxy solution to handle several nodes running 48wu at a time. Presently my list has come down to Squid, Varnish, and NGINX; with NGINX being more of a web server that can proxy. I would also very much like to set up with fail-over capability as well.

Presently all my other projects are able to run on the nodes(48-56 threads per) in my racks but I can't do the same for either LHC or CPDN. Hence the proxy...

Any help would be appreciated.

Cheers
ID: 44784 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1268
Credit: 8,421,616
RAC: 2,139
Message 44785 - Posted: 22 Apr 2021, 8:10:47 UTC
Last modified: 22 Apr 2021, 8:11:18 UTC

ID: 44785 · Report as offensive     Reply Quote
TribbleRED
Avatar

Send message
Joined: 9 May 10
Posts: 14
Credit: 3,104,448
RAC: 0
Message 44786 - Posted: 22 Apr 2021, 8:53:51 UTC - in response to Message 44785.  
Last modified: 22 Apr 2021, 8:57:08 UTC

Doh.

I had searched for threads to answer these questions but I didn't seem to find this one. Thank you for posting the link.

So it appears the one that is suggested is Squid, but why? Is it for it's reverse proxy capabilities or because its free...? There could be better solutions than Squid and like I said I am also in need of some fail-over capability.

Thoughts?
ID: 44786 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,923,295
RAC: 137,719
Message 44787 - Posted: 22 Apr 2021, 9:28:01 UTC - in response to Message 44786.  

Feel free to use any HTTP proxy you are familiar with, especially if you claim there is a better one than Squid.

CERN and other large science organizations from all over the world use hundreds of Squids for many years to serve billions of requests per day. Cloudflare distributes LHC@home data via "openhtc.io" using hundreds of Squids in their datacenters.

Fail-over can easily be implemented if your local apps support it - and that's the pitfall.
It's BOINC that does NOT support an automatic fail-over configuration.

48 concurrently running tasks would just be a warmup for a single Squid instance.
ID: 44787 · Report as offensive     Reply Quote
TribbleRED
Avatar

Send message
Joined: 9 May 10
Posts: 14
Credit: 3,104,448
RAC: 0
Message 44790 - Posted: 22 Apr 2021, 17:26:07 UTC - in response to Message 44787.  
Last modified: 22 Apr 2021, 17:27:17 UTC

But why is it used? Specifically Squid. Do you know?
ID: 44790 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,923,295
RAC: 137,719
Message 44803 - Posted: 24 Apr 2021, 11:24:24 UTC - in response to Message 44790.  

This is a 24-h-statistics page for atlasfrontier-ai.cern.ch:
http://wlcg-squid-monitor.cern.ch/awstats/bin/awstats.pl?config=atlasfrontier.cern.ch&databasebreak=day&day=16

Total requests served: 188015011
Served from the cache: 175401418 (93%; HITS and MEM_HITS)

Top 4 clients are other proxies inside the CERN network:
ca-mey-frontier-*.cern.ch
Each of them made 40-46 million requests (176.33 million in total) and distributed them to their clients.

Guessed they run at the same efficiency of 93% their clients made 2.52 billion requests within 1 day.
You wouldn't want to treat your original servers with that many requests, hence a proxy (cache) cascade.


As already mentioned atlasfrontier-ai.cern.ch is just 1 out of many proxies used by CERN and others.
They use Squid for many years as it is fast, reliable and fulfills all other major and minor requirements.
ID: 44803 · Report as offensive     Reply Quote

Questions and Answers : Unix/Linux : Best/Preferred Proxy


©2024 CERN