Message boards : ATLAS application : CernVM-FS- Central using for all PC's at home
Message board moderation

To post messages, you must log in.

AuthorMessage
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45573 - Posted: 30 Oct 2021, 5:35:16 UTC
Last modified: 30 Oct 2021, 6:31:08 UTC

Is it possible to manage CernVM-FS in a storage-System at home?
HPC- High performance computing
http://cernvm.cern.ch
squid:https://cvmfs.readthedocs.io/en/stable/cpt-squid.html
ID: 45573 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45574 - Posted: 30 Oct 2021, 16:03:01 UTC - in response to Message 45573.  

Installing a Frontier squid cache server

The frontier-squid software package is a patched version of the standard squid http proxy cache software, pre-configured for use by the Frontier distributed database caching system. This installation is recommended for use by Frontier in the LHC CMS & ATLAS projects, and also works well with the CernVM FileSystem. Many people also use it for other applications as well; if you have any questions or comments about general use of this package contact frontier-talk@cern.ch.

We need a solution from Cern-IT to use frontier squid instead of normal squid.
ID: 45574 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,018,512
RAC: 136,348
Message 45580 - Posted: 31 Oct 2021, 8:37:23 UTC - in response to Message 45574.  

Frontier Squid is a package provided by CERN/Fermilab (and others) for use in large datacenter environments.
It includes scripts that makes it easy to centralize the configuration or updates in a multi Squid environment as well as to connect Squid to monitoring systems like the global WLCG monitoring.
It can, of course, also be used in a small home cluster environment.

Feel free to download a Frontier Squid package, compare it with a standard Squid and share your experience.



Large vs. small
Large means a datacenter with thousands of computers generating more than 100 million HTTP requests per day.
Small means a cluster with up to 500 computers generating 10 million HTTP requests per day.
ID: 45580 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45581 - Posted: 31 Oct 2021, 9:05:55 UTC - in response to Message 45574.  
Last modified: 31 Oct 2021, 9:08:02 UTC

We need a solution from Cern-IT to use frontier squid instead of normal squid.

Personal squid on a desktop/laptop

If you want to install a Frontier squid on your personal desktop or laptop, just follow the same instructions as under Software above, except:

For the NET_LOCAL acl, use "127.0.0.1/32"
For the cache_dir size you can leave it at the default 10000 or even perhaps cut it down to 5000 if you want to.
ID: 45581 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,018,512
RAC: 136,348
Message 45582 - Posted: 31 Oct 2021, 9:17:56 UTC - in response to Message 45581.  

I suspect nobody would be able to follow this.
Could you please explain the ideas behind your self-citation and the (obvious) snippet from a certain Frontier Squid page "Personal squid on ... 5000 if you want to."
ID: 45582 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45583 - Posted: 31 Oct 2021, 9:24:07 UTC - in response to Message 45582.  

It's a question for Cern-IT and not more.
ID: 45583 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,018,512
RAC: 136,348
Message 45584 - Posted: 31 Oct 2021, 9:29:22 UTC - in response to Message 45583.  

It's a question for Cern-IT ...

I doubt anybody there will understand what you want.
ID: 45584 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45586 - Posted: 31 Oct 2021, 22:42:49 UTC - in response to Message 45584.  

It's a question for Cern-IT ...

I doubt anybody there will understand what you want.

Will it explain with my small English knowlegde.
Native VM with RHEL8 - CentOS8.
When it is possible to tranfer a frontier-squite package to the VM when the task is starting,
it can control the requests for Atlas during the running of the task.

How this is possible for the applet in Windows, don't know, also
in CMS during running phase of the Task.
Noone from the most Volunteers here in the Projects is able to install the normal squid for this Projects.

Had it for a test only with your help. BUT HAVE NO BOTTLELECK, Seeing the long time of running tasks from my side.
In the moment one GByte download in Atlas in 2 and a half Minutes.

If someone is a expert with 40 Years IT knowledge, seeing this complex of a Proxy.
We need a solution from Cern-IT.
btw WCG is running without it since the beginning.
ID: 45586 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,018,512
RAC: 136,348
Message 45593 - Posted: 2 Nov 2021, 11:29:29 UTC - in response to Message 45586.  

Native VM with RHEL8 - CentOS8.
When it is possible to tranfer a frontier-squite package to the VM when the task is starting,
it can control the requests for Atlas during the running of the task.

There's no "Native VM" provided by the project.
ATLAS/Theory native run directly on a Linux based computer.

You run VirtualBox on a Windows host.
Then you create your own VM on that host to run CentOS8 as guest.
That's what you call your "Native VM".
That self created VM has nothing to do with the preconfigured VMs distributed by the LHC project server.

From the project's perspective there's no major difference between CentOS8 running on real hardware or CentOS8 running inside a self created VM.
Transferring a software packet (e.g. frontier squid) wouldn't be a problem, but you would need root rights to install and configure it in that CentOS8.
I wouldn't want anybody from outside to be root on one of my real computers, hence also not on my self created VMs.

Finally this is the major reason why a Squid packet serving all computers of a local cluster must be installed and configured by the local admin.
In addition this Squid instance must know which IPs are in use at the local network, an information CERN-IT doesn't have.



For the NET_LOCAL acl, use "127.0.0.1/32"

A citation you made in another post.
Unfortunately you didn't understand why this has been written in the Frontier manual.

It's a suggestion originally for people running a laptop from CERN who want to have a Squid instance on that laptop.
In that case the Frontier Squid package can be used and it can be limited to serve only processes on the entire laptop.
With this setting it would only be possible to use the Squid from processes running "native" on the entire laptop, e.g. a local CVMFS client.
It would not even be possible to access the Squid from a VM on the same laptop.
The reason is that 127.0.0.1 is an IP from a reserved range that is only valid inside it's runtime environment.
VM's are always running within a separate environment and have their own IP range 127.a.b.c.
A communication between both (or multiple) of those ranges is not possible, you would instead have to use other IPs for that, e.g. 192.168.f.g.




Bottlenecks

In this context a bottleneck can be caused by (not necessarily a complete list)
- a single bad/old network cable
- a powerline adapter somewhere in the network
- a weak wi-fi connection somewhere in the network
- a slow disk (not necessarily on the Squid box)
- a slow firewall
- a badly configured virus protection; perhaps it tests the files multiple times - on the Squid box and again on the target client
=> there's no need to run the LHC files through a virus protection software



It's not of any value to mention WCG.
Their data structure doesn't benefit from a client side proxy - the LHC data structure does.
ID: 45593 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45595 - Posted: 2 Nov 2021, 14:24:20 UTC - in response to Message 45593.  
Last modified: 2 Nov 2021, 14:52:00 UTC

Had no problems in the LAN before squid and have no problems after stopping squid two weeks ago (Download Atlas 2 GByte or 1 GByte).
Max. downloadspeed 70 MBit/s on all Win10pro and CentOS8/7-VM's!
IPv4 LAN-Adress and Portnumber of the squid are shown in the TASK-Protocol.
ID: 45595 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 45598 - Posted: 3 Nov 2021, 6:42:41 UTC - in response to Message 45595.  

This stats is shown here:https://lhcathome.cern.ch/lhcathome/host_stats.php
Windows total 8084 9.4360 %
Linux total 5550 89.8357 %
Most using of LHC@home is with Linux. Stats are Stats (Total Hosts??)
There is more work done with the many hosts on Cern-side as with our hosts.
It need a solution for us to reduce the requests of Atlas and CMS in a userfriendly way.
Halfknowledge is a point for a discussion, but no solution.
ID: 45598 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,189,216
RAC: 104,502
Message 47552 - Posted: 21 Nov 2022, 11:07:59 UTC

Why use frontier-squid instead of regular squid?

The most important reason is that with frontier-squid you get the benefit of many years of collective operational experience on the WLCG. The frontier-squid package contains configuration defaults and bug fixes that are known to work well with the applications used on the grid, plus some extra features in the packaging (see below).

http://twiki.cern.ch//twiki/bin/view/Frontier/InstallSquid
ID: 47552 · Report as offensive     Reply Quote

Message boards : ATLAS application : CernVM-FS- Central using for all PC's at home


©2024 CERN