Posts by Darrell

1) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39104) Posted 12 Jun 2019 by Darrell Post: @ computezrmle: PurpleHat wrote: You would go to boinc manager to each computer at Options->Other Options-> HTTP Proxy and put in ip and port to host that are running squid. If squid is not running boinc-manager would tell you that need internet access and if you are running more projects then LHC it is possible to exclude this in same section. Simply add projecturl in [Don´t use proxy for] True. If using Windows and a Client Configuration file, adding the following may be easier if multiple machines use the same configuration: <cc_config> <options> <proxy_info> <http_server_name>[your server here]</http_server_name> <http_server_port>[your server port number here]</http_server_port> <no_proxy>[comma separated list of URLs to not* use the proxy]*</no_proxy> <no_autodetect>0</no_autodetect> </proxy_info> </options> </cc_config>
2) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39098) Posted 10 Jun 2019 by Darrell Post: @ computezrmle: I already tried removing "dns_nameservers 1.1.1.1 1.0.0.1 # fast DNS resolvers", and it fails with "CONNECT lhcathome.cern.ch:443 HTTP/1.1" 503 151 "-" "BOINC client (windows_x86_64 7.14.2)" TAG_NONE:HIER_NONE Replacing it gives "CONNECT lhcathome.cern.ch:443 HTTP/1.1" 200 45331 "-" "BOINC client (windows_x86_64 7.14.2)" TCP_TUNNEL:HIER_DIRECT so it appears it must operate differently in Windows vs. Linux. I will test your suggestions to see which works and which others don't when I have more time available. As for right now, I am running. Ahh, so much to learn, so little time to do so.
3) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39092) Posted 10 Jun 2019 by Darrell Post: @ computezrmle: I agree that local caching doesn't work with "localhost" or "127.x.y.z" or "::". It DOES work for me with the NAME of the local computer, e.g., "DDW3770K" in the HTTP Proxy address. NOTE HOWEVER, it uses the second LAN interface IP address on this computer EVEN THOUGH IT IS DISABLED. I don't have a computer with only a single interface available at the moment for testing, so YMMV. Anyone able to confirm using the computer name will work with a single LAN interface on the same computer? Also, using the name from another computer on the LAN instead of an IP address also works.
4) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39091) Posted 10 Jun 2019 by Darrell Post: This is what I did to install under Windows: I started here: http://www.squid-cache.org/ then clicked "download" in the menu Since I cannot compile Squid, I clicked on "binary package of Squid." part way down Then clicked link "Squid-3.5" in 1.2.17.1 to get the 3.5 version for Windows Under the 64bit, I clicked "http://squid.diladele.com/" Then selected "Squid for Windows" "DOWNLOAD MSI" which starts a 33.6mb download of Squid I installed it on disk "S" (requires Admin privilege) Using Notepad, I modified "S:\Squid\etc\squid\squid.conf" as per the below: # Squid configuration for BOINC under Windows Squid configuration for BOINC under Windows Squid configuration for BOINC under Windows Squid configuration for BOINC under Windows # Based on squid version 3.5 # See also: http://www.squid-cache.org/ # NOTE: I AM NOT AN EXPERT IN SQUID, and don't know exactly what is and is not required. However, it is sufficient even if not all necessary for my use. # Where I found info about it: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4611&postid=36101#36101 # Many thanks! # Define your local hosts/networks here # Examples: # acl crunchers src 172.16.0.20 # acl crunchers src 172.16.0.25 # acl localnet src 172.16.0.0/12 # acl localnet src fc00::/7 acl crunchers src 192.168.1.120-192.168.1.128 # my local LAN addresses acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range # required for some extras acl to_httpport port 80 acl to_http8000 port 8000 acl to_squidport port 3128 acl SSL_ports port 443 # ssl acl Safe_ports port 443 # https acl Safe_ports port 80 # http acl Safe_ports port 1025-65535 # unregistered ports acl CONNECT method CONNECT follow_x_forwarded_for allow localhost follow_x_forwarded_for deny all # # Start of extra section 1 # Requests that need special handling # worldcommunitygrid doesn't like data from the local cache # use the following lines as template if other projects also have problems acl wcg_nocache dstdomain .worldcommunitygrid.org always_direct allow wcg_nocache cache deny wcg_nocache # if CVMFS uses geoapi, ensure it's checked directly acl cvmfs_geoapi urlpath_regex -i ^/+cvmfs/+[0-9a-z._~-]+/+api/+[0-9a-z._~-]+/+geo/+[0-9a-z._~-]+/+[0-9a-z.,_~-]+ always_direct allow cvmfs_geoapi cache deny cvmfs_geoapi # avoids polluting the disk cache with typical onetimers, e.g. ATLAS job data acl boinc_nocache urlpath_regex -i /download[0-9a-z._~-]*/+[0-9a-z._~-]+/+.+ cache deny boinc_nocache # seriously: do NOT cache that! # Based on a frontier cache suggestion from Fermilab acl PragmaNoCache req_header Pragma no-cache cache deny PragmaNoCache # End of extra section 1 # # # Start of extra section 2 # not used in this basic configuration # include /etc/squid/extensions.d/cern_extensions.conf # End of extra section 2 # # # Start of extra section 3 # not used in this basic configuration # parent cache configuration # doesn't improve performance but gains more data for analysis #include /etc/squid/parents.d/s1x-cvmfs_openhtc_io.conf #include /etc/squid/parents.d/lhcb-portal-dirac_cern_ch.conf #include /etc/squid/parents.d/cvmfs-stratum-one_cern_ch.conf #include /etc/squid/parents.d/cmsfrontier_cern_ch.conf #include /etc/squid/parents.d/lcgft-atlas_gridpp_rl_ac_uk.conf #include /etc/squid/parents.d/lhchomeproxy_cern_ch.conf # End of extra section 3 # # Deny requests to certain unsafe ports http_access deny !Safe_ports # Deny CONNECT to other than secure SSL ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager # The following should be uncommented to protect innocent # web applications running on the proxy server who think the only # one who can access services on "localhost" is a local user #http_access deny to_localhost # # INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS # see ACL definition above # Examples: # http_access allow crunchers # http_access allow localnet http_access allow crunchers http_access allow localnet http_access allow localhost http_access deny all # http_port # don't bind it to an IP that is accessible from outside unless you know what you do. # Examples: # http_port 172.16.0.77:3128 http_port 3128 # # Start of extra section 4 # not used in this basic configuration # used for additional ports # Example: # http_port 172.16.0.77:3129 intercept # End of extra section 4 # # if your machine has more than one IP # Example: # tcp_outgoing_address 172.16.0.88 # Required OFF for intercepted traffic from LHCb VMs client_dst_passthru off # You don't believe this is enough? # For sure, it is! cache_mem 192 MB maximum_object_size_in_memory 24 KB memory_replacement_policy heap GDSF # Keep it large enough to store vdi files in the cache. # See extra section 1 how to avoid onetimers eating up your storage. # min-size=xxx keeps very small files away from your disk cache_replacement_policy heap LFUDA maximum_object_size 6144 MB cache_dir aufs /var/cache/squid 32000 16 64 min-size=7937 # logformat has to be changed according to your needs and the capabilities of your logfile analyser # See: http://www.squid-cache.org/Versions/v3/3.5/cfgman/logformat.html # if unsure, use the default setting logformat my_awstats %>A %lp %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh access_log stdio:/var/log/squid/access_squid.log logformat=my_awstats strip_query_terms off netdb_filename none coredump_dir none ftp_user anonymous@ # max_stale 1 week #default # extended to be prepared for a project reset max_stale 37 days # 1 line is required to avoid the ancient default settings # be conservative # don't violate the HTTP standards refresh_pattern . 0 0% 0 # Required for Windows (but perhaps not for Linux?) dns_nameservers 1.1.1.1 1.0.0.1 # fast DNS resolvers store_avg_object_size 1 MB # booster 1! collapsed_forwarding on # booster 2! client_persistent_connections on server_persistent_connections on digest_generation off log_icp_queries off # use your preferred language here error_default_language en dns_defnames on #depends on your local setup dns_v4_first on forwarded_for transparent ##### End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf End of squid.conf I started the Squid program (S:\Squid\bin\Diladele.Squid.Tray.exe or link on the desktop). Then on my OTHER machines under BOINC Manager, options, Other options, HTTP Proxy entered the IPAddress of THIS machine and port 3128. Local caching doesn't seem to work but from the other machines, it does.
5) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39090) Posted 9 Jun 2019 by Darrell Post: @ computezrmle: I am a rank noobie in Squid knowledge, but I got it "running" under Windows. This means I don't get error messages or crashes, BOINC runs and gets data, etc. However, I also don't have tools or techniques to know if it is effective or not. Where do I get the programs to analyze its logs? Also, with regard to your post just above, it appears to me that using the "::" IPV6 form for "this computer" is also not effective. Can you confirm? Thanks.
6) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39084) Posted 9 Jun 2019 by Darrell Post: @ PurpleHat: Ahh, I see a bit of light! Thanks for pointing out that the post IS the config (for Linux-type). I will have to try to translate into Windowese to try it. And also your pointers. Thanks again, PurpleHat
7) Message boards : Number crunching : Setting up a local squid cache for a home cluster - old comments (Message 39079) Posted 8 Jun 2019 by Darrell Post: I second what Dingo wrote. I am running 4x16GB i7 computers and 2x32GB Threadrippers with LHC in about half their capacities, mostly Theory and a little Atlas. I am knowledgeable in computing and Windows, but I am not in Squid. Perhaps one of the experts here could write a tutorial for us non-experts on how to do it. It would help us AND Cern by reducing traffic.
8) Message boards : ATLAS application : Very long tasks in the queue (Message 29542) Posted 22 Mar 2017 by Darrell Post: @ Yeti: I think NOT DONE! I am running my first 16 core Atlas with an estimated runtime of 1hr 20min, and it is already at 13hr 47min.
9) Questions and Answers : Windows : Unable to set priority of Virtualbox processing (Message 29234) Posted 13 Mar 2017 by Darrell Post: @ marmot: Thanks for the reference. It looks to be quite powerful and I will need some time to check it out. Sorry for the slow reply.
10) Questions and Answers : Windows : Unable to set priority of Virtualbox processing (Message 29233) Posted 13 Mar 2017 by Darrell Post: @ Crystal Pellet: Thank you for the technique - it works great, and I apologize for being so slow returning to thank you.
11) Questions and Answers : Windows : Unable to set priority of Virtualbox processing (Message 28723) Posted 30 Jan 2017 by Darrell Post: When the Virtualbox applications run on all or nearly all cores, they lockout other processing, including the keyboard, and other applications. Can the applications/Virtualbox processing priority be set lower to allow low CPU usage above them (e.g., windows explorer)?

LHC@home