Message boards : ATLAS application : Squid proxies may need restart
Message board moderation

To post messages, you must log in.

AuthorMessage
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 376
Credit: 13,871,290
RAC: 8,650
Message 42774 - Posted: 2 Jun 2020, 14:32:57 UTC

Hi all,

This message is only relevant if you run your own squid proxy server for ATLAS tasks.

After the CERN database outage last week, a problem was seen with the cached information on squid proxy servers all over the ATLAS Grid which can cause tasks to fail. The solution to the problem is to restart the squid service, so if you are running your own squid please restart it in order to avoid potential problems.

The ATLAS-managed squid servers which tasks use by default were restarted earlier today, so if you saw strange failures in tasks between Thursday last week and now this might have been the reason.
ID: 42774 · Report as offensive     Reply Quote
CloverField

Send message
Joined: 17 Oct 06
Posts: 63
Credit: 33,889,205
RAC: 47,519
Message 42777 - Posted: 2 Jun 2020, 17:33:54 UTC - in response to Message 42774.  

Hi all,

This message is only relevant if you run your own squid proxy server for ATLAS tasks.

After the CERN database outage last week, a problem was seen with the cached information on squid proxy servers all over the ATLAS Grid which can cause tasks to fail. The solution to the problem is to restart the squid service, so if you are running your own squid please restart it in order to avoid potential problems.

The ATLAS-managed squid servers which tasks use by default were restarted earlier today, so if you saw strange failures in tasks between Thursday last week and now this might have been the reason.


By restart do you just mean squid -k restart
or deleting the cache and starting fresh?
ID: 42777 · Report as offensive     Reply Quote
Greger

Send message
Joined: 9 Jan 15
Posts: 151
Credit: 431,596,822
RAC: 0
Message 42779 - Posted: 2 Jun 2020, 18:24:07 UTC

So today's power outage was not entirely unnecessary. Have not cleared cache.

restart the squid service
It sounds like restart daemon is enough.
ID: 42779 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2049
Credit: 154,306,356
RAC: 146,312
Message 42780 - Posted: 2 Jun 2020, 18:28:39 UTC - in response to Message 42777.  

This is also in regards to your post in the Theory thread:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5431&postid=42775


You may first check your access.log and cache.log.
Do you notice error messages that correspond to your issues?

If no, squid is most likely running fine and the issues are caused by something else.

If yes, you should clear the cache and restart fresh.

You may also insert the following line in your squid.conf and do a "squid -k reconfigure".
shutdown_lifetime 3 seconds

This avoids the 60 seconds default delay when you shutdown/restart squid but I'm not 100% sure if changing this timeout requires a squid -k restart. At least Squid will be prepared for the next restart.
ID: 42780 · Report as offensive     Reply Quote
CloverField

Send message
Joined: 17 Oct 06
Posts: 63
Credit: 33,889,205
RAC: 47,519
Message 42786 - Posted: 2 Jun 2020, 23:52:47 UTC - in response to Message 42780.  

This is also in regards to your post in the Theory thread:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5431&postid=42775


You may first check your access.log and cache.log.
Do you notice error messages that correspond to your issues?

If no, squid is most likely running fine and the issues are caused by something else.

If yes, you should clear the cache and restart fresh.

You may also insert the following line in your squid.conf and do a "squid -k reconfigure".
shutdown_lifetime 3 seconds

This avoids the 60 seconds default delay when you shutdown/restart squid but I'm not 100% sure if changing this timeout requires a squid -k restart. At least Squid will be prepared for the next restart.


The logs look good to me. And post the squid restart everything seems to be fine.
I just got some atlas tasks though so I assume they will kill at least one theory.
If I get another stuck one. Ill nuke the cache and also do a project reset to see if that solves the issue.
ID: 42786 · Report as offensive     Reply Quote
Aurum
Avatar

Send message
Joined: 12 Jun 18
Posts: 109
Credit: 38,273,822
RAC: 15,220
Message 42864 - Posted: 13 Jun 2020, 21:34:57 UTC - in response to Message 42780.  
Last modified: 13 Jun 2020, 21:35:26 UTC

You may also insert the following line in your squid.conf and do a "squid -k reconfigure".
shutdown_lifetime 3 seconds
This avoids the 60 seconds default delay when you shutdown/restart squid but I'm not 100% sure if changing this timeout requires a squid -k restart. At least Squid will be prepared for the next restart.
Not sure where to stick it but this spot felt oh so right:
# You don't believe this is enough? For sure, it is!
cache_mem 192 MB
maximum_object_size_in_memory 24 KB
memory_replacement_policy heap GDSF

shutdown_lifetime 3 seconds

No idea what I'm doing. Is there an LHC Squid Care & Feeding Guide anywhere?
ID: 42864 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 1622
Credit: 75,604,709
RAC: 224,236
Message 46401 - Posted: 2 Mar 2022, 7:35:25 UTC

CentOs8-VM (RedHat):
systemctl stop squid in console.
This will take a few seconds (nearly 10 seconds). No more command is needed.
ID: 46401 · Report as offensive     Reply Quote

Message boards : ATLAS application : Squid proxies may need restart


©2022 CERN