Message boards :
ATLAS application :
Atlas Native Transient HTTP Errors Uploading Resultfile
Message board moderation
Author | Message |
---|---|
Send message Joined: 2 Sep 04 Posts: 455 Credit: 201,155,667 RAC: 29,782 |
Hi, after quit some time of abstinenz form LHC I now tried to run Atlas Native inside a VirtualBox with Ubuntu 20.04. All seemed to do fine, I could see the Athena-Tasks running, the WU has finished and tried to upload the results. All files but one where uploaded succesfull: Manni VL_CI01_31501 Started upload of tLNLDm26CV1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmF51LDm3rzjam_0_r1453389463_ATLAS_result Started upload of tuXMDmju8U1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmxm1LDmNZfCSo_0_r1401751060_ATLAS_result Project communication failed: attempting access to reference site Temporarily failed upload of tLNLDm26CV1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmF51LDm3rzjam_0_r1453389463_ATLAS_result: transient HTTP error Backing off 03:38:09 on upload of tLNLDm26CV1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmF51LDm3rzjam_0_r1453389463_ATLAS_result Internet access OK - project servers may be temporarily down. Temporarily failed upload of tuXMDmju8U1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmxm1LDmNZfCSo_0_r1401751060_ATLAS_result: transient HTTP error Backing off 03:58:24 on upload of tuXMDmju8U1nfZGDcpSWOuwoABFKDmABFKDmTzMbDmxm1LDmNZfCSo_0_r1401751060_ATLAS_result So, who has a problem? The Upload-Server, my local Squid or my firewall, letting something not through ? Supporting BOINC, a great concept ! |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
Hi Yeti, All files ... where uploaded Your Theory task reported a success. ... but one ... The huge one. Which Squid did you use? The one you used in the past or a new one (on the Linux VM)? In case it's the latter, is it version >4.x? In that case please downgrade to the most recent 4.x. |
Send message Joined: 2 Sep 04 Posts: 455 Credit: 201,155,667 RAC: 29,782 |
Your Theory task reported a success.Yes, but it had only run 3 - 5 Minutes and I was not shure, if this is okay. Which Squid did you use?4.14 The one you used in the past or a new one (on the Linux VM)?The "old" one, I'm running since you helped me setting it up years ago Supporting BOINC, a great concept ! |
Send message Joined: 2 Sep 04 Posts: 455 Credit: 201,155,667 RAC: 29,782 |
|
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
Your Theory task reported a success. Theory native is not very verbose, but this line points out it was a success: 18:00:08 CEST +02:00 2022-07-11: cranky-0.0.32: [INFO] Container 'runc' finished with status code 0. Regarding ATLAS: Meanwhile there are some valid tasks. Nonetheless, on your Linux VM please run the following commands and post the output: cvmfs_config stat atlas cvmfs_config showconfig -s atlas |grep -E 'CVMFS_(HTTP_PROXY|SERVER_URL|USE_CDN)' |
Send message Joined: 2 Sep 04 Posts: 455 Credit: 201,155,667 RAC: 29,782 |
cvmfs_config stat atlasroot@mannivl:~# cvmfs_config stat atlas VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE 2.9.4.0 88846 9 42784 106736 2 78 2489500 4096001 1564 130560 0 98473 99.975 16748 4563 http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1 cvmfs_config showconfig -s atlas |grep -E 'CVMFS_(HTTP_PROXY|SERVER_URL|USE_CDN)'root@mannivl:~# cvmfs_config showconfig -s atlas |grep -E 'CVMFS_(HTTP_PROXY|SERVER_URL|USE_CDN)' CVMFS_HTTP_PROXY='auto;DIRECT' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf CVMFS_SERVER_URL='http://s1cern-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1ral-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch;http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1ihep-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch;http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/atlas.cern.ch' # from /etc/cvmfs/domain.d/cern.ch.conf CVMFS_USE_CDN=yes # from /etc/cvmfs/default.local Supporting BOINC, a great concept ! |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
I suggest to replace CVMFS_HTTP_PROXY='auto;DIRECT' with CVMFS_HTTP_PROXY="http://<name_or_IP_of_your_local_squid>:<port>;DIRECT" in /etc/cvmfs/default.local Then run "cvmfs_config reload" and check with "cvmfs_config stat" if "DIRECT" changes to the local proxy's IP. |
Send message Joined: 2 Sep 04 Posts: 455 Credit: 201,155,667 RAC: 29,782 |
|
Send message Joined: 3 Nov 12 Posts: 58 Credit: 141,021,081 RAC: 68,848 |
What's wrong with squid 5.x? Gave it a try, but no uploads to ATLAS. Had to switch back to 4.x :-( |
Send message Joined: 2 May 07 Posts: 2242 Credit: 173,901,518 RAC: 2,790 |
CMS and Theory-native have no problem with Squid 5.5. |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
According to the Squid bugtracker Squid 5.x suffers from a bug that affects the POST method: https://bugs.squid-cache.org/show_bug.cgi?id=5214 v4.x is not affected v6.x may include a fix but is a development version v5.x does not (yet) include a backport of the v6 fix Theory dos not use the POST method => it works fine with Squid 5.x. ATLAS returns the result file to the BOINC client which uploads it using POST. CMS uploads subtask results directly from within the VM using PUT (which is also affected). Hence, as of now ATLAS and CMS will not upload their results if - a Squid 5.x is configured for BOINC (ATLAS) - a Squid 5.x is used in a complex firewall environment that transparently forces HTTP traffic through a Squid 5.x (ATLAS and CMS) Conclusion: V4.x is still the recommended Squid version. |
Send message Joined: 2 May 07 Posts: 2242 Credit: 173,901,518 RAC: 2,790 |
2022-10-30 10:39:57 (4540): Guest Log: [INFO] Reloading and probing the CVMFS configuration 2022-10-30 10:40:03 (4540): Guest Log: [INFO] Probing /cvmfs/cvmfs-config.cern.ch... OK 2022-10-30 10:40:03 (4540): Guest Log: [INFO] Probing /cvmfs/grid.cern.ch... OK 2022-10-30 10:40:06 (4540): Guest Log: [INFO] Probing /cvmfs/cms.cern.ch... OK 2022-10-30 10:40:06 (4540): Guest Log: [INFO] Probing /cvmfs/singularity.opensciencegrid.org... OK 2022-10-30 10:40:06 (4540): Guest Log: [INFO] Probing /cvmfs/cms-ib.cern.ch... OK 2022-10-30 10:40:06 (4540): Guest Log: [INFO] Probing /cvmfs/oasis.opensciencegrid.org... OK 2022-10-30 10:40:07 (4540): Guest Log: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY 2022-10-30 10:40:07 (4540): Guest Log: [INFO] 2.7.2.0 http://s1cern-cvmfs.openhtc.io http://10.xxx.yyy.zz:3128 2022-10-30 10:40:07 (4540): Guest Log: [INFO] Reading volunteer information 2022-10-30 10:40:07 (4540): Guest Log: [INFO] Requesting an X509 credential from LHC@home 2022-10-30 10:40:08 (4540): Guest Log: [INFO] CMS application starting. Check log files. ...... 2022-10-31 03:20:44 (4540): Status Report: Elapsed Time: '60000.000000' 2022-10-31 03:20:44 (4540): Status Report: CPU Time: '3246.156250' 2022-10-31 04:40:48 (4540): Powering off VM. 2022-10-31 04:40:49 (4540): Successfully stopped VM. 2022-10-31 04:40:49 (4540): Deregistering VM. (boinc_babf107dda7402da, slot#5) https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10548292 |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
The log snippet (obviously from a CMS task) just shows that the VM's CVMFS client accepts an HTTP proxy. Any HTTP proxy would be shown like this, not only a Squid. The CVMFS client also doesn't know whether it's a v3, v4, or v... Squid version. CVMFS works fine even with a Squid v5 since it only makes downloads - it never uploads anything. That's one reason why Theory works fine with a Squid v5. It's the upload direction that is affected by the v5 bug. CMS's stderr.txt logs never show such errors. CMS tries to upload the result a couple of times before it gives up and starts a fresh subtask. |
Send message Joined: 2 May 07 Posts: 2242 Credit: 173,901,518 RAC: 2,790 |
This thread is better in folder number crunching - Squid. Seeing no problems for CMS-Tasks uploading, as you are writing. |
Send message Joined: 15 Jun 08 Posts: 2531 Credit: 253,722,201 RAC: 41,981 |
I clearly pointed out under which conditions this can happen. That information is useful for those who have a more complex LAN/Firewall than your's. For them Squid v4 is still the better choice. If you are not affected, ignore it. |
Send message Joined: 4 Oct 21 Posts: 10 Credit: 43,746,383 RAC: 16,241 |
I've used squid v5 from the start. You can tell boinc to bypass squid for result uploads: boinc manager -> Options -> other options -> http proxy -> don't use proxy for -> http://lhcathome-upload.cern.ch/lhcathome_cgi/file_upload_handler Or put it in cc_config. Squid is useless for result uploads anyway. The workaround mentioned in the bug above works also. With client_request_buffer_max_size 512 MB atlas uploads go straight through or usually on their 1st retry. |
©2024 CERN