LHC@home

Today my CMS tasks don't upload subtask results.
wmagentJob.log shows the messages below.
Might be caused by missing permissions on the data bridge.
Any other ideas?

2021-11-04 12:04:01,100:ERROR:StageOutImpl:Attempt 0 to stage out failed.
Automatically retrying in 300 secs
 Error details:
<@========== WMException Start ==========@>
Exception Class: StageOutError
Message: Command exited non-zero, ExitCode:13
Output: stdout: Thu Nov  4 12:03:54 CET 2021
Copying 121627572 bytes file:///srv/job/WMTaskSpace/cmsRun1/FEVTDEBUGoutput.root => https://data-bridge.cern.ch/myfed/cms-output/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/D1F0630A-5F5F-2648-AAC4-7C4F71AD9456.root
gfal-copy exit status: 13
ERROR: gfal-copy exited with 13
Cleaning up failed file:
Thu Nov  4 12:04:00 CET 2021
https://data-bridge.cern.ch/myfed/cms-output/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/D1F0630A-5F5F-2648-AAC4-7C4F71AD9456.root	MISSING

stderr: /srv/startup_environment.sh: line 2: BASHOPTS: readonly variable
/srv/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/srv/startup_environment.sh: line 28: EUID: readonly variable
/srv/startup_environment.sh: line 137: PPID: readonly variable
/srv/startup_environment.sh: line 145: SHELLOPTS: readonly variable
/srv/startup_environment.sh: line 159: UID: readonly variable
/srv/startup_environment.sh: line 191: syntax error near unexpected token `('
/srv/startup_environment.sh: line 191: `export probe_cvmfs_repos () '
gfal-copy error: 13 (Permission denied) - TRANSFER  ERROR: Copy failed with mode streamed, with error: Authentication error, reached maximum number of attempts
/srv/startup_environment.sh: line 2: BASHOPTS: readonly variable
/srv/startup_environment.sh: line 9: BASH_VERSINFO: readonly variable
/srv/startup_environment.sh: line 28: EUID: readonly variable
/srv/startup_environment.sh: line 137: PPID: readonly variable
/srv/startup_environment.sh: line 145: SHELLOPTS: readonly variable
/srv/startup_environment.sh: line 159: UID: readonly variable
/srv/startup_environment.sh: line 191: syntax error near unexpected token `('
/srv/startup_environment.sh: line 191: `export probe_cvmfs_repos () '
 
	ClassName : None
	ModuleName : WMCore.Storage.StageOutError
	MethodName : __init__
	ClassInstance : None
	FileName : /srv/job/WMCore.zip/WMCore/Storage/StageOutError.py
	LineNumber : 32
	ErrorNr : 0
	Command : #!/bin/bash
env -i X509_USER_PROXY=$X509_USER_PROXY JOBSTARTDIR=$JOBSTARTDIR bash -c '. $JOBSTARTDIR/startup_environment.sh; date; gfal-copy -t 2400 -T 2400 -p   file:///srv/job/WMTaskSpace/cmsRun1/FEVTDEBUGoutput.root https://data-bridge.cern.ch/myfed/cms-output/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/D1F0630A-5F5F-2648-AAC4-7C4F71AD9456.root'
            EXIT_STATUS=$?
            echo "gfal-copy exit status: $EXIT_STATUS"
            if [[ $EXIT_STATUS != 0 ]]; then
               echo "ERROR: gfal-copy exited with $EXIT_STATUS"
               echo "Cleaning up failed file:"
               env -i X509_USER_PROXY=$X509_USER_PROXY JOBSTARTDIR=$JOBSTARTDIR bash -c '. $JOBSTARTDIR/startup_environment.sh; date; gfal-rm -t 600 https://data-bridge.cern.ch/myfed/cms-output/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/D1F0630A-5F5F-2648-AAC4-7C4F71AD9456.root '
            fi
            exit $EXIT_STATUS
            
	ExitCode : 13
	ErrorCode : 60311
	ErrorType : GeneralStageOutFailure

Traceback: 

<@---------- WMException End ----------@>

2 typical examples from my proxy log.

boinc_client_a 2080 - - [04/Nov/2021:12:19:17 +0100] "PUT http://vc-cms-output.s3.cern.ch/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/D1F0630A-5F5F-2648-AAC4-7C4F71AD9456.root?AWSAccessKeyId=xxxx&Signature=xxxx&Expires=1636031956 HTTP/1.1" 100 193963 "-" "gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29" TCP_MISS_ABORTED:HIER_DIRECT
boinc_client_a 2080 - - [04/Nov/2021:12:19:24 +0100] "PUT http://vc-cms-output.s3.cern.ch/store/unmerged/logs/prod/2021/11/4/ireid_TC_SLC7_IDR_CMS_Home_211102_174358_9607/SinglePiE50HCAL_pythia8_2018_GenSimFull/0005/0/286958b5-5066-4f3a-b46a-ce9e6374aa8b-854-0-logArchive.tar.gz?AWSAccessKeyId=xxxx&Signature=xxxx&Expires=1636031962 HTTP/1.1" 200 336103 "-" "gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29" TCP_MISS:HIER_DIRECT
.
.
.
boinc_client_b 2080 - - [04/Nov/2021:12:21:50 +0100] "PUT http://vc-cms-output.s3.cern.ch/store/unmerged/DMWM_Test/RelValSinglePiE50HCAL/GEN-SIM/TC_SLC7_CMS_Home_IDRv5d-v11/00005/ED1AB787-DAE4-BF4E-9CE9-211862286A89.root?AWSAccessKeyId=xxxx&Signature=xxxx&Expires=1636032109 HTTP/1.1" 100 329295 "-" "gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29" TCP_MISS_ABORTED:HIER_DIRECT
boinc_client_b 2080 - - [04/Nov/2021:12:21:56 +0100] "PUT http://vc-cms-output.s3.cern.ch/store/unmerged/logs/prod/2021/11/4/ireid_TC_SLC7_IDR_CMS_Home_211102_174358_9607/SinglePiE50HCAL_pythia8_2018_GenSimFull/0005/0/286958b5-5066-4f3a-b46a-ce9e6374aa8b-857-0-logArchive.tar.gz?AWSAccessKeyId=xxxx&Signature=xxxx&Expires=1636032115 HTTP/1.1" 200 337117 "-" "gfal2-util/1.5.3 gfal2/2.18.1 neon/0.0.29" TCP_MISS:HIER_DIRECT

Uploads from the CMS VM to /store/unmerged/DMWM_Test/ fail.
Uploads from the CMS VM to /store/unmerged/logs/ succeed.