| Name | zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm_0 |
| Workunit | 233530188 |
| Created | 7 Jul 2025, 23:37:43 UTC |
| Sent | 7 Jul 2025, 23:37:47 UTC |
| Report deadline | 15 Jul 2025, 23:37:47 UTC |
| Received | 18 Jul 2025, 20:24:56 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10857191 |
| Run time | 9 days 18 hours 56 min 42 sec |
| CPU time | 12 days 11 hours 30 min 55 sec |
| Validate state | Task was reported too late to validate |
| Credit | 0.00 |
| Device peak FLOPS | 23.95 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 2.84 GB |
| Peak swap size | 29.19 GB |
| Peak disk usage | 2.38 GB |
Stderr output
<core_client_version>8.1.0</core_client_version>
<![CDATA[
<stderr_txt>
19:38:11 (3019908): wrapper (7.7.26015): starting
19:38:11 (3019908): wrapper: running run_atlas (--nthreads 8)
[2025-07-07 19:38:11] Arguments: --nthreads 8
[2025-07-07 19:38:11] Threads: 8
[2025-07-07 19:38:11] Checking for CVMFS
[2025-07-07 19:38:11] Probing /cvmfs/atlas.cern.ch... OK
[2025-07-07 19:38:11] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-07-07 19:38:11] Running cvmfs_config stat atlas.cern.ch
[2025-07-07 19:38:11] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-07-07 19:38:11] 2.12.6.0 3401378 16510 278016 148055 3 14 23791499 32935937 1534 16776704 0 82025753 99.732 54460692 30945 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-07-07 19:38:11] CVMFS is ok
[2025-07-07 19:38:11] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-07-07 19:38:11] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-07-07 19:38:11] Further information can be found at the LHC@home message board.
[2025-07-07 19:38:11] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-07-07 19:38:11] Checking for apptainer binary...
[2025-07-07 19:38:11] Using apptainer found in PATH at /usr/bin/apptainer
[2025-07-07 19:38:11] Running /usr/bin/apptainer --version
[2025-07-07 19:38:11] apptainer version 1.4.1-1.el9
[2025-07-07 19:38:11] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-07-07 19:38:12] c-8-1.aglt2.org
[2025-07-07 19:38:12] apptainer works
[2025-07-07 19:38:12] Set ATHENA_PROC_NUMBER=8
[2025-07-07 19:38:12] Set ATHENA_CORE_NUMBER=8
[2025-07-07 19:38:12] Starting ATLAS job with PandaID=6724307185
[2025-07-07 19:38:12] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
09:36:51 (4154): wrapper (7.7.26015): starting
09:36:51 (4154): wrapper: running run_atlas (--nthreads 8)
[2025-07-08 09:36:51] Arguments: --nthreads 8
[2025-07-08 09:36:51] Threads: 8
[2025-07-08 09:36:51] This job has been restarted, cleaning up previous attempt
[2025-07-08 09:36:51] Checking for CVMFS
[2025-07-08 09:36:52] Probing /cvmfs/atlas.cern.ch... OK
[2025-07-08 09:36:52] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-07-08 09:36:52] Running cvmfs_config stat atlas.cern.ch
[2025-07-08 09:36:53] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-07-08 09:36:53] 2.12.6.0 4471 0 30412 148076 3 1 23815809 32935937 0 16776704 0 0 0.000 1291 71722 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-07-08 09:36:53] CVMFS is ok
[2025-07-08 09:36:53] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-07-08 09:36:53] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-07-08 09:36:53] Further information can be found at the LHC@home message board.
[2025-07-08 09:36:53] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-07-08 09:36:53] Checking for apptainer binary...
[2025-07-08 09:36:53] Using apptainer found in PATH at /usr/bin/apptainer
[2025-07-08 09:36:53] Running /usr/bin/apptainer --version
[2025-07-08 09:36:53] apptainer version 1.4.1-1.el9
[2025-07-08 09:36:53] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-07-08 09:36:53] c-8-1.aglt2.org
[2025-07-08 09:36:53] apptainer works
[2025-07-08 09:36:53] Set ATHENA_PROC_NUMBER=8
[2025-07-08 09:36:53] Set ATHENA_CORE_NUMBER=8
[2025-07-08 09:36:53] Starting ATLAS job with PandaID=6724307185
[2025-07-08 09:36:53] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
[2025-07-08 09:39:59] Job failed
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + export RUNTIME_CONFIG_DIR=/tmp/boinchome/slots/0/
[2025-07-08 09:39:59] + RUNTIME_CONFIG_DIR=/tmp/boinchome/slots/0/
[2025-07-08 09:39:59] + mv ATLAS.root_0 EVNT.45482146._000029.pool.root.1
[2025-07-08 09:39:59] mv: ‘ATLAS.root_0’ and ‘EVNT.45482146._000029.pool.root.1’ are the same file
[2025-07-08 09:39:59] + tar --strip-components=5 -xvf input.tar.gz
[2025-07-08 09:39:59] + umask 077
[2025-07-08 09:39:59] ++ date +%s
[2025-07-08 09:39:59] + ACCOUNTING_STARTTIME=1751981813
[2025-07-08 09:39:59] + echo 'Detecting resource accounting method available for the job.'
[2025-07-08 09:39:59] Detecting resource accounting method available for the job.
[2025-07-08 09:39:59] + JOB_ACCOUNTING=
[2025-07-08 09:39:59] + command -v arc-job-cgroup
[2025-07-08 09:39:59] + '[' -z '' ']'
[2025-07-08 09:39:59] + GNU_TIME=/usr/bin/time
[2025-07-08 09:39:59] + echo 'Looking for /usr/bin/time tool for accounting measurements'
[2025-07-08 09:39:59] Looking for /usr/bin/time tool for accounting measurements
[2025-07-08 09:39:59] + '[' '!' -z /usr/bin/time ']'
[2025-07-08 09:39:59] + /usr/bin/time --version
[2025-07-08 09:39:59] + echo 'GNU time found and will be used for job accounting.'
[2025-07-08 09:39:59] GNU time found and will be used for job accounting.
[2025-07-08 09:39:59] + JOB_ACCOUNTING=gnutime
[2025-07-08 09:39:59] + '[' -z gnutime ']'
[2025-07-08 09:39:59] + export PANDA_JSID=harvester-CERN_central_ACTA
[2025-07-08 09:39:59] + PANDA_JSID=harvester-CERN_central_ACTA
[2025-07-08 09:39:59] + export GTAG=http://aipanda403.cern.ch/data/jobs/2025-07-07/BOINC_MCORE/6724307185.out
[2025-07-08 09:39:59] + GTAG=http://aipanda403.cern.ch/data/jobs/2025-07-07/BOINC_MCORE/6724307185.out
[2025-07-08 09:39:59] + export PILOT_NOKILL=YES
[2025-07-08 09:39:59] + PILOT_NOKILL=YES
[2025-07-08 09:39:59] + export GRID_GLOBAL_JOBID=zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-08 09:39:59] + GRID_GLOBAL_JOBID=zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-08 09:39:59] + export GRID_GLOBAL_JOBURL=https://arc-boinc-03.cern.ch:443/arex/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-08 09:39:59] + GRID_GLOBAL_JOBURL=https://arc-boinc-03.cern.ch:443/arex/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-08 09:39:59] + export GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest
[2025-07-08 09:39:59] + GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest
[2025-07-08 09:39:59] + export GRID_GLOBAL_JOBHOST=arc-boinc-03.cern.ch
[2025-07-08 09:39:59] + GRID_GLOBAL_JOBHOST=arc-boinc-03.cern.ch
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + RUNTIME_JOB_DIR=/tmp/boinchome/slots/0
[2025-07-08 09:39:59] + RUNTIME_JOB_STDIN=/dev/null
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + RUNTIME_JOB_STDOUT=/tmp/boinchome/slots/0/log.45482148._017145.job.log.1
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + RUNTIME_JOB_STDERR=/tmp/boinchome/slots/0/log.45482148._017145.job.log.1
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + RUNTIME_JOB_DIAG=/tmp/boinchome/slots/0.diag
[2025-07-08 09:39:59] + '[' '!' -z '' ']'
[2025-07-08 09:39:59] + RUNTIME_LOCAL_SCRATCH_DIR=
[2025-07-08 09:39:59] + RUNTIME_LOCAL_SCRATCH_MOVE_TOOL=mv
[2025-07-08 09:39:59] + RUNTIME_FRONTEND_SEES_NODE=
[2025-07-08 09:39:59] + RUNTIME_NODE_SEES_FRONTEND=yes
[2025-07-08 09:39:59] + '[' '!' -z '' ']'
[2025-07-08 09:39:59] + '[' -z yes ']'
[2025-07-08 09:39:59] ++ pwd
[2025-07-08 09:39:59] + RUNTIME_JOB_DIAG=/tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-08 09:39:59] + runtimeenvironments=
[2025-07-08 09:39:59] + runtimeenvironments='APPS/HEP/ATLAS-SITE;'
[2025-07-08 09:39:59] + RTE_function_0 1
[2025-07-08 09:39:59] + export PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-08 09:39:59] + PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-08 09:39:59] + export BOINC_APP=ATLAS
[2025-07-08 09:39:59] + BOINC_APP=ATLAS
[2025-07-08 09:39:59] + export WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-08 09:39:59] + WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-08 09:39:59] + export RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-08 09:39:59] + RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-08 09:39:59] + '[' -z 1 ']'
[2025-07-08 09:39:59] + '[' 1 == 1 ']'
[2025-07-08 09:39:59] + shift
[2025-07-08 09:39:59] + export ATLAS_LOCAL_AREA=/tmp/boinchome/slots/0/
[2025-07-08 09:39:59] + ATLAS_LOCAL_AREA=/tmp/boinchome/slots/0/
[2025-07-08 09:39:59] + '[' 0 -ne 0 ']'
[2025-07-08 09:39:59] + echo 'runtimeenvironments=APPS/HEP/ATLAS-SITE;'
[2025-07-08 09:39:59] + '[' -f /etc/os-release ']'
[2025-07-08 09:39:59] +++ cat /etc/os-release
[2025-07-08 09:39:59] ++ eval 'NAME="CentOS' 'Linux"' 'VERSION="7' '(Core)"' 'ID="centos"' 'ID_LIKE="rhel' 'fedora"' 'VERSION_ID="7"' 'PRETTY_NAME="CentOS' Linux 7 '(Core)"' 'ANSI_COLOR="0;31"' 'CPE_NAME="cpe:/o:centos:centos:7"' 'HOME_URL="https://www.centos.org/"' 'BUG_REPORT_URL="https://bugs.centos.org/"' 'CENTOS_MANTISBT_PROJECT="CentOS-7"' 'CENTOS_MANTISBT_PROJECT_VERSION="7"' 'REDHAT_SUPPORT_PRODUCT="centos"' 'REDHAT_SUPPORT_PRODUCT_VERSION="7"'
[2025-07-08 09:39:59] +++ NAME='CentOS Linux'
[2025-07-08 09:39:59] +++ VERSION='7 (Core)'
[2025-07-08 09:39:59] +++ ID=centos
[2025-07-08 09:39:59] +++ ID_LIKE='rhel fedora'
[2025-07-08 09:39:59] +++ VERSION_ID=7
[2025-07-08 09:39:59] +++ PRETTY_NAME='CentOS Linux 7 (Core)'
[2025-07-08 09:39:59] +++ ANSI_COLOR='0;31'
[2025-07-08 09:39:59] +++ CPE_NAME=cpe:/o:centos:centos:7
[2025-07-08 09:39:59] +++ HOME_URL=https://www.centos.org/
[2025-07-08 09:39:59] +++ BUG_REPORT_URL=https://bugs.centos.org/
[2025-07-08 09:39:59] +++ CENTOS_MANTISBT_PROJECT=CentOS-7
[2025-07-08 09:39:59] +++ CENTOS_MANTISBT_PROJECT_VERSION=7
[2025-07-08 09:39:59] +++ REDHAT_SUPPORT_PRODUCT=centos
[2025-07-08 09:39:59] +++ REDHAT_SUPPORT_PRODUCT_VERSION=7
[2025-07-08 09:39:59] ++ echo 'CentOS Linux 7 (Core)'
[2025-07-08 09:39:59] + SYSTEM_SOFTWARE='CentOS Linux 7 (Core)'
[2025-07-08 09:39:59] + '[' -n 'CentOS Linux 7 (Core)' ']'
[2025-07-08 09:39:59] + echo 'systemsoftware=CentOS Linux 7 (Core)'
[2025-07-08 09:39:59] + HOME=/tmp/boinchome/slots/0
[2025-07-08 09:39:59] + export HOME
[2025-07-08 09:39:59] + cd /tmp/boinchome/slots/0
[2025-07-08 09:39:59] + '[' '!' -z '' ']'
[2025-07-08 09:39:59] + '[' -z '' ']'
[2025-07-08 09:39:59] ++ /bin/hostname -f
[2025-07-08 09:39:59] + nodename=c-8-1.aglt2.org
[2025-07-08 09:39:59] + echo nodename=c-8-1.aglt2.org
[2025-07-08 09:39:59] + echo Processors=2
[2025-07-08 09:39:59] + echo Nodecount=1
[2025-07-08 09:39:59] + echo Benchmark=HEPSPEC:1.0
[2025-07-08 09:39:59] + executable=./runpilot2-wrapper.sh
[2025-07-08 09:39:59] + '[' '!' -f ./runpilot2-wrapper.sh ']'
[2025-07-08 09:39:59] ++ dd if=./runpilot2-wrapper.sh count=1
[2025-07-08 09:39:59] ++ head -n 1
[2025-07-08 09:39:59] ++ tr -d '\0'
[2025-07-08 09:39:59] + line1='#!/bin/bash'
[2025-07-08 09:39:59] ++ echo '#!/bin/bash'
[2025-07-08 09:39:59] ++ sed -n 's/^#! *//p'
[2025-07-08 09:39:59] + shebang=/bin/bash
[2025-07-08 09:39:59] ++ echo /bin/bash
[2025-07-08 09:39:59] ++ awk '{print $1}'
[2025-07-08 09:39:59] + interpreter=/bin/bash
[2025-07-08 09:39:59] + '[' /bin/bash = /usr/bin/env ']'
[2025-07-08 09:39:59] + '[' x/bin/bash = x ']'
[2025-07-08 09:39:59] + type /bin/bash
[2025-07-08 09:39:59] + '[' xgnutime = xgnutime ']'
[2025-07-08 09:39:59] + /usr/bin/time -o /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag -a -f 'WallTime=%es\nKernelTime=%Ss\nUserTime=%Us\nCPUUsage=%P\nMaxResidentMemory=%MkB\nAverageResidentMemory=%tkB\nAverageTotalMemory=%KkB\nAverageUnsharedMemory=%DkB\nAverageUnsharedStack=%pkB\nAverageSharedMemory=%XkB\nPageSize=%ZB\nMajorPageFaults=%F\nMinorPageFaults=%R\nSwaps=%W\nForcedSwitches=%c\nWaitSwitches=%w\nInputs=%I\nOutputs=%O\nSocketReceived=%r\nSocketSent=%s\nSignals=%k\n' ./runpilot2-wrapper.sh -q BOINC_MCORE -j managed --pilot-user ATLAS --harvester-submit-mode PUSH -w generic --job-type managed --resource-type MCORE --pilotversion 3.10.4.12 -z -t --piloturl local --mute --container
[2025-07-08 09:39:59] + RESULT=2
[2025-07-08 09:39:59] + sed -i -e /nodename=/d /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-08 09:39:59] ++ grep domain_name init_data.xml
[2025-07-08 09:39:59] ++ awk -F '>' '{print $2}'
[2025-07-08 09:39:59] ++ awk -F '<' '{print $1}'
[2025-07-08 09:39:59] ++ sed -e 's# #_#g'
[2025-07-08 09:39:59] + hostname=c-8-1.aglt2.org
[2025-07-08 09:39:59] ++ grep user_name init_data.xml
[2025-07-08 09:39:59] ++ awk -F '>' '{print $2}'
[2025-07-08 09:39:59] ++ awk -F '<' '{print $1}'
[2025-07-08 09:39:59] ++ sed -e 's# #_#g'
[2025-07-08 09:39:59] + username=AGLT2
[2025-07-08 09:39:59] + nodename=AGLT2@c-8-1.aglt2.org
[2025-07-08 09:39:59] + echo nodename=AGLT2@c-8-1.aglt2.org
[2025-07-08 09:39:59] + '[' -n 8 ']'
[2025-07-08 09:39:59] + sed -i -e s/Processors=1/Processors=8/ /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-08 09:39:59] + echo exitcode=2
[2025-07-08 09:39:59] + runtimeenvironments=
[2025-07-08 09:39:59] + runtimeenvironments='APPS/HEP/ATLAS-SITE;'
[2025-07-08 09:39:59] + RTE_function_0 2
[2025-07-08 09:39:59] + export PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-08 09:39:59] + PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-08 09:39:59] + export BOINC_APP=ATLAS
[2025-07-08 09:39:59] + BOINC_APP=ATLAS
[2025-07-08 09:39:59] + export WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-08 09:39:59] + WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-08 09:39:59] + export RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-08 09:39:59] + RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-08 09:39:59] + '[' -z 2 ']'
[2025-07-08 09:39:59] + '[' 2 == 1 ']'
[2025-07-08 09:39:59] + '[' 0 -ne 0 ']'
[2025-07-08 09:39:59] + echo 'zip all output files'
[2025-07-08 09:39:59] + flist='*.diag '
[2025-07-08 09:39:59] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-08 09:39:59] + '[' -e ./heartbeat.json ']'
[2025-07-08 09:39:59] + flist='*.diag ./heartbeat.json'
[2025-07-08 09:39:59] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-08 09:39:59] + '[' -e ./output.list ']'
[2025-07-08 09:39:59] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-08 09:39:59] + '[' -e ./log.45482148._017145.job.log.1 ']'
[2025-07-08 09:39:59] + flist='*.diag ./heartbeat.json ./log.45482148._017145.job.log.1'
[2025-07-08 09:39:59] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-08 09:39:59] + '[' -e ./log.45482148._017145.job.log.1 ']'
[2025-07-08 09:39:59] + flist='*.diag ./heartbeat.json ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1'
[2025-07-08 09:39:59] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-08 09:39:59] + '[' -e ./gmlog/ ']'
[2025-07-08 09:39:59] + '[' -f output.list ']'
[2025-07-08 09:39:59] + tar cvf result.tar.gz zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag ./heartbeat.json ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1
[2025-07-08 09:39:59] + '[' xgnutime = xcgroup ']'
[2025-07-08 09:39:59] + '[' -n '' ']'
[2025-07-08 09:39:59] + '[' -n '' ']'
[2025-07-08 09:39:59] ++ date +%s
[2025-07-08 09:39:59] + ACCOUNTING_ENDTIME=1751981999
[2025-07-08 09:39:59] ++ date -d '1970-01-01 UTC 1751981813 seconds' +%Y%m%d%H%M%SZ
[2025-07-08 09:39:59] + echo LRMSStartTime=20250708093653Z
[2025-07-08 09:39:59] ++ date -d '1970-01-01 UTC 1751981999 seconds' +%Y%m%d%H%M%SZ
[2025-07-08 09:39:59] + echo LRMSEndTime=20250708093959Z
[2025-07-08 09:39:59] + echo walltime=186
[2025-07-08 09:39:59] + echo exitcode=2
[2025-07-08 09:39:59] + exit 2
[2025-07-08 09:39:59] ./PanDA_Pilot-6724307185/log.EVNTtoHITS
[2025-07-08 09:39:59] ./PanDA_Pilot-6724307185/pilotlog.txt
[2025-07-08 09:39:59] ./PanDA_Pilot-6724307185/prmon.log
[2025-07-08 09:39:59] ./PanDA_Pilot-6724307185/PoolFileCatalog.xml
[2025-07-08 09:39:59] ./pilotlog.txt
[2025-07-08 09:39:59] ./runtime_log
[2025-07-08 09:39:59] ./pilot3/pilot/util/__pycache__/loggingsupport.cpython-39.pyc
[2025-07-08 09:39:59] ./pilot3/pilot/util/__pycache__/realtimelogger.cpython-39.pyc
[2025-07-08 09:39:59] ./pilot3/pilot/util/__pycache__/lokirealtimelogger.cpython-39.pyc
[2025-07-08 09:39:59] ./pilot3/pilot/util/lokirealtimelogger.py
[2025-07-08 09:39:59] ./pilot3/pilot/util/realtimelogger.py
[2025-07-08 09:39:59] ./pilot3/pilot/util/loggingsupport.py
[2025-07-08 09:39:59] ./log.45482148._017145.job.log.1
[2025-07-08 09:39:59] ./runtime_log.err
09:49:11 (65406): wrapper (7.7.26015): starting
09:49:11 (65406): wrapper: running run_atlas (--nthreads 8)
[2025-07-08 09:49:11] Arguments: --nthreads 8
[2025-07-08 09:49:11] Threads: 8
[2025-07-08 09:49:11] This job has been restarted, cleaning up previous attempt
[2025-07-08 09:49:11] Checking for CVMFS
[2025-07-08 09:49:11] Probing /cvmfs/atlas.cern.ch... OK
[2025-07-08 09:49:11] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-07-08 09:49:11] Running cvmfs_config stat atlas.cern.ch
[2025-07-08 09:49:11] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-07-08 09:49:11] 2.12.6.0 4471 12 52180 148076 3 187 23836622 32935936 0 16776704 0 148579 99.989 2781 56755 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-07-08 09:49:11] CVMFS is ok
[2025-07-08 09:49:11] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-07-08 09:49:11] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-07-08 09:49:11] Further information can be found at the LHC@home message board.
[2025-07-08 09:49:11] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-07-08 09:49:11] Checking for apptainer binary...
[2025-07-08 09:49:11] Using apptainer found in PATH at /usr/bin/apptainer
[2025-07-08 09:49:11] Running /usr/bin/apptainer --version
[2025-07-08 09:49:11] apptainer version 1.4.1-1.el9
[2025-07-08 09:49:11] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-07-08 09:49:11] c-8-1.aglt2.org
[2025-07-08 09:49:11] apptainer works
[2025-07-08 09:49:11] Set ATHENA_PROC_NUMBER=8
[2025-07-08 09:49:11] Set ATHENA_CORE_NUMBER=8
[2025-07-08 09:49:11] Starting ATLAS job with PandaID=6724307185
[2025-07-08 09:49:11] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
06:29:41 (3691104): wrapper (7.7.26015): starting
06:29:41 (3691104): wrapper: running run_atlas (--nthreads 10)
[2025-07-18 06:29:41] Arguments: --nthreads 10
[2025-07-18 06:29:41] Threads: 10
[2025-07-18 06:29:41] This job has been restarted, cleaning up previous attempt
[2025-07-18 06:29:41] Checking for CVMFS
[2025-07-18 06:29:41] Probing /cvmfs/atlas.cern.ch... OK
[2025-07-18 06:29:41] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-07-18 06:29:41] Running cvmfs_config stat atlas.cern.ch
[2025-07-18 06:29:42] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-07-18 06:29:42] 2.13.0.0 4177459 5303 217384 148468 0 253 28409790 32935936 16616 16776704 0 26361101 99.606 31449597 30477 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-07-18 06:29:42] CVMFS is ok
[2025-07-18 06:29:42] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-07-18 06:29:42] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-07-18 06:29:42] Further information can be found at the LHC@home message board.
[2025-07-18 06:29:42] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-07-18 06:29:42] Checking for apptainer binary...
[2025-07-18 06:29:42] Using apptainer found in PATH at /usr/bin/apptainer
[2025-07-18 06:29:42] Running /usr/bin/apptainer --version
[2025-07-18 06:29:43] apptainer version 1.4.1-1.el9
[2025-07-18 06:29:43] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-07-18 06:29:44] c-8-1.aglt2.org
[2025-07-18 06:29:44] apptainer works
[2025-07-18 06:29:44] Set ATHENA_PROC_NUMBER=10
[2025-07-18 06:29:44] Set ATHENA_CORE_NUMBER=10
[2025-07-18 06:29:44] Starting ATLAS job with PandaID=6724307185
[2025-07-18 06:29:44] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
[2025-07-18 06:33:19] Job failed
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + export RUNTIME_CONFIG_DIR=/tmp/boinchome/slots/0/
[2025-07-18 06:33:19] + RUNTIME_CONFIG_DIR=/tmp/boinchome/slots/0/
[2025-07-18 06:33:19] + mv ATLAS.root_0 EVNT.45482146._000029.pool.root.1
[2025-07-18 06:33:19] mv: ‘ATLAS.root_0’ and ‘EVNT.45482146._000029.pool.root.1’ are the same file
[2025-07-18 06:33:19] + tar --strip-components=5 -xvf input.tar.gz
[2025-07-18 06:33:19] + umask 077
[2025-07-18 06:33:19] ++ date +%s
[2025-07-18 06:33:19] + ACCOUNTING_STARTTIME=1752834585
[2025-07-18 06:33:19] + echo 'Detecting resource accounting method available for the job.'
[2025-07-18 06:33:19] Detecting resource accounting method available for the job.
[2025-07-18 06:33:19] + JOB_ACCOUNTING=
[2025-07-18 06:33:19] + command -v arc-job-cgroup
[2025-07-18 06:33:19] + '[' -z '' ']'
[2025-07-18 06:33:19] + GNU_TIME=/usr/bin/time
[2025-07-18 06:33:19] + echo 'Looking for /usr/bin/time tool for accounting measurements'
[2025-07-18 06:33:19] Looking for /usr/bin/time tool for accounting measurements
[2025-07-18 06:33:19] + '[' '!' -z /usr/bin/time ']'
[2025-07-18 06:33:19] + /usr/bin/time --version
[2025-07-18 06:33:19] + echo 'GNU time found and will be used for job accounting.'
[2025-07-18 06:33:19] GNU time found and will be used for job accounting.
[2025-07-18 06:33:19] + JOB_ACCOUNTING=gnutime
[2025-07-18 06:33:19] + '[' -z gnutime ']'
[2025-07-18 06:33:19] + export PANDA_JSID=harvester-CERN_central_ACTA
[2025-07-18 06:33:19] + PANDA_JSID=harvester-CERN_central_ACTA
[2025-07-18 06:33:19] + export GTAG=http://aipanda403.cern.ch/data/jobs/2025-07-07/BOINC_MCORE/6724307185.out
[2025-07-18 06:33:19] + GTAG=http://aipanda403.cern.ch/data/jobs/2025-07-07/BOINC_MCORE/6724307185.out
[2025-07-18 06:33:19] + export PILOT_NOKILL=YES
[2025-07-18 06:33:19] + PILOT_NOKILL=YES
[2025-07-18 06:33:19] + export GRID_GLOBAL_JOBID=zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-18 06:33:19] + GRID_GLOBAL_JOBID=zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-18 06:33:19] + export GRID_GLOBAL_JOBURL=https://arc-boinc-03.cern.ch:443/arex/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-18 06:33:19] + GRID_GLOBAL_JOBURL=https://arc-boinc-03.cern.ch:443/arex/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm
[2025-07-18 06:33:19] + export GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest
[2025-07-18 06:33:19] + GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest
[2025-07-18 06:33:19] + export GRID_GLOBAL_JOBHOST=arc-boinc-03.cern.ch
[2025-07-18 06:33:19] + GRID_GLOBAL_JOBHOST=arc-boinc-03.cern.ch
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + RUNTIME_JOB_DIR=/tmp/boinchome/slots/0
[2025-07-18 06:33:19] + RUNTIME_JOB_STDIN=/dev/null
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + RUNTIME_JOB_STDOUT=/tmp/boinchome/slots/0/log.45482148._017145.job.log.1
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + RUNTIME_JOB_STDERR=/tmp/boinchome/slots/0/log.45482148._017145.job.log.1
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + RUNTIME_JOB_DIAG=/tmp/boinchome/slots/0.diag
[2025-07-18 06:33:19] + '[' '!' -z '' ']'
[2025-07-18 06:33:19] + RUNTIME_LOCAL_SCRATCH_DIR=
[2025-07-18 06:33:19] + RUNTIME_LOCAL_SCRATCH_MOVE_TOOL=mv
[2025-07-18 06:33:19] + RUNTIME_FRONTEND_SEES_NODE=
[2025-07-18 06:33:19] + RUNTIME_NODE_SEES_FRONTEND=yes
[2025-07-18 06:33:19] + '[' '!' -z '' ']'
[2025-07-18 06:33:19] + '[' -z yes ']'
[2025-07-18 06:33:19] ++ pwd
[2025-07-18 06:33:19] + RUNTIME_JOB_DIAG=/tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-18 06:33:19] + runtimeenvironments=
[2025-07-18 06:33:19] + runtimeenvironments='APPS/HEP/ATLAS-SITE;'
[2025-07-18 06:33:19] + RTE_function_0 1
[2025-07-18 06:33:19] + export PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-18 06:33:19] + PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-18 06:33:19] + export BOINC_APP=ATLAS
[2025-07-18 06:33:19] + BOINC_APP=ATLAS
[2025-07-18 06:33:19] + export WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-18 06:33:19] + WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-18 06:33:19] + export RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-18 06:33:19] + RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-18 06:33:19] + '[' -z 1 ']'
[2025-07-18 06:33:19] + '[' 1 == 1 ']'
[2025-07-18 06:33:19] + shift
[2025-07-18 06:33:19] + export ATLAS_LOCAL_AREA=/tmp/boinchome/slots/0/
[2025-07-18 06:33:19] + ATLAS_LOCAL_AREA=/tmp/boinchome/slots/0/
[2025-07-18 06:33:19] + '[' 0 -ne 0 ']'
[2025-07-18 06:33:19] + echo 'runtimeenvironments=APPS/HEP/ATLAS-SITE;'
[2025-07-18 06:33:19] + '[' -f /etc/os-release ']'
[2025-07-18 06:33:19] +++ cat /etc/os-release
[2025-07-18 06:33:19] ++ eval 'NAME="CentOS' 'Linux"' 'VERSION="7' '(Core)"' 'ID="centos"' 'ID_LIKE="rhel' 'fedora"' 'VERSION_ID="7"' 'PRETTY_NAME="CentOS' Linux 7 '(Core)"' 'ANSI_COLOR="0;31"' 'CPE_NAME="cpe:/o:centos:centos:7"' 'HOME_URL="https://www.centos.org/"' 'BUG_REPORT_URL="https://bugs.centos.org/"' 'CENTOS_MANTISBT_PROJECT="CentOS-7"' 'CENTOS_MANTISBT_PROJECT_VERSION="7"' 'REDHAT_SUPPORT_PRODUCT="centos"' 'REDHAT_SUPPORT_PRODUCT_VERSION="7"'
[2025-07-18 06:33:19] +++ NAME='CentOS Linux'
[2025-07-18 06:33:19] +++ VERSION='7 (Core)'
[2025-07-18 06:33:19] +++ ID=centos
[2025-07-18 06:33:19] +++ ID_LIKE='rhel fedora'
[2025-07-18 06:33:19] +++ VERSION_ID=7
[2025-07-18 06:33:19] +++ PRETTY_NAME='CentOS Linux 7 (Core)'
[2025-07-18 06:33:19] +++ ANSI_COLOR='0;31'
[2025-07-18 06:33:19] +++ CPE_NAME=cpe:/o:centos:centos:7
[2025-07-18 06:33:19] +++ HOME_URL=https://www.centos.org/
[2025-07-18 06:33:19] +++ BUG_REPORT_URL=https://bugs.centos.org/
[2025-07-18 06:33:19] +++ CENTOS_MANTISBT_PROJECT=CentOS-7
[2025-07-18 06:33:19] +++ CENTOS_MANTISBT_PROJECT_VERSION=7
[2025-07-18 06:33:19] +++ REDHAT_SUPPORT_PRODUCT=centos
[2025-07-18 06:33:19] +++ REDHAT_SUPPORT_PRODUCT_VERSION=7
[2025-07-18 06:33:19] ++ echo 'CentOS Linux 7 (Core)'
[2025-07-18 06:33:19] + SYSTEM_SOFTWARE='CentOS Linux 7 (Core)'
[2025-07-18 06:33:19] + '[' -n 'CentOS Linux 7 (Core)' ']'
[2025-07-18 06:33:19] + echo 'systemsoftware=CentOS Linux 7 (Core)'
[2025-07-18 06:33:19] + HOME=/tmp/boinchome/slots/0
[2025-07-18 06:33:19] + export HOME
[2025-07-18 06:33:19] + cd /tmp/boinchome/slots/0
[2025-07-18 06:33:19] + '[' '!' -z '' ']'
[2025-07-18 06:33:19] + '[' -z '' ']'
[2025-07-18 06:33:19] ++ /bin/hostname -f
[2025-07-18 06:33:19] + nodename=c-8-1.aglt2.org
[2025-07-18 06:33:19] + echo nodename=c-8-1.aglt2.org
[2025-07-18 06:33:19] + echo Processors=2
[2025-07-18 06:33:19] + echo Nodecount=1
[2025-07-18 06:33:19] + echo Benchmark=HEPSPEC:1.0
[2025-07-18 06:33:19] + executable=./runpilot2-wrapper.sh
[2025-07-18 06:33:19] + '[' '!' -f ./runpilot2-wrapper.sh ']'
[2025-07-18 06:33:19] ++ head -n 1
[2025-07-18 06:33:19] ++ dd if=./runpilot2-wrapper.sh count=1
[2025-07-18 06:33:19] ++ tr -d '\0'
[2025-07-18 06:33:19] + line1='#!/bin/bash'
[2025-07-18 06:33:19] ++ echo '#!/bin/bash'
[2025-07-18 06:33:19] ++ sed -n 's/^#! *//p'
[2025-07-18 06:33:19] + shebang=/bin/bash
[2025-07-18 06:33:19] ++ echo /bin/bash
[2025-07-18 06:33:19] ++ awk '{print $1}'
[2025-07-18 06:33:19] + interpreter=/bin/bash
[2025-07-18 06:33:19] + '[' /bin/bash = /usr/bin/env ']'
[2025-07-18 06:33:19] + '[' x/bin/bash = x ']'
[2025-07-18 06:33:19] + type /bin/bash
[2025-07-18 06:33:19] + '[' xgnutime = xgnutime ']'
[2025-07-18 06:33:19] + /usr/bin/time -o /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag -a -f 'WallTime=%es\nKernelTime=%Ss\nUserTime=%Us\nCPUUsage=%P\nMaxResidentMemory=%MkB\nAverageResidentMemory=%tkB\nAverageTotalMemory=%KkB\nAverageUnsharedMemory=%DkB\nAverageUnsharedStack=%pkB\nAverageSharedMemory=%XkB\nPageSize=%ZB\nMajorPageFaults=%F\nMinorPageFaults=%R\nSwaps=%W\nForcedSwitches=%c\nWaitSwitches=%w\nInputs=%I\nOutputs=%O\nSocketReceived=%r\nSocketSent=%s\nSignals=%k\n' ./runpilot2-wrapper.sh -q BOINC_MCORE -j managed --pilot-user ATLAS --harvester-submit-mode PUSH -w generic --job-type managed --resource-type MCORE --pilotversion 3.10.4.12 -z -t --piloturl local --mute --container
[2025-07-18 06:33:19] + RESULT=2
[2025-07-18 06:33:19] + sed -i -e /nodename=/d /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-18 06:33:19] ++ grep domain_name init_data.xml
[2025-07-18 06:33:19] ++ awk -F '<' '{print $1}'
[2025-07-18 06:33:19] ++ awk -F '>' '{print $2}'
[2025-07-18 06:33:19] ++ sed -e 's# #_#g'
[2025-07-18 06:33:19] + hostname=c-8-1.aglt2.org
[2025-07-18 06:33:19] ++ awk -F '>' '{print $2}'
[2025-07-18 06:33:19] ++ awk -F '<' '{print $1}'
[2025-07-18 06:33:19] ++ sed -e 's# #_#g'
[2025-07-18 06:33:19] ++ grep user_name init_data.xml
[2025-07-18 06:33:19] + username=AGLT2
[2025-07-18 06:33:19] + nodename=AGLT2@c-8-1.aglt2.org
[2025-07-18 06:33:19] + echo nodename=AGLT2@c-8-1.aglt2.org
[2025-07-18 06:33:19] + '[' -n 10 ']'
[2025-07-18 06:33:19] + sed -i -e s/Processors=1/Processors=10/ /tmp/boinchome/slots/0/zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-18 06:33:19] + echo exitcode=2
[2025-07-18 06:33:19] + runtimeenvironments=
[2025-07-18 06:33:19] + runtimeenvironments='APPS/HEP/ATLAS-SITE;'
[2025-07-18 06:33:19] + RTE_function_0 2
[2025-07-18 06:33:19] + export PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-18 06:33:19] + PROJECT_ROOT=/boincdata/boinc/project/lhcathome
[2025-07-18 06:33:19] + export BOINC_APP=ATLAS
[2025-07-18 06:33:19] + BOINC_APP=ATLAS
[2025-07-18 06:33:19] + export WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-18 06:33:19] + WU_TEMPLATE=templates/ATLAS_IN_DYNAMIC
[2025-07-18 06:33:19] + export RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-18 06:33:19] + RESULT_TEMPLATE=templates/ATLAS_OUT_2
[2025-07-18 06:33:19] + '[' -z 2 ']'
[2025-07-18 06:33:19] + '[' 2 == 1 ']'
[2025-07-18 06:33:19] + '[' 0 -ne 0 ']'
[2025-07-18 06:33:19] + echo 'zip all output files'
[2025-07-18 06:33:19] + flist='*.diag '
[2025-07-18 06:33:19] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-18 06:33:19] + '[' -e ./heartbeat.json ']'
[2025-07-18 06:33:19] + flist='*.diag ./heartbeat.json'
[2025-07-18 06:33:19] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-18 06:33:19] + '[' -e ./output.list ']'
[2025-07-18 06:33:19] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-18 06:33:19] + '[' -e ./log.45482148._017145.job.log.1 ']'
[2025-07-18 06:33:19] + flist='*.diag ./heartbeat.json ./log.45482148._017145.job.log.1'
[2025-07-18 06:33:19] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-18 06:33:19] + '[' -e ./log.45482148._017145.job.log.1 ']'
[2025-07-18 06:33:19] + flist='*.diag ./heartbeat.json ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1'
[2025-07-18 06:33:19] + for f in ./heartbeat.json ./output.list ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1 ./gmlog/
[2025-07-18 06:33:19] + '[' -e ./gmlog/ ']'
[2025-07-18 06:33:19] + '[' -f output.list ']'
[2025-07-18 06:33:19] + tar cvf result.tar.gz zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag ./heartbeat.json ./log.45482148._017145.job.log.1 ./log.45482148._017145.job.log.1
[2025-07-18 06:33:19] + '[' xgnutime = xcgroup ']'
[2025-07-18 06:33:19] + '[' -n '' ']'
[2025-07-18 06:33:19] + '[' -n '' ']'
[2025-07-18 06:33:19] ++ date +%s
[2025-07-18 06:33:19] + ACCOUNTING_ENDTIME=1752834799
[2025-07-18 06:33:19] ++ date -d '1970-01-01 UTC 1752834585 seconds' +%Y%m%d%H%M%SZ
[2025-07-18 06:33:19] + echo LRMSStartTime=20250718062945Z
[2025-07-18 06:33:19] ++ date -d '1970-01-01 UTC 1752834799 seconds' +%Y%m%d%H%M%SZ
[2025-07-18 06:33:19] + echo LRMSEndTime=20250718063319Z
[2025-07-18 06:33:19] + echo walltime=214
[2025-07-18 06:33:19] + echo exitcode=2
[2025-07-18 06:33:19] + exit 2
[2025-07-18 06:33:19] ./PanDA_Pilot-6724307185/log.EVNTtoHITS
[2025-07-18 06:33:19] ./PanDA_Pilot-6724307185/pilotlog.txt
[2025-07-18 06:33:19] ./PanDA_Pilot-6724307185/prmon.log
[2025-07-18 06:33:19] ./PanDA_Pilot-6724307185/PoolFileCatalog.xml
[2025-07-18 06:33:19] ./pilotlog.txt
[2025-07-18 06:33:19] ./runtime_log
[2025-07-18 06:33:19] ./pilot3/pilot/util/__pycache__/loggingsupport.cpython-39.pyc
[2025-07-18 06:33:19] ./pilot3/pilot/util/__pycache__/realtimelogger.cpython-39.pyc
[2025-07-18 06:33:19] ./pilot3/pilot/util/__pycache__/lokirealtimelogger.cpython-39.pyc
[2025-07-18 06:33:19] ./pilot3/pilot/util/lokirealtimelogger.py
[2025-07-18 06:33:19] ./pilot3/pilot/util/realtimelogger.py
[2025-07-18 06:33:19] ./pilot3/pilot/util/loggingsupport.py
[2025-07-18 06:33:19] ./log.45482148._017145.job.log.1
[2025-07-18 06:33:19] ./runtime_log.err
06:46:25 (3757494): wrapper (7.7.26015): starting
06:46:25 (3757494): wrapper: running run_atlas (--nthreads 10)
[2025-07-18 06:46:25] Arguments: --nthreads 10
[2025-07-18 06:46:25] Threads: 10
[2025-07-18 06:46:25] This job has been restarted, cleaning up previous attempt
[2025-07-18 06:46:25] Checking for CVMFS
[2025-07-18 06:46:26] Probing /cvmfs/atlas.cern.ch... OK
[2025-07-18 06:46:26] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-07-18 06:46:26] Running cvmfs_config stat atlas.cern.ch
[2025-07-18 06:46:26] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-07-18 06:46:26] 2.13.0.0 4177459 5319 222088 148468 3 423 28924685 32935936 16329 16776704 0 26426881 99.595 31632756 30190 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-07-18 06:46:26] CVMFS is ok
[2025-07-18 06:46:26] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-07-18 06:46:26] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-07-18 06:46:26] Further information can be found at the LHC@home message board.
[2025-07-18 06:46:26] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-07-18 06:46:26] Checking for apptainer binary...
[2025-07-18 06:46:26] Using apptainer found in PATH at /usr/bin/apptainer
[2025-07-18 06:46:26] Running /usr/bin/apptainer --version
[2025-07-18 06:46:27] apptainer version 1.4.1-1.el9
[2025-07-18 06:46:27] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-07-18 06:46:27] c-8-1.aglt2.org
[2025-07-18 06:46:27] apptainer works
[2025-07-18 06:46:27] Set ATHENA_PROC_NUMBER=10
[2025-07-18 06:46:27] Set ATHENA_CORE_NUMBER=10
[2025-07-18 06:46:27] Starting ATLAS job with PandaID=6724307185
[2025-07-18 06:46:27] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
[2025-07-18 16:18:22] *** The last 200 lines of the pilot log: ***
[2025-07-18 16:18:22] 2025-07-18 20:17:43,755 | INFO | PID=3762839 has CPU usage=5.0% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i P
[2025-07-18 16:18:22] 2025-07-18 20:17:43,755 | INFO | .. there are 37 such processes running
[2025-07-18 16:18:22] 2025-07-18 20:17:45,761 | INFO | monitor loop #1725: job 0:6724307185 is in state 'finished'
[2025-07-18 16:18:22] 2025-07-18 20:17:45,762 | INFO | will abort job monitoring soon since job state=finished (job is still in queue)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,710 | INFO | CPU arch script returned: x86-64-v3
[2025-07-18 16:18:22] 2025-07-18 20:17:47,727 | INFO | found 20 cores (10 cores per socket, 2 sockets) HT, CPU MHz: 2600.1801749999977
[2025-07-18 16:18:22] 2025-07-18 20:17:47,737 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6724307185/memory_monitor_summary.json (trf name=prmon)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,742 | INFO | extracted standard info from prmon json
[2025-07-18 16:18:22] 2025-07-18 20:17:47,742 | INFO | extracted standard memory fields from prmon json
[2025-07-18 16:18:22] 2025-07-18 20:17:47,742 | WARNING | GPU info not found in prmon json: 'gpu'
[2025-07-18 16:18:22] 2025-07-18 20:17:47,753 | WARNING | format EVNTtoHITS has no such key: dbData
[2025-07-18 16:18:22] 2025-07-18 20:17:47,753 | WARNING | format EVNTtoHITS has no such key: dbTime
[2025-07-18 16:18:22] 2025-07-18 20:17:47,799 | INFO | fitting pss+swap vs Time
[2025-07-18 16:18:22] 2025-07-18 20:17:47,800 | INFO | sum of square deviations: 54452222213.93559
[2025-07-18 16:18:22] 2025-07-18 20:17:47,837 | INFO | sum of deviations: -411528928444.6714
[2025-07-18 16:18:22] 2025-07-18 20:17:47,837 | INFO | mean x: 1752852767.5607142
[2025-07-18 16:18:22] 2025-07-18 20:17:47,837 | INFO | mean y: 2431524.0785714285
[2025-07-18 16:18:22] 2025-07-18 20:17:47,838 | INFO | -- intersect: 13249817798.259447
[2025-07-18 16:18:22] 2025-07-18 20:17:47,838 | INFO | intersect: 13249817798.259447
[2025-07-18 16:18:22] 2025-07-18 20:17:47,838 | INFO | chi2: 5.1267262425628255
[2025-07-18 16:18:22] 2025-07-18 20:17:47,844 | INFO | sum of square deviations: 53006757668.25945
[2025-07-18 16:18:22] 2025-07-18 20:17:47,869 | INFO | sum of deviations: -348071155280.8431
[2025-07-18 16:18:22] 2025-07-18 20:17:47,869 | INFO | mean x: 1752852615.0702703
[2025-07-18 16:18:22] 2025-07-18 20:17:47,869 | INFO | mean y: 2438196.354954955
[2025-07-18 16:18:22] 2025-07-18 20:17:47,869 | INFO | -- intersect: 11512620324.131002
[2025-07-18 16:18:22] 2025-07-18 20:17:47,869 | INFO | intersect: 11512620324.131002
[2025-07-18 16:18:22] 2025-07-18 20:17:47,870 | INFO | chi2: 4.382885920995738
[2025-07-18 16:18:22] 2025-07-18 20:17:47,870 | INFO | current chi2=4.382885920995738 (change=14.509070435468486 %)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,870 | INFO | right removable region: 554
[2025-07-18 16:18:22] 2025-07-18 20:17:47,871 | INFO | sum of square deviations: 53006395328.259445
[2025-07-18 16:18:22] 2025-07-18 20:17:47,886 | INFO | sum of deviations: -524941656034.98346
[2025-07-18 16:18:22] 2025-07-18 20:17:47,886 | INFO | mean x: 1752852920.0702703
[2025-07-18 16:18:22] 2025-07-18 20:17:47,886 | INFO | mean y: 2443461.0252252254
[2025-07-18 16:18:22] 2025-07-18 20:17:47,886 | INFO | -- intersect: 17361584918.374702
[2025-07-18 16:18:22] 2025-07-18 20:17:47,886 | INFO | intersect: 17361584918.374702
[2025-07-18 16:18:22] 2025-07-18 20:17:47,887 | INFO | chi2: 3.118178435688166
[2025-07-18 16:18:22] 2025-07-18 20:17:47,887 | INFO | current chi2=3.118178435688166 (change=39.177980485858676 %)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,888 | INFO | sum of square deviations: 51586382879.979744
[2025-07-18 16:18:22] 2025-07-18 20:17:47,905 | INFO | sum of deviations: -558931919786.8999
[2025-07-18 16:18:22] 2025-07-18 20:17:47,905 | INFO | mean x: 1752853072.58
[2025-07-18 16:18:22] 2025-07-18 20:17:47,905 | INFO | mean y: 2447107.3727272726
[2025-07-18 16:18:22] 2025-07-18 20:17:47,905 | INFO | -- intersect: 18994387969.768974
[2025-07-18 16:18:22] 2025-07-18 20:17:47,905 | INFO | intersect: 18994387969.768974
[2025-07-18 16:18:22] 2025-07-18 20:17:47,906 | INFO | chi2: 2.8506367694918766
[2025-07-18 16:18:22] 2025-07-18 20:17:47,906 | INFO | current chi2=2.8506367694918766 (change=8.580062742215848 %)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,906 | INFO | left removable region: 20
[2025-07-18 16:18:22] 2025-07-18 20:17:47,907 | INFO | sum of square deviations: 47213674238.03922
[2025-07-18 16:18:22] 2025-07-18 20:17:47,922 | INFO | sum of deviations: -518846385080.5615
[2025-07-18 16:18:22] 2025-07-18 20:17:47,922 | INFO | mean x: 1752853194.6011236
[2025-07-18 16:18:22] 2025-07-18 20:17:47,922 | INFO | mean y: 2457336.936329588
[2025-07-18 16:18:22] 2025-07-18 20:17:47,922 | INFO | -- intersect: 19265129820.556973
[2025-07-18 16:18:22] 2025-07-18 20:17:47,922 | INFO | intersect: 19265129820.556973
[2025-07-18 16:18:22] 2025-07-18 20:17:47,923 | INFO | chi2: 1.9164641176821955
[2025-07-18 16:18:22] 2025-07-18 20:17:47,923 | INFO | -- intersect: 19265129820.556973
[2025-07-18 16:18:22] 2025-07-18 20:17:47,924 | INFO | current memory leak: -10.99 B/s (using 534 data points, chi2=1.92)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,924 | INFO | could have reported an average CPU frequency of 2600 MHz (22 samples)
[2025-07-18 16:18:22] 2025-07-18 20:17:47,928 | INFO | ..............................
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . Timing measurements:
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . get job = 0 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . initial setup = 1 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . payload setup = 26 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . stage-in = 0 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . payload execution = 34135 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . stage-out = 5 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | . log creation = 1 s
[2025-07-18 16:18:22] 2025-07-18 20:17:47,929 | INFO | ..............................
[2025-07-18 16:18:22] 2025-07-18 20:17:48,193 | INFO |
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | job summary report
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | --------------------------------------------------
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | PanDA job id: 6724307185
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | task id: 45482148
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | errors: (none)
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | status: LOG_TRANSFER = DONE
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | pilot state: finished
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | transexitcode: 0
[2025-07-18 16:18:22] 2025-07-18 20:17:48,194 | INFO | exeerrorcode: 0
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | exeerrordiag:
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | exitcode: 0
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | exitmsg: OK
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | cpuconsumptiontime: 71612 s
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | nevents: 400
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | neventsw: 0
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | pid: 3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | pgrp: 3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | corecount: 10
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | event service: False
[2025-07-18 16:18:22] 2025-07-18 20:17:48,195 | INFO | sizes: {0: 2397289, 1: 2397488, 12: 2397488, 15: 2397778, 23: 2397806, 34166: 2437365, 34167: 2437492, 34174: 2446546, 34177: 2446602, 34203: 2446796}
[2025-07-18 16:18:22] 2025-07-18 20:17:48,196 | INFO | --------------------------------------------------
[2025-07-18 16:18:22] 2025-07-18 20:17:48,196 | INFO |
[2025-07-18 16:18:22] 2025-07-18 20:17:48,196 | INFO | executing command: ls -lF /tmp/boinchome/slots/0
[2025-07-18 16:18:22] 2025-07-18 20:17:48,325 | INFO | queue jobs had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,325 | INFO | queue payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue data_in had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue data_out had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue current_data_in had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue validated_jobs had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue validated_payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue monitored_payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue finished_jobs had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,326 | INFO | queue finished_payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue finished_data_in had 1 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue finished_data_out had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue failed_jobs had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue failed_payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue failed_data_in had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue failed_data_out had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue completed_jobs had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue completed_jobids has 1 job(s)
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue realtimelog_payloads had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,327 | INFO | queue messages had 0 job(s) [purged]
[2025-07-18 16:18:22] 2025-07-18 20:17:48,328 | INFO | job 6724307185 has completed (purged errors)
[2025-07-18 16:18:22] 2025-07-18 20:17:48,328 | INFO | overall cleanup function is called
[2025-07-18 16:18:22] 2025-07-18 20:17:49,340 | INFO | --- collectZombieJob: --- 10, [3771108]
[2025-07-18 16:18:22] 2025-07-18 20:17:49,341 | INFO | zombie collector waiting for pid 3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:49,341 | INFO | harmless exception when collecting zombies: [Errno 10] No child processes
[2025-07-18 16:18:22] 2025-07-18 20:17:49,341 | INFO | collected zombie processes
[2025-07-18 16:18:22] 2025-07-18 20:17:49,341 | INFO | will attempt to kill all subprocesses of pid=3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:50,263 | INFO | process IDs to be killed: [3771108] (in reverse order)
[2025-07-18 16:18:22] 2025-07-18 20:17:50,505 | WARNING | found no corresponding commands to process id(s)
[2025-07-18 16:18:22] 2025-07-18 20:17:50,505 | INFO | Do not look for orphan processes in BOINC jobs
[2025-07-18 16:18:22] 2025-07-18 20:17:50,543 | INFO | did not find any defunct processes belonging to 3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:50,582 | INFO | did not find any defunct processes belonging to 3771108
[2025-07-18 16:18:22] 2025-07-18 20:17:50,583 | INFO | ready for new job
[2025-07-18 16:18:22] 2025-07-18 20:17:50,583 | INFO | pilot has finished with previous job - re-establishing logging
[2025-07-18 16:18:22] 2025-07-18 20:17:50,592 | INFO | ***************************************
[2025-07-18 16:18:22] 2025-07-18 20:17:50,592 | INFO | *** PanDA Pilot version 3.10.4.12 ***
[2025-07-18 16:18:22] 2025-07-18 20:17:50,592 | INFO | ***************************************
[2025-07-18 16:18:22] 2025-07-18 20:17:50,592 | INFO |
[2025-07-18 16:18:22] 2025-07-18 20:17:50,607 | INFO | architecture information:
[2025-07-18 16:18:22] 2025-07-18 20:17:50,611 | INFO | executing command: cat /etc/os-release
[2025-07-18 16:18:22] 2025-07-18 20:17:50,669 | INFO | cat /etc/os-release:
[2025-07-18 16:18:22] NAME="CentOS Linux"
[2025-07-18 16:18:22] VERSION="7 (Core)"
[2025-07-18 16:18:22] ID="centos"
[2025-07-18 16:18:22] ID_LIKE="rhel fedora"
[2025-07-18 16:18:22] VERSION_ID="7"
[2025-07-18 16:18:22] PRETTY_NAME="CentOS Linux 7 (Core)"
[2025-07-18 16:18:22] ANSI_COLOR="0;31"
[2025-07-18 16:18:22] CPE_NAME="cpe:/o:centos:centos:7"
[2025-07-18 16:18:22] HOME_URL="https://www.centos.org/"
[2025-07-18 16:18:22] BUG_REPORT_URL="https://bugs.centos.org/"
[2025-07-18 16:18:22]
[2025-07-18 16:18:22] CENTOS_MANTISBT_PROJECT="CentOS-7"
[2025-07-18 16:18:22] CENTOS_MANTISBT_PROJECT_VERSION="7"
[2025-07-18 16:18:22] REDHAT_SUPPORT_PRODUCT="centos"
[2025-07-18 16:18:22] REDHAT_SUPPORT_PRODUCT_VERSION="7"
[2025-07-18 16:18:22]
[2025-07-18 16:18:22] 2025-07-18 20:17:50,669 | INFO | ***************************************
[2025-07-18 16:18:22] 2025-07-18 20:17:51,173 | INFO | executing command: df -mP /tmp/boinchome/slots/0
[2025-07-18 16:18:22] 2025-07-18 20:17:51,297 | INFO | sufficient remaining disk space (76295438336 B)
[2025-07-18 16:18:22] 2025-07-18 20:17:51,297 | WARNING | since timefloor is set to 0, pilot was only allowed to run one job
[2025-07-18 16:18:22] 2025-07-18 20:17:51,298 | INFO | current server update state: UPDATING_FINAL
[2025-07-18 16:18:22] 2025-07-18 20:17:51,298 | INFO | update_server=False
[2025-07-18 16:18:22] 2025-07-18 20:17:51,298 | WARNING | setting graceful_stop since proceed_with_getjob() returned False (pilot will end)
[2025-07-18 16:18:22] 2025-07-18 20:17:51,313 | WARNING | data:queue_monitoring:received graceful stop - abort after this iteration
[2025-07-18 16:18:22] 2025-07-18 20:17:51,767 | INFO | all data control threads have been joined
[2025-07-18 16:18:22] 2025-07-18 20:17:51,793 | WARNING | job:job_monitor:received graceful stop - abort after this iteration
[2025-07-18 16:18:22] 2025-07-18 20:17:51,794 | INFO | aborting loop
[2025-07-18 16:18:22] 2025-07-18 20:17:52,312 | INFO | [job] retrieve thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,437 | INFO | all payload control threads have been joined
[2025-07-18 16:18:22] 2025-07-18 20:17:52,569 | INFO | [payload] failed_post thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,619 | INFO | [job] create_data_payload thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,619 | INFO | [job] validate thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,714 | WARNING | data:copytool_out:received graceful stop - abort after this iteration
[2025-07-18 16:18:22] 2025-07-18 20:17:52,774 | INFO | [data] control thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,804 | INFO | [job] job monitor thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:52,819 | INFO | all job control threads have been joined
[2025-07-18 16:18:22] 2025-07-18 20:17:52,877 | INFO | [payload] run_realtimelog thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,211 | WARNING | job:queue_monitor:received graceful stop - abort after this iteration
[2025-07-18 16:18:22] 2025-07-18 20:17:53,298 | INFO | [data] copytool_in thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,391 | INFO | [payload] execute_payloads thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,443 | INFO | [payload] control thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,762 | INFO | [payload] validate_pre thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,799 | INFO | [payload] validate_post thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:53,825 | INFO | [job] control thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:54,217 | INFO | [job] queue monitor thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:54,721 | INFO | [data] copytool_out thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:55,322 | INFO | [data] queue_monitor thread has finished
[2025-07-18 16:18:22] 2025-07-18 20:17:56,137 | INFO | only monitor.control thread still running - safe to abort: ['<_MainThread(MainThread, started 140664044136256)>', '<ExcThread(monitor, started 140663156877056)>']
[2025-07-18 16:18:22] 2025-07-18 20:18:01,164 | INFO | all workflow threads have been joined
[2025-07-18 16:18:22] 2025-07-18 20:18:01,165 | INFO | end of generic workflow (traces error code: 0)
[2025-07-18 16:18:22] 2025-07-18 20:18:01,165 | INFO | traces error code: 0
[2025-07-18 16:18:22] 2025-07-18 20:18:01,165 | INFO | pilot has finished (exit code=0, shell exit code=0)
[2025-07-18 16:18:22] 2025-07-18 20:18:20,714 | INFO | PID=3762839 has CPU usage=3.4% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i P
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | INFO | .. there are 37 such processes running
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | INFO | found 0 job(s) in 20 queues
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | WARNING | pilot monitor received instruction that args.graceful_stop has been set
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | WARNING | will wait for a maximum of 300 s for threads to finish
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | WARNING | job_aborted has been set - aborting pilot monitoring
[2025-07-18 16:18:22] 2025-07-18 20:18:20,715 | INFO | [monitor] control thread has ended
[2025-07-18 16:18:22] 2025-07-18 20:18:21,044 [wrapper] ==== pilot stdout END ====
[2025-07-18 16:18:22] 2025-07-18 20:18:21,063 [wrapper] ==== wrapper stdout RESUME ====
[2025-07-18 16:18:22] 2025-07-18 20:18:21,119 [wrapper] pilotpid: 3762839
[2025-07-18 16:18:22] 2025-07-18 20:18:21,123 [wrapper] Pilot exit status: 0
[2025-07-18 16:18:22] 2025-07-18 20:18:21,240 [wrapper] pandaids: 6724307185 6724307185 6724307185 6724307185 6724307185
[2025-07-18 16:18:22] 2025-07-18 20:18:21,763 [wrapper] cleanup supervisor_pilot 1873402 3762840
[2025-07-18 16:18:22] 2025-07-18 20:18:21,777 [wrapper] Test setup, not cleaning
[2025-07-18 16:18:22] 2025-07-18 20:18:21,798 [wrapper] apfmon messages muted
[2025-07-18 16:18:22] 2025-07-18 20:18:21,837 [wrapper] ==== wrapper stdout END ====
[2025-07-18 16:18:22] 2025-07-18 20:18:21,883 [wrapper] ==== wrapper stderr END ====
[2025-07-18 16:18:22] *** Error codes and diagnostics ***
[2025-07-18 16:18:22] "exeErrorCode": 0,
[2025-07-18 16:18:22] "exeErrorDiag": "",
[2025-07-18 16:18:22] "pilotErrorCode": 0,
[2025-07-18 16:18:22] "pilotErrorDiag": "",
[2025-07-18 16:18:22] *** Listing of results directory ***
[2025-07-18 16:18:22] total 887004
[2025-07-18 16:18:22] drwx------. 4 boincer umatlas 4096 Jun 17 06:04 pilot3
[2025-07-18 16:18:22] -rwx------. 1 boincer umatlas 36292 Jul 7 19:37 runpilot2-wrapper.sh
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 5112 Jul 7 19:37 queuedata.json
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 530466 Jul 7 19:37 pilot3.tar.gz
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 100 Jul 7 19:38 wrapper_26015_x86_64-pc-linux-gnu
[2025-07-18 16:18:22] -rwxr-xr-x. 1 boincer umatlas 7986 Jul 7 19:38 run_atlas
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 105 Jul 7 19:38 job.xml
[2025-07-18 16:18:22] -rw-r--r--. 3 boincer umatlas 317673579 Jul 7 19:38 EVNT.45482146._000029.pool.root.1
[2025-07-18 16:18:22] -rw-r--r--. 3 boincer umatlas 317673579 Jul 7 19:38 ATLAS.root_0
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 15095 Jul 7 19:38 start_atlas.sh
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 542903 Jul 7 19:38 input.tar.gz
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 0 Jul 7 19:38 boinc_lockfile
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 0 Jul 8 09:36 wrapper_sigint_15561
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 0 Jul 8 09:39 wrapper_sigkill_15561
[2025-07-18 16:18:22] drwxrwx--x. 2 boincer umatlas 4096 Jul 8 09:39 shared
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 0 Jul 18 06:30 wrapper_sigint_3697583
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 0 Jul 18 06:33 wrapper_sigkill_3697583
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 2611 Jul 18 06:46 pandaJob.out
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 977668 Jul 18 06:47 agis_schedconf.cvmfs.json
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 6661 Jul 18 16:12 init_data.xml
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 263007918 Jul 18 16:16 HITS.45482148._017145.pool.root.1
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 535 Jul 18 16:16 boinc_task_state.xml
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 1056 Jul 18 16:17 memory_monitor_summary.json
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 1593055 Jul 18 16:17 agis_ddmendpoints.agis.ALL.json
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 573651 Jul 18 16:17 log.45482148._017145.job.log.tgz.1
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 99 Jul 18 16:17 pilot_heartbeat.json
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 7842 Jul 18 16:17 heartbeat.json
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 31 Jul 18 16:17 wrapper_checkpoint.txt
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 823 Jul 18 16:18 pilotlog.txt
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 2432904 Jul 18 16:18 log.45482148._017145.job.log.1
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 357 Jul 18 16:18 output.list
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 620 Jul 18 16:18 runtime_log
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 3031040 Jul 18 16:18 result.tar.gz
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 2121 Jul 18 16:18 zzFLDm6ZAt7n9Rq4apOajLDm4fhM0noT9bVof3QYDmLUsKDmimL4mm.diag
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 8733 Jul 18 16:18 runtime_log.err
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 8192 Jul 18 16:18 boinc_mmap_file
[2025-07-18 16:18:22] -rw-r--r--. 1 boincer umatlas 55849 Jul 18 16:18 stderr.txt
[2025-07-18 16:18:22] HITS file was successfully produced:
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 263007918 Jul 18 16:16 shared/HITS.pool.root.1
[2025-07-18 16:18:22] *** Contents of shared directory: ***
[2025-07-18 16:18:22] total 570592
[2025-07-18 16:18:22] -rw-r--r--. 3 boincer umatlas 317673579 Jul 7 19:38 ATLAS.root_0
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 15095 Jul 7 19:38 start_atlas.sh
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 542903 Jul 7 19:38 input.tar.gz
[2025-07-18 16:18:22] -rw-------. 1 boincer umatlas 263007918 Jul 18 16:16 HITS.pool.root.1
[2025-07-18 16:18:22] -rw-r--r--. 2 boincer umatlas 3031040 Jul 18 16:18 result.tar.gz
16:18:43 (3757494): run_atlas exited; CPU time 72977.984673
16:18:43 (3757494): called boinc_finish(0)
</stderr_txt>
]]>
©2025 CERN