Stderr output

<core_client_version>8.1.0</core_client_version>
<![CDATA[
<stderr_txt>
15:44:25 (789552): wrapper (7.7.26015): starting
15:44:25 (789552): wrapper: running run_atlas (--nthreads 8)
[2025-04-03 15:44:25] Arguments: --nthreads 8
[2025-04-03 15:44:25] Threads: 8
[2025-04-03 15:44:25] Checking for CVMFS
[2025-04-03 15:44:25] Probing /cvmfs/atlas.cern.ch... OK
[2025-04-03 15:44:26] Probing /cvmfs/atlas-condb.cern.ch... OK
[2025-04-03 15:44:26] Running cvmfs_config stat atlas.cern.ch
[2025-04-03 15:44:27] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2025-04-03 15:44:27] 2.12.6.0 2445012 23679 272432 144463 1 196 27552621 32935937 21120 130560 0 10947198 99.837 3257376 25834 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2025-04-03 15:44:27] CVMFS is ok
[2025-04-03 15:44:27] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2025-04-03 15:44:27] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2025-04-03 15:44:27] Further information can be found at the LHC@home message board.
[2025-04-03 15:44:27] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2025-04-03 15:44:27] Checking for apptainer binary...
[2025-04-03 15:44:27] Using apptainer found in PATH at /usr/bin/apptainer
[2025-04-03 15:44:27] Running /usr/bin/apptainer --version
[2025-04-03 15:44:27] apptainer version 1.4.0-1.el9
[2025-04-03 15:44:27] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2025-04-03 15:44:28] c6-12-27-1.aglt2.org
[2025-04-03 15:44:28] apptainer works
[2025-04-03 15:44:28] Set ATHENA_PROC_NUMBER=8
[2025-04-03 15:44:28] Set ATHENA_CORE_NUMBER=8
[2025-04-03 15:44:28] Starting ATLAS job with PandaID=6584894904
[2025-04-03 15:44:28] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
[2025-04-03 17:29:11]  *** The last 200 lines of the pilot log: ***
[2025-04-03 17:29:11] 2025-04-03 21:25:50,478 | INFO     | payload log (payload.stdout) within allowed size limit (2147483648 B): 9339 B
[2025-04-03 17:29:11] 2025-04-03 21:25:50,479 | INFO     | executing command: df -mP /tmp/boinchome/slots/0
[2025-04-03 17:29:11] 2025-04-03 21:25:50,557 | INFO     | sufficient remaining disk space (92946825216 B)
[2025-04-03 17:29:11] 2025-04-03 21:25:50,558 | INFO     | work directory size check will use 2537553920 B as a max limit (10% grace limit added)
[2025-04-03 17:29:11] 2025-04-03 21:25:50,558 | INFO     | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6584894904: 49650750 B (within 2537553920 B limit)
[2025-04-03 17:29:11] 2025-04-03 21:25:50,559 | INFO     | total size of present files: 46014871 B (workdir size: 49650750 B)
[2025-04-03 17:29:11] 2025-04-03 21:25:50,559 | INFO     | output file /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/HITS.44004367._003573.pool.root.1 is within allowed size limit (46014871 B < 536870912000 B)
[2025-04-03 17:29:11] 2025-04-03 21:25:52,331 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:25:52,331 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:25:54,851 | INFO     | monitor loop #343: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:26:11,194 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:26:11,194 | INFO     | CPU consumption time changed by a factor of 1.0062690753113916 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:26:11,195 | INFO     | (instant) CPU consumption time for pid=813809: 12199)
[2025-04-03 17:29:11] 2025-04-03 21:26:11,195 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:26:15,434 | INFO     | PID=797751 has CPU usage=8.7% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i PR
[2025-04-03 17:29:11] 2025-04-03 21:26:15,434 | INFO     | .. there are 21 such processes running
[2025-04-03 17:29:11] 2025-04-03 21:26:15,465 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:15,466 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:17,971 | INFO     | monitor loop #344: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:26:29,316 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:26:29,316 | INFO     | CPU consumption time changed by a factor of 1.0059021231248464 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:26:29,317 | INFO     | (instant) CPU consumption time for pid=813809: 12271)
[2025-04-03 17:29:11] 2025-04-03 21:26:29,317 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:26:30,805 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:30,805 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:33,315 | INFO     | monitor loop #345: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:26:40,197 | INFO     | PID=797751 has CPU usage=8.7% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i PR
[2025-04-03 17:29:11] 2025-04-03 21:26:40,197 | INFO     | .. there are 21 such processes running
[2025-04-03 17:29:11] 2025-04-03 21:26:46,901 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:26:46,901 | INFO     | CPU consumption time changed by a factor of 1.0043191263955669 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:26:46,902 | INFO     | (instant) CPU consumption time for pid=813809: 12324)
[2025-04-03 17:29:11] 2025-04-03 21:26:46,902 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:26:48,048 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:48,049 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:26:50,559 | INFO     | monitor loop #346: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:27:05,640 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:27:05,640 | INFO     | CPU consumption time changed by a factor of 1.0059234014930218 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:27:05,640 | INFO     | (instant) CPU consumption time for pid=813809: 12397)
[2025-04-03 17:29:11] 2025-04-03 21:27:05,671 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:27:07,184 | INFO     | PID=797751 has CPU usage=9.3% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i PR
[2025-04-03 17:29:11] 2025-04-03 21:27:07,185 | INFO     | .. there are 21 such processes running
[2025-04-03 17:29:11] 2025-04-03 21:27:07,795 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:07,795 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:10,327 | INFO     | monitor loop #347: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:27:27,951 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:27:27,951 | INFO     | CPU consumption time changed by a factor of 1.005001209970154 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:27:27,952 | INFO     | (instant) CPU consumption time for pid=813809: 12459)
[2025-04-03 17:29:11] 2025-04-03 21:27:27,952 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:27:27,975 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:27:27,999 | INFO     | max memory (maxPSS) used by the payload is within the allowed limit: 2486448 B (2.0 (queuedata.pilot_rss_grace) * 8000 (queuedata.maxrss) * 4.0 (scale) = 65536000.
[2025-04-03 17:29:11] 2025-04-03 21:27:28,000 | INFO     | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 3383112 B
[2025-04-03 17:29:11] 2025-04-03 21:27:28,000 | INFO     | payload log (payload.stdout) within allowed size limit (2147483648 B): 9339 B
[2025-04-03 17:29:11] 2025-04-03 21:27:28,000 | INFO     | executing command: df -mP /tmp/boinchome/slots/0
[2025-04-03 17:29:11] 2025-04-03 21:27:28,143 | INFO     | sufficient remaining disk space (92939485184 B)
[2025-04-03 17:29:11] 2025-04-03 21:27:28,143 | INFO     | work directory size check will use 2537553920 B as a max limit (10% grace limit added)
[2025-04-03 17:29:11] 2025-04-03 21:27:28,144 | INFO     | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6584894904: 49654535 B (within 2537553920 B limit)
[2025-04-03 17:29:11] 2025-04-03 21:27:28,145 | INFO     | total size of present files: 46017037 B (workdir size: 49654535 B)
[2025-04-03 17:29:11] 2025-04-03 21:27:28,145 | INFO     | output file /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/HITS.44004367._003573.pool.root.1 is within allowed size limit (46017037 B < 536870912000 B)
[2025-04-03 17:29:11] 2025-04-03 21:27:31,081 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:31,240 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:33,883 | INFO     | monitor loop #348: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:27:34,241 | INFO     | PID=797751 has CPU usage=8.4% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i PR
[2025-04-03 17:29:11] 2025-04-03 21:27:34,241 | INFO     | .. there are 21 such processes running
[2025-04-03 17:29:11] 2025-04-03 21:27:49,285 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:27:49,286 | INFO     | CPU consumption time changed by a factor of 1.0058592182358135 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:27:49,286 | INFO     | (instant) CPU consumption time for pid=813809: 12532)
[2025-04-03 17:29:11] 2025-04-03 21:27:49,286 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:27:49,527 | INFO     | 6147s have passed since pilot start - server update state is 'RUNNING'
[2025-04-03 17:29:11] 2025-04-03 21:27:51,752 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:51,761 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:27:54,266 | INFO     | monitor loop #349: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:28:01,464 | INFO     | PID=797751 has CPU usage=6.7% CMD=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.20-x86_64-centos7/bin/python3 pilot3/pilot.py -q BOINC_MCORE -i PR
[2025-04-03 17:29:11] 2025-04-03 21:28:01,464 | INFO     | .. there are 21 such processes running
[2025-04-03 17:29:11] 2025-04-03 21:28:08,329 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:28:08,329 | INFO     | CPU consumption time changed by a factor of 1.0069422278965847 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:28:08,330 | INFO     | (instant) CPU consumption time for pid=813809: 12619)
[2025-04-03 17:29:11] 2025-04-03 21:28:08,330 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:28:11,105 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:28:11,105 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:28:13,617 | INFO     | monitor loop #350: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/common/exception.py", line 466, in run
[2025-04-03 17:29:11]     self._target(**self.kwargs)
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/control/monitor.py", line 208, in control
[2025-04-03 17:29:11]     raise PilotException(error) from error
[2025-04-03 17:29:11] received exception from bucket queue in generic workflow: error code: 1301, message: An unknown pilot exception has occurred
[2025-04-03 17:29:11] details: 'cmdline'
[2025-04-03 17:29:11] monitor: exception caught: 'cmdline'
[2025-04-03 17:29:11] unexpected exception caught by thread run() function: (<class 'pilot.common.exception.PilotException'>, PilotException(KeyError('cmdline')), <traceback object at 0x7f9856e60500>)
[2025-04-03 17:29:11] Traceback (most recent call last):
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/control/monitor.py", line 183, in control
[2025-04-03 17:29:11]     processes = get_process_info('python3 pilot3/pilot.py', pid=getpid())
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/util/psutils.py", line 402, in get_process_info
[2025-04-03 17:29:11]     cmdline = proc.info['cmdline']
[2025-04-03 17:29:11] KeyError: 'cmdline'
[2025-04-03 17:29:11] 
[2025-04-03 17:29:11] The above exception was the direct cause of the following exception:
[2025-04-03 17:29:11] 
[2025-04-03 17:29:11] Traceback (most recent call last):
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/common/exception.py", line 466, in run
[2025-04-03 17:29:11]     self._target(**self.kwargs)
[2025-04-03 17:29:11]   File "/tmp/boinchome/slots/0/pilot3/pilot/control/monitor.py", line 208, in control
[2025-04-03 17:29:11]     raise PilotException(error) from error
[2025-04-03 17:29:11] pilot.common.exception.PilotException: error code: 1301, message: An unknown pilot exception has occurred
[2025-04-03 17:29:11] details: 'cmdline'
[2025-04-03 17:29:11] 
[2025-04-03 17:29:11] None
[2025-04-03 17:29:11] exception has been put in bucket queue belonging to thread 'monitor'
[2025-04-03 17:29:11] setting graceful stop in 10 s since there is no point in continuing
[2025-04-03 17:29:11] 2025-04-03 21:28:29,863 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:28:29,863 | INFO     | CPU consumption time changed by a factor of 1.0054679451620572 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:28:29,864 | INFO     | (instant) CPU consumption time for pid=813809: 12688)
[2025-04-03 17:29:11] 2025-04-03 21:28:29,864 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:28:33,035 | INFO     | number of running child processes to parent process 813809: 6
[2025-04-03 17:29:11] 2025-04-03 21:28:33,035 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:28:35,541 | INFO     | monitor loop #351: job 0:6584894904 is in state 'running'
[2025-04-03 17:29:11] 2025-04-03 21:28:37,607 | WARNING  | data:queue_monitoring:received graceful stop - abort after this iteration
[2025-04-03 17:29:11] 2025-04-03 21:28:37,845 | INFO     | all data control threads have been joined
[2025-04-03 17:29:11] 2025-04-03 21:28:38,195 | WARNING  | data:copytool_out:received graceful stop - abort after this iteration
[2025-04-03 17:29:11] 2025-04-03 21:28:38,292 | INFO     | breaking -- sending SIGTERM to pid=813809
[2025-04-03 17:29:11] 2025-04-03 21:28:38,292 | INFO     | breaking -- sleep 10 s before sending SIGKILL pid=813809
[2025-04-03 17:29:11] 2025-04-03 21:28:38,627 | INFO     | all job control threads have been joined
[2025-04-03 17:29:11] 2025-04-03 21:28:38,638 | INFO     | [job] retrieve thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:38,857 | INFO     | [data] control thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:38,868 | INFO     | all payload control threads have been joined
[2025-04-03 17:29:11] 2025-04-03 21:28:38,990 | WARNING  | job:queue_monitor:received graceful stop - abort after this iteration
[2025-04-03 17:29:11] 2025-04-03 21:28:38,990 | WARNING  | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s
[2025-04-03 17:29:11] 2025-04-03 21:28:39,136 | INFO     | [payload] validate_pre thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:39,635 | INFO     | [job] control thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:39,639 | INFO     | [data] copytool_in thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:39,639 | INFO     | [job] validate thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:39,646 | INFO     | [job] create_data_payload thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:39,876 | INFO     | [payload] control thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:40,047 | INFO     | [payload] validate_post thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:40,054 | INFO     | [payload] failed_post thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:40,202 | INFO     | [data] copytool_out thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:41,616 | INFO     | [data] queue_monitor thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:43,709 | INFO     | job.realtimelogging is not enabled
[2025-04-03 17:29:11] 2025-04-03 21:28:44,715 | INFO     | [payload] run_realtimelog thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:47,601 | INFO     | system is under heavy CPU load
[2025-04-03 17:29:11] 2025-04-03 21:28:47,601 | INFO     | CPU consumption time changed by a factor of 0.0004728877679697352 (below the limit of 10)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,601 | INFO     | (instant) CPU consumption time for pid=813809: 6)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,602 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,628 | INFO     | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_output.txt (trf name=prmon)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,635 | INFO     | max memory (maxPSS) used by the payload is within the allowed limit: 2486448 B (2.0 (queuedata.pilot_rss_grace) * 8000 (queuedata.maxrss) * 4.0 (scale) = 65536000.
[2025-04-03 17:29:11] 2025-04-03 21:28:47,636 | INFO     | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 3384058 B
[2025-04-03 17:29:11] 2025-04-03 21:28:47,666 | INFO     | payload log (payload.stdout) within allowed size limit (2147483648 B): 9339 B
[2025-04-03 17:29:11] 2025-04-03 21:28:47,666 | INFO     | executing command: df -mP /tmp/boinchome/slots/0
[2025-04-03 17:29:11] 2025-04-03 21:28:47,817 | INFO     | sufficient remaining disk space (92933193728 B)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,817 | INFO     | work directory size check will use 2537553920 B as a max limit (10% grace limit added)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,818 | INFO     | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6584894904: 49655884 B (within 2537553920 B limit)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,819 | INFO     | total size of present files: 46017037 B (workdir size: 49655884 B)
[2025-04-03 17:29:11] 2025-04-03 21:28:47,819 | INFO     | output file /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/HITS.44004367._003573.pool.root.1 is within allowed size limit (46017037 B < 536870912000 B)
[2025-04-03 17:29:11] 2025-04-03 21:28:48,434 | INFO     | 
[2025-04-03 17:29:11] 
[2025-04-03 17:29:11] finished pid=813809 exit_code=None state=failed
[2025-04-03 17:29:11] 
[2025-04-03 17:29:11] 2025-04-03 21:28:48,434 | WARNING  | detected unset exit_code from wait_graceful - reset to -1
[2025-04-03 17:29:11] 2025-04-03 21:28:48,490 | INFO     | using pid=828804 to kill prmon
[2025-04-03 17:29:11] 2025-04-03 21:28:48,490 | INFO     | stopping utility process 'MemoryMonitor' with signal 10
[2025-04-03 17:29:11] 2025-04-03 21:28:48,490 | INFO     | process 828804 no longer exists
[2025-04-03 17:29:11] 2025-04-03 21:28:48,490 | INFO     | utility process 813868 cleanup finished with status=True
[2025-04-03 17:29:11] 2025-04-03 21:28:48,490 | INFO     | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20)
[2025-04-03 17:29:11] 2025-04-03 21:28:50,968 | INFO     | number of running child processes to parent process 813809: 1
[2025-04-03 17:29:11] 2025-04-03 21:28:50,968 | INFO     | maximum number of monitored processes: 6
[2025-04-03 17:29:11] 2025-04-03 21:28:50,969 | WARNING  | job:job_monitor:received graceful stop - abort after this iteration
[2025-04-03 17:29:11] 2025-04-03 21:28:50,989 | INFO     | will abort loop
[2025-04-03 17:29:11] 2025-04-03 21:28:51,506 | INFO     | copied /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/memory_monitor_summary.json to /tmp/boinchome/slots/0
[2025-04-03 17:29:11] 2025-04-03 21:28:51,995 | INFO     | [job] job monitor thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:52,021 | INFO     | found no lingering processes
[2025-04-03 17:29:11] 2025-04-03 21:28:52,021 | INFO     | CPU consumption time: 35.5 s (rounded to 36 s)
[2025-04-03 17:29:11] 2025-04-03 21:28:52,021 | WARNING  | main payload execution returned non-zero exit code: -1
[2025-04-03 17:29:11] 2025-04-03 21:28:52,022 | INFO     | scanning dmesg message for subprocess=821026 for memory errors
[2025-04-03 17:29:11] 2025-04-03 21:28:52,022 | INFO     | executing command: dmesg|grep 821026
[2025-04-03 17:29:11] 2025-04-03 21:28:52,294 | WARNING  | job report does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/jobReport.json
[2025-04-03 17:29:11] 2025-04-03 21:28:52,294 | WARNING  | metadata does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/metadata.xml
[2025-04-03 17:29:11] 2025-04-03 21:28:52,294 | WARNING  | file does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6584894904/metadata.xml
[2025-04-03 17:29:11] 2025-04-03 21:28:52,295 | INFO     | generated guid for lfn=HITS.44004367._003573.pool.root.1: 7D2D84FC-9E71-4CC8-8BB0-0764D1C8BCEE
[2025-04-03 17:29:11] 2025-04-03 21:28:52,295 | WARNING  | aborting payload error diagnosis since an error has already been set: [1315, 1187]
[2025-04-03 17:29:11] 2025-04-03 21:28:53,412 | INFO     | [payload] execute_payloads thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:28:55,017 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 140292361279296)>', '<ExcThread(queue_monitor, started 140291851925248)>']
[2025-04-03 17:29:11] 2025-04-03 21:28:57,031 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 140292361279296)>', '<ExcThread(queue_monitor, started 140291851925248)>']
[2025-04-03 17:29:11] 2025-04-03 21:28:59,043 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 140292361279296)>', '<ExcThread(queue_monitor, started 140291851925248)>']
[2025-04-03 17:29:11] 2025-04-03 21:29:01,055 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 140292361279296)>', '<ExcThread(queue_monitor, started 140291851925248)>']
[2025-04-03 17:29:11] 2025-04-03 21:29:02,108 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 140292361279296)>', '<ExcThread(queue_monitor, started 140291851925248)>']
[2025-04-03 17:29:11] 2025-04-03 21:29:02,108 | INFO     | [job] queue monitor thread has finished
[2025-04-03 17:29:11] 2025-04-03 21:29:03,074 | INFO     | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140292361279296)>'])
[2025-04-03 17:29:11] 2025-04-03 21:29:08,100 | INFO     | all workflow threads have been joined
[2025-04-03 17:29:11] 2025-04-03 21:29:08,100 | INFO     | end of generic workflow (traces error code: 0)
[2025-04-03 17:29:11] 2025-04-03 21:29:08,100 | INFO     | traces error code: 0
[2025-04-03 17:29:11] 2025-04-03 21:29:08,100 | INFO     | pilot has finished (exit code=0, shell exit code=0)
[2025-04-03 17:29:11] 2025-04-03 21:29:10,368 [wrapper] ==== pilot stdout END ====
[2025-04-03 17:29:11] 2025-04-03 21:29:10,403 [wrapper] ==== wrapper stdout RESUME ====
[2025-04-03 17:29:11] 2025-04-03 21:29:10,448 [wrapper] pilotpid: 797751
[2025-04-03 17:29:11] 2025-04-03 21:29:10,463 [wrapper] Pilot exit status: 0
[2025-04-03 17:29:11] 2025-04-03 21:29:10,580 [wrapper] pandaids: 6584894904
[2025-04-03 17:29:11] 2025-04-03 21:29:10,846 [wrapper] cleanup supervisor_pilot 1355618 797752
[2025-04-03 17:29:11] 2025-04-03 21:29:10,866 [wrapper] Test setup, not cleaning
[2025-04-03 17:29:11] 2025-04-03 21:29:10,877 [wrapper] apfmon messages muted
[2025-04-03 17:29:11] 2025-04-03 21:29:10,880 [wrapper] ==== wrapper stdout END ====
[2025-04-03 17:29:11] 2025-04-03 21:29:10,900 [wrapper] ==== wrapper stderr END ====
[2025-04-03 17:29:11]  *** Error codes and diagnostics ***
[2025-04-03 17:29:11]  *** Listing of results directory ***
[2025-04-03 17:29:11] total 223912
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas    516735 Mar 31 06:50 pilot3.tar.gz
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas      5112 Apr  2 15:24 queuedata.json
[2025-04-03 17:29:11] -rwx------. 1 boincer umatlas     37140 Apr  2 15:25 runpilot2-wrapper.sh
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas       100 Apr  3 15:44 wrapper_26015_x86_64-pc-linux-gnu
[2025-04-03 17:29:11] -rwxr-xr-x. 1 boincer umatlas      7986 Apr  3 15:44 run_atlas
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas       105 Apr  3 15:44 job.xml
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas 223980815 Apr  3 15:44 EVNT.44004365._000143.pool.root.1
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas     17571 Apr  3 15:44 start_atlas.sh
[2025-04-03 17:29:11] drwxrwx--x. 2 boincer umatlas      4096 Apr  3 15:44 shared
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas    530802 Apr  3 15:44 input.tar.gz
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas         0 Apr  3 15:44 boinc_lockfile
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas      2548 Apr  3 15:44 pandaJob.out
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas       467 Apr  3 15:44 setup.sh.local
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas    979952 Apr  3 15:45 agis_schedconf.cvmfs.json
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas   1603525 Apr  3 15:45 agis_ddmendpoints.agis.ALL.json
[2025-04-03 17:29:11] drwx------. 6 boincer umatlas      4096 Apr  3 15:45 pilot3
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas       916 Apr  3 17:17 heartbeat.json
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas        95 Apr  3 17:28 pilot_heartbeat.json
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas       529 Apr  3 17:28 boinc_task_state.xml
[2025-04-03 17:29:11] drwxrwx---. 2 boincer umatlas      4096 Apr  3 17:28 PanDA_Pilot-6584894904
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas      1045 Apr  3 17:28 memory_monitor_summary.json
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas    464449 Apr  3 17:29 pilotlog.txt
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas      6858 Apr  3 17:29 init_data.xml
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas      8192 Apr  3 17:29 boinc_mmap_file
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas        27 Apr  3 17:29 wrapper_checkpoint.txt
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas    487923 Apr  3 17:29 log.44004367._003573.job.log.1
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas       571 Apr  3 17:29 runtime_log
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas    501760 Apr  3 17:29 result.tar.gz
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas     10756 Apr  3 17:29 runtime_log.err
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas       645 Apr  3 17:29 QKqNDmltIK7nsSi4ap6QjLDmwznN0nGgGQJmkKkKDmHyFLDmnYwinn.diag
[2025-04-03 17:29:11] -rw-r--r--. 1 boincer umatlas     25668 Apr  3 17:29 stderr.txt
[2025-04-03 17:29:11] No HITS result produced
[2025-04-03 17:29:11]  *** Contents of shared directory: ***
[2025-04-03 17:29:11] total 219764
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas 223980815 Apr  3 15:44 ATLAS.root_0
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas     17571 Apr  3 15:44 start_atlas.sh
[2025-04-03 17:29:11] -rw-r--r--. 2 boincer umatlas    530802 Apr  3 15:44 input.tar.gz
[2025-04-03 17:29:11] -rw-------. 1 boincer umatlas    501760 Apr  3 17:29 result.tar.gz
17:29:12 (789552): run_atlas exited; CPU time 378.389391
17:29:12 (789552): called boinc_finish(0)

</stderr_txt>
]]>

©2025 CERN