| Name | ehMMDmVjJO6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmr6XLDm02SNQo_0 |
| Workunit | 226258407 |
| Created | 24 Oct 2024, 20:36:29 UTC |
| Sent | 24 Oct 2024, 20:47:49 UTC |
| Report deadline | 1 Nov 2024, 20:47:49 UTC |
| Received | 25 Oct 2024, 1:31:13 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10804956 |
| Run time | 4 hours 32 min 1 sec |
| CPU time | 7 hours 39 min 57 sec |
| Validate state | Valid |
| Credit | 745.32 |
| Device peak FLOPS | 43.24 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 1.53 GB |
| Peak swap size | 31.77 GB |
| Peak disk usage | 1.12 GB |
Stderr output
<core_client_version>7.7.0</core_client_version> <![CDATA[ <stderr_txt> 16:52:50 (16649): wrapper (7.7.26015): starting 16:52:50 (16649): wrapper: running run_atlas (--nthreads 8) [2024-10-24 16:52:50] Arguments: --nthreads 8 [2024-10-24 16:52:50] Threads: 8 [2024-10-24 16:52:50] Checking for CVMFS [2024-10-24 16:52:50] Probing /cvmfs/atlas.cern.ch... OK [2024-10-24 16:52:50] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-24 16:52:50] Running cvmfs_config stat atlas.cern.ch [2024-10-24 16:52:50] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-24 16:52:50] 2.11.2.0 85160 1440 126212 138690 1 542 9248097 18432001 25488 130560 0 1530180 99.031 1819182 63212 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-24 16:52:50] CVMFS is ok [2024-10-24 16:52:50] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-24 16:52:50] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-24 16:52:50] Further information can be found at the LHC@home message board. [2024-10-24 16:52:50] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-24 16:52:50] Checking for apptainer binary... [2024-10-24 16:52:50] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-24 16:52:50] Running /usr/bin/apptainer --version [2024-10-24 16:52:51] apptainer version 1.3.2-1.el7 [2024-10-24 16:52:51] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-24 16:52:52] c-212-11.aglt2.org [2024-10-24 16:52:52] apptainer works [2024-10-24 16:52:52] Set ATHENA_PROC_NUMBER=8 [2024-10-24 16:52:52] Set ATHENA_CORE_NUMBER=8 [2024-10-24 16:52:52] Starting ATLAS job with PandaID=6377708675 [2024-10-24 16:52:52] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh 18:46:07 (66739): wrapper (7.7.26015): starting 18:46:07 (66739): wrapper: running run_atlas (--nthreads 8) [2024-10-24 18:46:07] Arguments: --nthreads 8 [2024-10-24 18:46:07] Threads: 8 [2024-10-24 18:46:07] This job has been restarted, cleaning up previous attempt [2024-10-24 18:46:07] Checking for CVMFS [2024-10-24 18:46:07] Probing /cvmfs/atlas.cern.ch... OK [2024-10-24 18:46:07] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-24 18:46:07] Running cvmfs_config stat atlas.cern.ch [2024-10-24 18:46:08] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-24 18:46:08] 2.11.2.0 85160 1554 112148 138697 0 466 10382945 18432001 26969 130560 0 2556502 99.399 2636973 65562 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-24 18:46:08] CVMFS is ok [2024-10-24 18:46:08] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-24 18:46:08] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-24 18:46:08] Further information can be found at the LHC@home message board. [2024-10-24 18:46:08] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-24 18:46:08] Checking for apptainer binary... [2024-10-24 18:46:08] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-24 18:46:08] Running /usr/bin/apptainer --version [2024-10-24 18:46:08] apptainer version 1.3.2-1.el7 [2024-10-24 18:46:08] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-24 18:46:10] c-212-11.aglt2.org [2024-10-24 18:46:10] apptainer works [2024-10-24 18:46:10] Set ATHENA_PROC_NUMBER=8 [2024-10-24 18:46:10] Set ATHENA_CORE_NUMBER=8 [2024-10-24 18:46:10] Starting ATLAS job with PandaID=6377708675 [2024-10-24 18:46:10] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/0 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh [2024-10-24 21:30:52] *** The last 200 lines of the pilot log: *** [2024-10-24 21:30:52] 2024-10-25 01:27:00,490 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:27:03,009 | INFO | monitor loop #658: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:27:14,287 | INFO | (instant) CPU consumption time for pid=43342: 283.52 (rounded to 284) [2024-10-24 21:30:52] 2024-10-25 01:27:14,287 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:27:14,287 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:27:14,303 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:27:16,237 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:27:16,238 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:27:18,749 | INFO | monitor loop #659: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:27:28,326 | INFO | 9654s have passed since pilot start [2024-10-24 21:30:52] 2024-10-25 01:27:30,443 | INFO | (instant) CPU consumption time for pid=43342: 283.99 (rounded to 284) [2024-10-24 21:30:52] 2024-10-25 01:27:30,443 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:27:30,443 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:27:30,453 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:27:32,335 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:27:32,335 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:27:34,858 | INFO | monitor loop #660: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:27:46,336 | INFO | (instant) CPU consumption time for pid=43342: 284.23 (rounded to 284) [2024-10-24 21:30:52] 2024-10-25 01:27:46,336 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:27:46,336 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:27:46,339 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:27:46,340 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:27:46,343 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2687479 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-24 21:30:52] 2024-10-25 01:27:46,343 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:27:46,344 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-24 21:30:52] 2024-10-25 01:27:46,344 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-24 21:30:52] 2024-10-25 01:27:46,344 | INFO | executing command: df -mP /tmp/boinchome/slots/0 [2024-10-24 21:30:52] 2024-10-25 01:27:46,527 | INFO | sufficient remaining disk space (101240012800 B) [2024-10-24 21:30:52] 2024-10-25 01:27:46,527 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-24 21:30:52] 2024-10-25 01:27:46,529 | INFO | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6377708675: 77633309 B (within 5767168000 B limit) [2024-10-24 21:30:52] 2024-10-25 01:27:46,554 | INFO | pfn file=/tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-24 21:30:52] 2024-10-25 01:27:46,555 | INFO | total size of present files: 0 B (workdir size: 77633309 B) [2024-10-24 21:30:52] 2024-10-25 01:27:46,555 | INFO | output file size check: skipping output file /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 since it does not exist [2024-10-24 21:30:52] 2024-10-25 01:27:47,528 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:27:47,528 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:27:50,037 | INFO | monitor loop #661: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:28:01,065 | INFO | (instant) CPU consumption time for pid=43342: 284.24 (rounded to 284) [2024-10-24 21:30:52] 2024-10-25 01:28:01,066 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:28:01,066 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:28:01,077 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:28:02,041 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:28:02,041 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:28:04,546 | INFO | monitor loop #662: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:28:16,141 | INFO | (instant) CPU consumption time for pid=43342: 284.92 (rounded to 285) [2024-10-24 21:30:52] 2024-10-25 01:28:16,142 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:28:16,142 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:28:16,145 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:28:19,051 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:28:19,052 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:28:21,557 | INFO | monitor loop #663: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:28:32,827 | INFO | (instant) CPU consumption time for pid=43342: 284.93 (rounded to 285) [2024-10-24 21:30:52] 2024-10-25 01:28:32,827 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:28:32,827 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:28:32,840 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:28:33,896 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:28:33,897 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:28:36,420 | INFO | monitor loop #664: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:28:47,708 | INFO | (instant) CPU consumption time for pid=43342: 285.56 (rounded to 286) [2024-10-24 21:30:52] 2024-10-25 01:28:47,708 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:28:47,708 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:28:47,716 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:28:48,760 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:28:48,760 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:28:51,265 | INFO | monitor loop #665: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:29:02,159 | INFO | (instant) CPU consumption time for pid=43342: 285.65999999999997 (rounded to 286) [2024-10-24 21:30:52] 2024-10-25 01:29:02,159 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:29:02,160 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:29:02,162 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:29:02,163 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:29:02,165 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2687479 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-24 21:30:52] 2024-10-25 01:29:02,166 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:29:02,166 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-24 21:30:52] 2024-10-25 01:29:02,166 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-24 21:30:52] 2024-10-25 01:29:02,166 | INFO | executing command: df -mP /tmp/boinchome/slots/0 [2024-10-24 21:30:52] 2024-10-25 01:29:02,308 | INFO | sufficient remaining disk space (101233721344 B) [2024-10-24 21:30:52] 2024-10-25 01:29:02,308 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-24 21:30:52] 2024-10-25 01:29:02,310 | INFO | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6377708675: 78744154 B (within 5767168000 B limit) [2024-10-24 21:30:52] 2024-10-25 01:29:02,310 | INFO | pfn file=/tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-24 21:30:52] 2024-10-25 01:29:02,310 | INFO | total size of present files: 0 B (workdir size: 78744154 B) [2024-10-24 21:30:52] 2024-10-25 01:29:02,310 | INFO | output file size check: skipping output file /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 since it does not exist [2024-10-24 21:30:52] 2024-10-25 01:29:03,120 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:29:03,120 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:29:05,634 | INFO | monitor loop #666: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:29:17,078 | INFO | (instant) CPU consumption time for pid=43342: 285.65999999999997 (rounded to 286) [2024-10-24 21:30:52] 2024-10-25 01:29:17,079 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:29:17,079 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:29:17,082 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:29:17,717 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:29:17,717 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:29:20,222 | INFO | monitor loop #667: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:29:23,453 | INFO | running: iteration=160 pid=43342 exit_code=None [2024-10-24 21:30:52] 2024-10-25 01:29:29,046 | INFO | 9774s have passed since pilot start [2024-10-24 21:30:52] 2024-10-25 01:29:34,434 | INFO | (instant) CPU consumption time for pid=43342: 286.39 (rounded to 286) [2024-10-24 21:30:52] 2024-10-25 01:29:34,434 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:29:34,434 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:29:34,437 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:29:35,672 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:29:35,672 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:29:38,176 | INFO | monitor loop #668: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:29:50,213 | INFO | (instant) CPU consumption time for pid=43342: 286.39 (rounded to 286) [2024-10-24 21:30:52] 2024-10-25 01:29:50,213 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:29:50,214 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:29:50,226 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:29:52,024 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:29:52,024 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:29:54,529 | INFO | monitor loop #669: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:30:08,173 | INFO | (instant) CPU consumption time for pid=43342: 287.1 (rounded to 287) [2024-10-24 21:30:52] 2024-10-25 01:30:08,173 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:30:08,173 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:30:08,181 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:30:09,250 | INFO | number of running child processes to parent process 43342: 13 [2024-10-24 21:30:52] 2024-10-25 01:30:09,251 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:30:11,762 | INFO | monitor loop #670: job 0:6377708675 is in state 'running' [2024-10-24 21:30:52] 2024-10-25 01:30:15,339 | CRITICAL | max running time (10000s) minus grace time (180s) has been exceeded - time to abort pilot [2024-10-24 21:30:52] 2024-10-25 01:30:15,339 | INFO | setting REACHED_MAXTIME and graceful stop [2024-10-24 21:30:52] 2024-10-25 01:30:15,340 | INFO | [monitor] control thread has ended [2024-10-24 21:30:52] 2024-10-25 01:30:15,949 | INFO | breaking -- sending SIGTERM to pid=43342 [2024-10-24 21:30:52] 2024-10-25 01:30:15,953 | INFO | breaking -- sleep 10 s before sending SIGKILL pid=43342 [2024-10-24 21:30:52] 2024-10-25 01:30:16,187 | INFO | all job control threads have been joined [2024-10-24 21:30:52] 2024-10-25 01:30:16,377 | INFO | all data control threads have been joined [2024-10-24 21:30:52] 2024-10-25 01:30:16,401 | INFO | [data] copytool_in thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:16,651 | INFO | [payload] validate_pre thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:16,724 | INFO | [job] create_data_payload thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:16,812 | INFO | all payload control threads have been joined [2024-10-24 21:30:52] 2024-10-25 01:30:17,193 | INFO | [job] control thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,300 | INFO | [job] retrieve thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,347 | INFO | [data] copytool_out thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,361 | INFO | [payload] failed_post thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,377 | INFO | [payload] validate_post thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,383 | INFO | [data] control thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,710 | INFO | [job] validate thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:17,818 | INFO | [payload] control thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:18,234 | WARNING | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s [2024-10-24 21:30:52] 2024-10-25 01:30:20,944 | INFO | job.realtimelogging is not enabled [2024-10-24 21:30:52] 2024-10-25 01:30:21,950 | INFO | [payload] run_realtimelog thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:22,376 | INFO | [data] queue_monitor thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:22,773 | INFO | (instant) CPU consumption time for pid=43342: 4.67 (rounded to 5) [2024-10-24 21:30:52] 2024-10-25 01:30:22,773 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:30:22,773 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:30:22,797 | INFO | neither /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json, nor /tmp/boinchome/slots/0/memory_monitor_summary.json exist [2024-10-24 21:30:52] 2024-10-25 01:30:22,797 | INFO | using path: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_output.txt (trf name=prmon) [2024-10-24 21:30:52] 2024-10-25 01:30:22,804 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2687479 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-24 21:30:52] 2024-10-25 01:30:22,805 | INFO | oom_score(pilot) = 0, oom_score(payload) = 0 [2024-10-24 21:30:52] 2024-10-25 01:30:22,805 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-24 21:30:52] 2024-10-25 01:30:22,805 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-24 21:30:52] 2024-10-25 01:30:22,806 | INFO | executing command: df -mP /tmp/boinchome/slots/0 [2024-10-24 21:30:52] 2024-10-25 01:30:22,915 | INFO | sufficient remaining disk space (101231624192 B) [2024-10-24 21:30:52] 2024-10-25 01:30:22,916 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-24 21:30:52] 2024-10-25 01:30:22,917 | INFO | size of work directory /tmp/boinchome/slots/0/PanDA_Pilot-6377708675: 78744470 B (within 5767168000 B limit) [2024-10-24 21:30:52] 2024-10-25 01:30:22,918 | INFO | pfn file=/tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-24 21:30:52] 2024-10-25 01:30:22,918 | INFO | total size of present files: 0 B (workdir size: 78744470 B) [2024-10-24 21:30:52] 2024-10-25 01:30:22,918 | INFO | output file size check: skipping output file /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/HITS.41779848._000131.pool.root.1 since it does not exist [2024-10-24 21:30:52] 2024-10-25 01:30:23,628 | INFO | number of running child processes to parent process 43342: 1 [2024-10-24 21:30:52] 2024-10-25 01:30:23,629 | INFO | maximum number of monitored processes: 16 [2024-10-24 21:30:52] 2024-10-25 01:30:23,629 | INFO | will abort loop [2024-10-24 21:30:52] 2024-10-25 01:30:24,634 | INFO | [job] job monitor thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:26,010 | INFO | [2024-10-24 21:30:52] [2024-10-24 21:30:52] finished pid=43342 exit_code=None state=failed [2024-10-24 21:30:52] [2024-10-24 21:30:52] 2024-10-25 01:30:26,010 | WARNING | detected unset exit_code from wait_graceful - reset to -1 [2024-10-24 21:30:52] 2024-10-25 01:30:26,020 | INFO | using pid=80430 to kill prmon [2024-10-24 21:30:52] 2024-10-25 01:30:26,024 | INFO | stopping utility process 'MemoryMonitor' with signal 10 [2024-10-24 21:30:52] 2024-10-25 01:30:26,024 | INFO | process 80430 no longer exists [2024-10-24 21:30:52] 2024-10-25 01:30:26,024 | INFO | utility process 43344 cleanup finished with status=True [2024-10-24 21:30:52] 2024-10-25 01:30:26,028 | INFO | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20) [2024-10-24 21:30:52] 2024-10-25 01:30:29,049 | INFO | copied /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/memory_monitor_summary.json to /tmp/boinchome/slots/0 [2024-10-24 21:30:52] 2024-10-25 01:30:29,051 | INFO | CPU consumption time: 236.48000000000002 s (rounded to 236 s) [2024-10-24 21:30:52] 2024-10-25 01:30:29,051 | WARNING | main payload execution returned non-zero exit code: -1 [2024-10-24 21:30:52] 2024-10-25 01:30:29,054 | INFO | scanning dmesg message for subprocess=82592 for memory errors [2024-10-24 21:30:52] 2024-10-25 01:30:29,055 | INFO | executing command: dmesg|grep 82592 [2024-10-24 21:30:52] 2024-10-25 01:30:29,699 | WARNING | job report does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/jobReport.json [2024-10-24 21:30:52] 2024-10-25 01:30:29,700 | WARNING | metadata does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/metadata.xml [2024-10-24 21:30:52] 2024-10-25 01:30:29,700 | WARNING | file does not exist: /tmp/boinchome/slots/0/PanDA_Pilot-6377708675/metadata.xml [2024-10-24 21:30:52] 2024-10-25 01:30:29,703 | INFO | generated guid for lfn=HITS.41779848._000131.pool.root.1: 35DD2F3C-D009-4C98-A1FF-E52668BFC65F [2024-10-24 21:30:52] 2024-10-25 01:30:29,703 | WARNING | aborting payload error diagnosis since an error has already been set: [1315, 1187] [2024-10-24 21:30:52] 2024-10-25 01:30:31,195 | INFO | [payload] execute_payloads thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:31,262 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:33,279 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:35,291 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:37,307 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:39,323 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:41,335 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:41,350 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140235485407040)>', '<ExcThread(queue_monitor, started 140234775852800)>'] [2024-10-24 21:30:52] 2024-10-25 01:30:41,350 | INFO | [job] queue monitor thread has finished [2024-10-24 21:30:52] 2024-10-25 01:30:43,347 | INFO | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140235485407040)>']) [2024-10-24 21:30:52] 2024-10-25 01:30:48,377 | INFO | all workflow threads have been joined [2024-10-24 21:30:52] 2024-10-25 01:30:48,389 | INFO | end of generic workflow (traces error code: 0) [2024-10-24 21:30:52] 2024-10-25 01:30:48,401 | INFO | traces error code: 0 [2024-10-24 21:30:52] 2024-10-25 01:30:48,417 | INFO | pilot has finished (exit code=0, shell exit code=0) [2024-10-24 21:30:52] 2024-10-25 01:30:49,850 [wrapper] ==== pilot stdout END ==== [2024-10-24 21:30:52] 2024-10-25 01:30:49,896 [wrapper] ==== wrapper stdout RESUME ==== [2024-10-24 21:30:52] 2024-10-25 01:30:49,935 [wrapper] pilotpid: 93161 [2024-10-24 21:30:52] 2024-10-25 01:30:49,984 [wrapper] Pilot exit status: 0 [2024-10-24 21:30:52] 2024-10-25 01:30:50,116 [wrapper] pandaids: 6377708675 6377708675 [2024-10-24 21:30:52] 2024-10-25 01:30:50,674 [wrapper] cleanup supervisor_pilot 49793 93162 [2024-10-24 21:30:52] 2024-10-25 01:30:50,728 [wrapper] Test setup, not cleaning [2024-10-24 21:30:52] 2024-10-25 01:30:50,806 [wrapper] ==== wrapper stdout END ==== [2024-10-24 21:30:52] 2024-10-25 01:30:50,864 [wrapper] ==== wrapper stderr END ==== [2024-10-24 21:30:52] 2024-10-25 01:30:50,977 [wrapper] apfmon messages muted [2024-10-24 21:30:52] *** Error codes and diagnostics *** [2024-10-24 21:30:52] *** Listing of results directory *** [2024-10-24 21:30:52] total 735356 [2024-10-24 21:30:52] drwx------ 4 boincer umatlas 4096 Oct 17 09:00 pilot3 [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 486088 Oct 24 16:04 pilot3.tar.gz [2024-10-24 21:30:52] -rwx------ 1 boincer umatlas 33946 Oct 24 16:35 runpilot2-wrapper.sh [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 5653 Oct 24 16:35 queuedata.json [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 100 Oct 24 16:52 wrapper_26015_x86_64-pc-linux-gnu [2024-10-24 21:30:52] -rwxr-xr-x 1 boincer umatlas 7986 Oct 24 16:52 run_atlas [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 105 Oct 24 16:52 job.xml [2024-10-24 21:30:52] -rw-r--r-- 3 boincer umatlas 373356008 Oct 24 16:52 EVNT.41779840._000003.pool.root.1 [2024-10-24 21:30:52] -rw-r--r-- 3 boincer umatlas 373356008 Oct 24 16:52 ATLAS.root_0 [2024-10-24 21:30:52] -rw-r--r-- 2 boincer umatlas 499789 Oct 24 16:52 input.tar.gz [2024-10-24 21:30:52] -rw-r--r-- 2 boincer umatlas 17569 Oct 24 16:52 start_atlas.sh [2024-10-24 21:30:52] drwxrwx--x 2 boincer umatlas 4096 Oct 24 16:52 shared [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 0 Oct 24 16:52 boinc_lockfile [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 1611897 Oct 24 16:53 agis_ddmendpoints.agis.ALL.json [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 2894 Oct 24 18:46 pandaJob.out [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 4209 Oct 24 18:46 stderr.txt [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 467 Oct 24 18:46 setup.sh.local [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 1019381 Oct 24 18:46 agis_schedconf.cvmfs.json [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 908 Oct 24 21:20 heartbeat.json [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 6388 Oct 24 21:22 init_data.xml [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 96 Oct 24 21:29 pilot_heartbeat.json [2024-10-24 21:30:52] drwxrwx--- 3 boincer umatlas 4096 Oct 24 21:30 PanDA_Pilot-6377708675 [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 1079 Oct 24 21:30 memory_monitor_summary.json [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 532 Oct 24 21:30 boinc_task_state.xml [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 809737 Oct 24 21:30 pilotlog.txt [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 824882 Oct 24 21:30 log.41779848._000131.job.log.1 [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 8192 Oct 24 21:30 boinc_mmap_file [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 28 Oct 24 21:30 wrapper_checkpoint.txt [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 571 Oct 24 21:30 runtime_log [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 839680 Oct 24 21:30 result.tar.gz [2024-10-24 21:30:52] -rw-r--r-- 1 boincer umatlas 10830 Oct 24 21:30 runtime_log.err [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 774 Oct 24 21:30 ehMMDmVjJO6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmr6XLDm02SNQo.diag [2024-10-24 21:30:52] No HITS result produced [2024-10-24 21:30:52] *** Contents of shared directory: *** [2024-10-24 21:30:52] total 365944 [2024-10-24 21:30:52] -rw-r--r-- 3 boincer umatlas 373356008 Oct 24 16:52 ATLAS.root_0 [2024-10-24 21:30:52] -rw-r--r-- 2 boincer umatlas 499789 Oct 24 16:52 input.tar.gz [2024-10-24 21:30:52] -rw-r--r-- 2 boincer umatlas 17569 Oct 24 16:52 start_atlas.sh [2024-10-24 21:30:52] -rw------- 1 boincer umatlas 839680 Oct 24 21:30 result.tar.gz 21:30:54 (66739): run_atlas exited; CPU time 265.870084 21:30:54 (66739): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN