| Name | FOJLDmFauO6n7OlckoGmwREmGgkVDo1bjSoqOlfODmEYELDmhcoRMn_0 |
| Workunit | 226345244 |
| Created | 26 Oct 2024, 11:58:28 UTC |
| Sent | 26 Oct 2024, 12:38:46 UTC |
| Report deadline | 3 Nov 2024, 12:38:46 UTC |
| Received | 26 Oct 2024, 17:39:53 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10804956 |
| Run time | 2 hours 45 min 20 sec |
| CPU time | 6 hours 4 min 45 sec |
| Validate state | Valid |
| Credit | 512.00 |
| Device peak FLOPS | 43.24 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 1.53 GB |
| Peak swap size | 31.65 GB |
| Peak disk usage | 827.22 MB |
Stderr output
<core_client_version>7.7.0</core_client_version> <![CDATA[ <stderr_txt> 10:54:20 (97428): wrapper (7.7.26015): starting 10:54:20 (97428): wrapper: running run_atlas (--nthreads 8) [2024-10-26 10:54:20] Arguments: --nthreads 8 [2024-10-26 10:54:20] Threads: 8 [2024-10-26 10:54:20] Checking for CVMFS [2024-10-26 10:54:20] Probing /cvmfs/atlas.cern.ch... OK [2024-10-26 10:54:20] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-26 10:54:20] Running cvmfs_config stat atlas.cern.ch [2024-10-26 10:54:21] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-26 10:54:21] 2.11.2.0 75320 1107 80176 138762 1 164 10754948 18432000 22161 130560 0 8471142 99.624 7081004 60962 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-26 10:54:21] CVMFS is ok [2024-10-26 10:54:21] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-26 10:54:21] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-26 10:54:21] Further information can be found at the LHC@home message board. [2024-10-26 10:54:21] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-26 10:54:21] Checking for apptainer binary... [2024-10-26 10:54:21] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-26 10:54:21] Running /usr/bin/apptainer --version [2024-10-26 10:54:21] apptainer version 1.3.2-1.el7 [2024-10-26 10:54:21] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-26 10:54:28] c-212-11.aglt2.org [2024-10-26 10:54:28] apptainer works [2024-10-26 10:54:28] Set ATHENA_PROC_NUMBER=8 [2024-10-26 10:54:28] Set ATHENA_CORE_NUMBER=8 [2024-10-26 10:54:28] Starting ATLAS job with PandaID=6379338382 [2024-10-26 10:54:28] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/2 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh [2024-10-26 13:39:35] *** The last 200 lines of the pilot log: *** [2024-10-26 13:39:35] 2024-10-26 17:35:36,447 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:35:38,951 | INFO | monitor loop #683: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:35:49,326 | INFO | (instant) CPU consumption time for pid=37938: 286.81 (rounded to 287) [2024-10-26 13:39:35] 2024-10-26 17:35:49,327 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:35:49,327 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:35:49,334 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:35:49,905 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:35:49,906 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:35:52,410 | INFO | monitor loop #684: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:36:02,926 | INFO | (instant) CPU consumption time for pid=37938: 287.44 (rounded to 287) [2024-10-26 13:39:35] 2024-10-26 17:36:02,927 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:02,927 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:02,930 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:36:03,205 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:36:03,205 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:36:05,712 | INFO | monitor loop #685: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:36:06,590 | INFO | 9648s have passed since pilot start [2024-10-26 13:39:35] 2024-10-26 17:36:16,554 | INFO | (instant) CPU consumption time for pid=37938: 287.45000000000005 (rounded to 287) [2024-10-26 13:39:35] 2024-10-26 17:36:16,554 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:16,554 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:16,557 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:36:17,382 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:36:17,383 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:36:19,887 | INFO | monitor loop #686: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:36:30,737 | INFO | (instant) CPU consumption time for pid=37938: 288.11 (rounded to 288) [2024-10-26 13:39:35] 2024-10-26 17:36:30,738 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:30,738 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:30,745 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:36:30,746 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-26 13:39:35] 2024-10-26 17:36:30,746 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6611 B [2024-10-26 13:39:35] 2024-10-26 17:36:30,746 | INFO | executing command: df -mP /tmp/boinchome/slots/2 [2024-10-26 13:39:35] 2024-10-26 17:36:30,848 | INFO | sufficient remaining disk space (101600722944 B) [2024-10-26 13:39:35] 2024-10-26 17:36:30,848 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-26 13:39:35] 2024-10-26 17:36:30,850 | INFO | size of work directory /tmp/boinchome/slots/2/PanDA_Pilot-6379338382: 89836610 B (within 5767168000 B limit) [2024-10-26 13:39:35] 2024-10-26 17:36:30,850 | INFO | pfn file=/tmp/boinchome/slots/2/PanDA_Pilot-6379338382/HITS.41779941._001885.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-26 13:39:35] 2024-10-26 17:36:30,850 | INFO | total size of present files: 0 B (workdir size: 89836610 B) [2024-10-26 13:39:35] 2024-10-26 17:36:30,850 | INFO | output file size check: skipping output file /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/HITS.41779941._001885.pool.root.1 since it does not exist [2024-10-26 13:39:35] 2024-10-26 17:36:31,348 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:36:31,349 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:36:33,853 | INFO | monitor loop #687: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:36:44,468 | INFO | (instant) CPU consumption time for pid=37938: 288.12 (rounded to 288) [2024-10-26 13:39:35] 2024-10-26 17:36:44,468 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:44,468 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:44,472 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:44,472 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:44,475 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2662882 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-26 13:39:35] 2024-10-26 17:36:44,476 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:36:45,061 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:36:45,062 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:36:47,566 | INFO | monitor loop #688: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:36:58,080 | INFO | (instant) CPU consumption time for pid=37938: 288.12 (rounded to 288) [2024-10-26 13:39:35] 2024-10-26 17:36:58,080 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:36:58,081 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:36:58,083 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:36:58,448 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:36:58,449 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:37:00,955 | INFO | monitor loop #689: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:37:11,672 | INFO | (instant) CPU consumption time for pid=37938: 288.78999999999996 (rounded to 289) [2024-10-26 13:39:35] 2024-10-26 17:37:11,672 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:37:11,672 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:37:11,676 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:37:12,016 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:37:12,017 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:37:14,521 | INFO | monitor loop #690: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:37:25,313 | INFO | (instant) CPU consumption time for pid=37938: 288.78999999999996 (rounded to 289) [2024-10-26 13:39:35] 2024-10-26 17:37:25,314 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:37:25,314 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:37:25,321 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:37:26,850 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:37:26,850 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:37:29,358 | INFO | monitor loop #691: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:37:40,355 | INFO | (instant) CPU consumption time for pid=37938: 289.5 (rounded to 290) [2024-10-26 13:39:35] 2024-10-26 17:37:40,355 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:37:40,356 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:37:40,363 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:37:41,054 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:37:41,055 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:37:43,559 | INFO | monitor loop #692: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:37:54,535 | INFO | (instant) CPU consumption time for pid=37938: 289.51 (rounded to 290) [2024-10-26 13:39:35] 2024-10-26 17:37:54,535 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:37:54,536 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:37:54,563 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:37:54,569 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-26 13:39:35] 2024-10-26 17:37:54,569 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6611 B [2024-10-26 13:39:35] 2024-10-26 17:37:54,569 | INFO | executing command: df -mP /tmp/boinchome/slots/2 [2024-10-26 13:39:35] 2024-10-26 17:37:54,745 | INFO | sufficient remaining disk space (101597577216 B) [2024-10-26 13:39:35] 2024-10-26 17:37:54,746 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-26 13:39:35] 2024-10-26 17:37:54,747 | INFO | size of work directory /tmp/boinchome/slots/2/PanDA_Pilot-6379338382: 90714383 B (within 5767168000 B limit) [2024-10-26 13:39:35] 2024-10-26 17:37:54,748 | INFO | pfn file=/tmp/boinchome/slots/2/PanDA_Pilot-6379338382/HITS.41779941._001885.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-26 13:39:35] 2024-10-26 17:37:54,748 | INFO | total size of present files: 0 B (workdir size: 90714383 B) [2024-10-26 13:39:35] 2024-10-26 17:37:54,748 | INFO | output file size check: skipping output file /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/HITS.41779941._001885.pool.root.1 since it does not exist [2024-10-26 13:39:35] 2024-10-26 17:37:55,014 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:37:55,015 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:37:57,519 | INFO | monitor loop #693: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:38:07,177 | INFO | 9769s have passed since pilot start [2024-10-26 13:39:35] 2024-10-26 17:38:08,526 | INFO | (instant) CPU consumption time for pid=37938: 289.51 (rounded to 290) [2024-10-26 13:39:35] 2024-10-26 17:38:08,527 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:38:08,527 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:38:08,530 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:38:08,530 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:38:08,534 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2662882 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-26 13:39:35] 2024-10-26 17:38:08,534 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:38:08,976 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:38:08,976 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:38:11,485 | INFO | monitor loop #694: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:38:22,260 | INFO | (instant) CPU consumption time for pid=37938: 290.15999999999997 (rounded to 290) [2024-10-26 13:39:35] 2024-10-26 17:38:22,260 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:38:22,260 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:38:22,270 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:38:23,012 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:38:23,012 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:38:25,517 | INFO | monitor loop #695: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:38:36,262 | INFO | (instant) CPU consumption time for pid=37938: 290.15999999999997 (rounded to 290) [2024-10-26 13:39:35] 2024-10-26 17:38:36,263 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:38:36,263 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:38:36,272 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:38:36,908 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:38:36,919 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:38:39,424 | INFO | monitor loop #696: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:38:50,890 | INFO | (instant) CPU consumption time for pid=37938: 290.81000000000006 (rounded to 291) [2024-10-26 13:39:35] 2024-10-26 17:38:50,896 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:38:50,896 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:38:50,899 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 13:39:35] 2024-10-26 17:38:51,614 | INFO | number of running child processes to parent process 37938: 13 [2024-10-26 13:39:35] 2024-10-26 17:38:51,615 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:38:52,230 | INFO | running: iteration=160 pid=37938 exit_code=None [2024-10-26 13:39:35] 2024-10-26 17:38:54,119 | INFO | monitor loop #697: job 0:6379338382 is in state 'running' [2024-10-26 13:39:35] 2024-10-26 17:38:59,396 | CRITICAL | max running time (10000s) minus grace time (180s) has been exceeded - time to abort pilot [2024-10-26 13:39:35] 2024-10-26 17:38:59,397 | INFO | setting REACHED_MAXTIME and graceful stop [2024-10-26 13:39:35] 2024-10-26 17:38:59,428 | INFO | [monitor] control thread has ended [2024-10-26 13:39:35] 2024-10-26 17:38:59,508 | INFO | all job control threads have been joined [2024-10-26 13:39:35] 2024-10-26 17:38:59,523 | INFO | all data control threads have been joined [2024-10-26 13:39:35] 2024-10-26 17:38:59,926 | WARNING | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s [2024-10-26 13:39:35] 2024-10-26 17:38:59,936 | INFO | all payload control threads have been joined [2024-10-26 13:39:35] 2024-10-26 17:39:00,384 | INFO | breaking -- sending SIGTERM to pid=37938 [2024-10-26 13:39:35] 2024-10-26 17:39:00,389 | INFO | breaking -- sleep 10 s before sending SIGKILL pid=37938 [2024-10-26 13:39:35] 2024-10-26 17:39:00,514 | INFO | [payload] failed_post thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:00,519 | INFO | [job] control thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:00,534 | INFO | [data] control thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:00,955 | INFO | [payload] control thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,007 | INFO | [data] copytool_in thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,019 | INFO | [job] retrieve thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,202 | INFO | [job] create_data_payload thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,447 | INFO | [payload] validate_post thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,759 | INFO | [job] validate thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:01,949 | INFO | [payload] validate_pre thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:02,251 | INFO | [data] copytool_out thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:03,512 | INFO | [data] queue_monitor thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:04,584 | INFO | (instant) CPU consumption time for pid=37938: 5.039999999999999 (rounded to 5) [2024-10-26 13:39:35] 2024-10-26 17:39:04,584 | INFO | neither /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json, nor /tmp/boinchome/slots/2/memory_monitor_summary.json exist [2024-10-26 13:39:35] 2024-10-26 17:39:04,584 | INFO | using path: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_output.txt (trf name=prmon) [2024-10-26 13:39:35] 2024-10-26 17:39:04,588 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 0 [2024-10-26 13:39:35] 2024-10-26 17:39:04,861 | INFO | number of running child processes to parent process 37938: 1 [2024-10-26 13:39:35] 2024-10-26 17:39:04,861 | INFO | maximum number of monitored processes: 17 [2024-10-26 13:39:35] 2024-10-26 17:39:04,862 | INFO | will abort loop [2024-10-26 13:39:35] 2024-10-26 17:39:05,867 | INFO | [job] job monitor thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:06,780 | INFO | job.realtimelogging is not enabled [2024-10-26 13:39:35] 2024-10-26 17:39:07,794 | INFO | [payload] run_realtimelog thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:10,476 | INFO | [2024-10-26 13:39:35] [2024-10-26 13:39:35] finished pid=37938 exit_code=None state=failed [2024-10-26 13:39:35] [2024-10-26 13:39:35] 2024-10-26 17:39:10,476 | WARNING | detected unset exit_code from wait_graceful - reset to -1 [2024-10-26 13:39:35] 2024-10-26 17:39:10,543 | INFO | using pid=58428 to kill prmon [2024-10-26 13:39:35] 2024-10-26 17:39:10,544 | INFO | stopping utility process 'MemoryMonitor' with signal 10 [2024-10-26 13:39:35] 2024-10-26 17:39:10,544 | INFO | process 58428 no longer exists [2024-10-26 13:39:35] 2024-10-26 17:39:10,544 | INFO | utility process 37950 cleanup finished with status=True [2024-10-26 13:39:35] 2024-10-26 17:39:10,544 | INFO | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20) [2024-10-26 13:39:35] 2024-10-26 17:39:13,609 | INFO | copied /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/memory_monitor_summary.json to /tmp/boinchome/slots/2 [2024-10-26 13:39:35] 2024-10-26 17:39:13,623 | INFO | CPU consumption time: 225.40999999999997 s (rounded to 225 s) [2024-10-26 13:39:35] 2024-10-26 17:39:13,626 | WARNING | main payload execution returned non-zero exit code: -1 [2024-10-26 13:39:35] 2024-10-26 17:39:13,665 | INFO | scanning dmesg message for subprocess=65705 for memory errors [2024-10-26 13:39:35] 2024-10-26 17:39:13,665 | INFO | executing command: dmesg|grep 65705 [2024-10-26 13:39:35] 2024-10-26 17:39:14,401 | WARNING | job report does not exist: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/jobReport.json [2024-10-26 13:39:35] 2024-10-26 17:39:14,402 | WARNING | metadata does not exist: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/metadata.xml [2024-10-26 13:39:35] 2024-10-26 17:39:14,403 | WARNING | file does not exist: /tmp/boinchome/slots/2/PanDA_Pilot-6379338382/metadata.xml [2024-10-26 13:39:35] 2024-10-26 17:39:14,405 | INFO | generated guid for lfn=HITS.41779941._001885.pool.root.1: 4CD73A83-35DA-4E63-A10B-C359977B0DE9 [2024-10-26 13:39:35] 2024-10-26 17:39:14,405 | WARNING | aborting payload error diagnosis since an error has already been set: [1315, 1187] [2024-10-26 13:39:35] 2024-10-26 17:39:16,174 | INFO | [payload] execute_payloads thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:16,396 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140184264464192)>', '<ExcThread(queue_monitor, started 140183789868800)>'] [2024-10-26 13:39:35] 2024-10-26 17:39:18,410 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140184264464192)>', '<ExcThread(queue_monitor, started 140183789868800)>'] [2024-10-26 13:39:35] 2024-10-26 17:39:20,423 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140184264464192)>', '<ExcThread(queue_monitor, started 140183789868800)>'] [2024-10-26 13:39:35] 2024-10-26 17:39:22,435 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140184264464192)>', '<ExcThread(queue_monitor, started 140183789868800)>'] [2024-10-26 13:39:35] 2024-10-26 17:39:23,040 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140184264464192)>', '<ExcThread(queue_monitor, started 140183789868800)>'] [2024-10-26 13:39:35] 2024-10-26 17:39:23,040 | INFO | [job] queue monitor thread has finished [2024-10-26 13:39:35] 2024-10-26 17:39:24,449 | INFO | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140184264464192)>']) [2024-10-26 13:39:35] 2024-10-26 17:39:29,523 | INFO | all workflow threads have been joined [2024-10-26 13:39:35] 2024-10-26 17:39:29,535 | INFO | end of generic workflow (traces error code: 0) [2024-10-26 13:39:35] 2024-10-26 17:39:29,592 | INFO | traces error code: 0 [2024-10-26 13:39:35] 2024-10-26 17:39:29,594 | INFO | pilot has finished (exit code=0, shell exit code=0) [2024-10-26 13:39:35] 2024-10-26 17:39:33,565 [wrapper] ==== pilot stdout END ==== [2024-10-26 13:39:35] 2024-10-26 17:39:33,573 [wrapper] ==== wrapper stdout RESUME ==== [2024-10-26 13:39:35] 2024-10-26 17:39:33,614 [wrapper] pilotpid: 10089 [2024-10-26 13:39:35] 2024-10-26 17:39:33,661 [wrapper] Pilot exit status: 0 [2024-10-26 13:39:35] 2024-10-26 17:39:33,803 [wrapper] pandaids: 6379338382 [2024-10-26 13:39:35] 2024-10-26 17:39:34,063 [wrapper] cleanup supervisor_pilot 92294 10090 [2024-10-26 13:39:35] 2024-10-26 17:39:34,110 [wrapper] Test setup, not cleaning [2024-10-26 13:39:35] 2024-10-26 17:39:34,143 [wrapper] ==== wrapper stdout END ==== [2024-10-26 13:39:35] 2024-10-26 17:39:34,169 [wrapper] ==== wrapper stderr END ==== [2024-10-26 13:39:35] 2024-10-26 17:39:34,262 [wrapper] apfmon messages muted [2024-10-26 13:39:35] *** Error codes and diagnostics *** [2024-10-26 13:39:35] *** Listing of results directory *** [2024-10-26 13:39:35] total 381496 [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 5653 Oct 26 07:55 queuedata.json [2024-10-26 13:39:35] -rwx------ 1 boincer umatlas 34589 Oct 26 07:56 runpilot2-wrapper.sh [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 486088 Oct 26 07:56 pilot3.tar.gz [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 100 Oct 26 10:54 wrapper_26015_x86_64-pc-linux-gnu [2024-10-26 13:39:35] -rwxr-xr-x 1 boincer umatlas 7986 Oct 26 10:54 run_atlas [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 105 Oct 26 10:54 job.xml [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 384286689 Oct 26 10:54 EVNT.41779939._000012.pool.root.1 [2024-10-26 13:39:35] drwxrwx--x 2 boincer umatlas 4096 Oct 26 10:54 shared [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 499914 Oct 26 10:54 input.tar.gz [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 17569 Oct 26 10:54 start_atlas.sh [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 0 Oct 26 10:54 boinc_lockfile [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 2888 Oct 26 10:54 pandaJob.out [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 467 Oct 26 10:54 setup.sh.local [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 1018694 Oct 26 10:55 agis_schedconf.cvmfs.json [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 1611897 Oct 26 10:55 agis_ddmendpoints.agis.ALL.json [2024-10-26 13:39:35] drwx------ 4 boincer umatlas 4096 Oct 26 10:55 pilot3 [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 911 Oct 26 13:29 heartbeat.json [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 6387 Oct 26 13:37 init_data.xml [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 95 Oct 26 13:38 pilot_heartbeat.json [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 530 Oct 26 13:39 boinc_task_state.xml [2024-10-26 13:39:35] drwxrwx--- 3 boincer umatlas 4096 Oct 26 13:39 PanDA_Pilot-6379338382 [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 1092 Oct 26 13:39 memory_monitor_summary.json [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 830268 Oct 26 13:39 pilotlog.txt [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 847273 Oct 26 13:39 log.41779941._001885.job.log.1 [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 571 Oct 26 13:39 runtime_log [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 860160 Oct 26 13:39 result.tar.gz [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 10745 Oct 26 13:39 runtime_log.err [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 649 Oct 26 13:39 FOJLDmFauO6n7OlckoGmwREmGgkVDo1bjSoqOlfODmEYELDmhcoRMn.diag [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 8192 Oct 26 13:39 boinc_mmap_file [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 27 Oct 26 13:39 wrapper_checkpoint.txt [2024-10-26 13:39:35] -rw-r--r-- 1 boincer umatlas 27942 Oct 26 13:39 stderr.txt [2024-10-26 13:39:35] No HITS result produced [2024-10-26 13:39:35] *** Contents of shared directory: *** [2024-10-26 13:39:35] total 376636 [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 384286689 Oct 26 10:54 ATLAS.root_0 [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 499914 Oct 26 10:54 input.tar.gz [2024-10-26 13:39:35] -rw-r--r-- 2 boincer umatlas 17569 Oct 26 10:54 start_atlas.sh [2024-10-26 13:39:35] -rw------- 1 boincer umatlas 860160 Oct 26 13:39 result.tar.gz 13:39:37 (97428): run_atlas exited; CPU time 259.552916 13:39:37 (97428): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN