| Name | 9khNDmNT4O6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmdWmLDmkgI4Do_0 |
| Workunit | 226372212 |
| Created | 26 Oct 2024, 22:30:38 UTC |
| Sent | 26 Oct 2024, 23:08:56 UTC |
| Report deadline | 3 Nov 2024, 23:08:56 UTC |
| Received | 27 Oct 2024, 1:55:09 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10804956 |
| Run time | 2 hours 45 min 8 sec |
| CPU time | 4 hours 53 min 15 sec |
| Validate state | Valid |
| Credit | 465.92 |
| Device peak FLOPS | 43.24 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 1.53 GB |
| Peak swap size | 2.18 GB |
| Peak disk usage | 748.71 MB |
Stderr output
<core_client_version>7.7.0</core_client_version> <![CDATA[ <stderr_txt> 19:09:45 (38704): wrapper (7.7.26015): starting 19:09:45 (38704): wrapper: running run_atlas (--nthreads 8) [2024-10-26 19:09:45] Arguments: --nthreads 8 [2024-10-26 19:09:45] Threads: 8 [2024-10-26 19:09:45] Checking for CVMFS [2024-10-26 19:09:46] Probing /cvmfs/atlas.cern.ch... OK [2024-10-26 19:09:46] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-26 19:09:46] Running cvmfs_config stat atlas.cern.ch [2024-10-26 19:09:47] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-26 19:09:47] 2.11.2.0 75320 1602 87672 138774 2 358 11433937 18432001 22917 130560 0 12364423 99.466 13153334 59653 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-26 19:09:47] CVMFS is ok [2024-10-26 19:09:47] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-26 19:09:47] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-26 19:09:47] Further information can be found at the LHC@home message board. [2024-10-26 19:09:47] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-26 19:09:47] Checking for apptainer binary... [2024-10-26 19:09:47] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-26 19:09:47] Running /usr/bin/apptainer --version [2024-10-26 19:09:47] apptainer version 1.3.2-1.el7 [2024-10-26 19:09:47] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-26 19:09:49] c-212-11.aglt2.org [2024-10-26 19:09:49] apptainer works [2024-10-26 19:09:49] Set ATHENA_PROC_NUMBER=8 [2024-10-26 19:09:49] Set ATHENA_CORE_NUMBER=8 [2024-10-26 19:09:50] Starting ATLAS job with PandaID=6379826799 [2024-10-26 19:09:50] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/4 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh [2024-10-26 21:54:50] *** The last 200 lines of the pilot log: *** [2024-10-26 21:54:50] 2024-10-27 01:51:16,429 | INFO | monitor loop #687: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:51:24,848 | INFO | 9651s have passed since pilot start [2024-10-26 21:54:50] 2024-10-27 01:51:26,917 | INFO | (instant) CPU consumption time for pid=87445: 289.40999999999997 (rounded to 289) [2024-10-26 21:54:50] 2024-10-27 01:51:26,918 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:51:26,918 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:51:26,921 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:51:26,922 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:51:26,925 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2667706 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-26 21:54:50] 2024-10-27 01:51:26,925 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:51:26,926 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651230 B [2024-10-26 21:54:50] 2024-10-27 01:51:26,926 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6611 B [2024-10-26 21:54:50] 2024-10-27 01:51:26,926 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-26 21:54:50] 2024-10-27 01:51:27,054 | INFO | sufficient remaining disk space (101956190208 B) [2024-10-26 21:54:50] 2024-10-27 01:51:27,054 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-26 21:54:50] 2024-10-27 01:51:27,056 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6379826799: 71190293 B (within 5767168000 B limit) [2024-10-26 21:54:50] 2024-10-27 01:51:27,057 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-26 21:54:50] 2024-10-27 01:51:27,057 | INFO | total size of present files: 0 B (workdir size: 71190293 B) [2024-10-26 21:54:50] 2024-10-27 01:51:27,057 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 since it does not exist [2024-10-26 21:54:50] 2024-10-27 01:51:27,711 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:51:27,712 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:51:30,215 | INFO | monitor loop #688: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:51:41,520 | INFO | (instant) CPU consumption time for pid=87445: 289.41999999999996 (rounded to 289) [2024-10-26 21:54:50] 2024-10-27 01:51:41,521 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:51:41,521 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:51:41,537 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:51:42,551 | INFO | number of running child processes to parent process 87445: 14 [2024-10-26 21:54:50] 2024-10-27 01:51:42,552 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:51:45,055 | INFO | monitor loop #689: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:51:55,799 | INFO | (instant) CPU consumption time for pid=87445: 290.14 (rounded to 290) [2024-10-26 21:54:50] 2024-10-27 01:51:55,799 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:51:55,799 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:51:55,803 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:51:56,144 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:51:56,144 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:51:58,656 | INFO | monitor loop #690: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:52:10,277 | INFO | (instant) CPU consumption time for pid=87445: 290.14 (rounded to 290) [2024-10-26 21:54:50] 2024-10-27 01:52:10,278 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:52:10,278 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:52:10,313 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:52:11,186 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:52:11,187 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:52:13,702 | INFO | monitor loop #691: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:52:24,566 | INFO | (instant) CPU consumption time for pid=87445: 290.83 (rounded to 291) [2024-10-26 21:54:50] 2024-10-27 01:52:24,567 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:52:24,567 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:52:24,570 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:52:25,496 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:52:25,497 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:52:28,002 | INFO | monitor loop #692: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:52:38,813 | INFO | (instant) CPU consumption time for pid=87445: 290.83 (rounded to 291) [2024-10-26 21:54:50] 2024-10-27 01:52:38,813 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:52:38,814 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:52:38,817 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:52:38,818 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:52:38,821 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2669938 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-26 21:54:50] 2024-10-27 01:52:38,822 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:52:38,822 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651230 B [2024-10-26 21:54:50] 2024-10-27 01:52:38,822 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6611 B [2024-10-26 21:54:50] 2024-10-27 01:52:38,822 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-26 21:54:50] 2024-10-27 01:52:38,978 | INFO | sufficient remaining disk space (102327386112 B) [2024-10-26 21:54:50] 2024-10-27 01:52:38,978 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-26 21:54:50] 2024-10-27 01:52:38,981 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6379826799: 71190615 B (within 5767168000 B limit) [2024-10-26 21:54:50] 2024-10-27 01:52:38,981 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-26 21:54:50] 2024-10-27 01:52:38,981 | INFO | total size of present files: 0 B (workdir size: 71190615 B) [2024-10-26 21:54:50] 2024-10-27 01:52:38,981 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 since it does not exist [2024-10-26 21:54:50] 2024-10-27 01:52:40,151 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:52:40,151 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:52:42,657 | INFO | monitor loop #693: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:52:53,328 | INFO | (instant) CPU consumption time for pid=87445: 291.27 (rounded to 291) [2024-10-26 21:54:50] 2024-10-27 01:52:53,332 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:52:53,333 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:52:53,340 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:52:53,932 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:52:53,932 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:52:56,446 | INFO | monitor loop #694: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:53:07,845 | INFO | (instant) CPU consumption time for pid=87445: 291.62 (rounded to 292) [2024-10-26 21:54:50] 2024-10-27 01:53:07,845 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:53:07,845 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:53:07,857 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:53:08,828 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:53:08,829 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:53:11,337 | INFO | monitor loop #695: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:53:22,329 | INFO | (instant) CPU consumption time for pid=87445: 291.62 (rounded to 292) [2024-10-26 21:54:50] 2024-10-27 01:53:22,329 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:53:22,329 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:53:22,333 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:53:23,817 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:53:23,818 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:53:25,692 | INFO | 9772s have passed since pilot start [2024-10-26 21:54:50] 2024-10-27 01:53:26,322 | INFO | monitor loop #696: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:53:37,357 | INFO | (instant) CPU consumption time for pid=87445: 292.34 (rounded to 292) [2024-10-26 21:54:50] 2024-10-27 01:53:37,357 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:53:37,358 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:53:37,361 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:53:37,919 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:53:37,919 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:53:40,424 | INFO | monitor loop #697: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:53:51,713 | INFO | (instant) CPU consumption time for pid=87445: 292.34999999999997 (rounded to 292) [2024-10-26 21:54:50] 2024-10-27 01:53:51,714 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:53:51,714 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:53:51,724 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:53:51,724 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:53:51,727 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2672447 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-26 21:54:50] 2024-10-27 01:53:51,728 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:53:51,728 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651230 B [2024-10-26 21:54:50] 2024-10-27 01:53:51,729 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6611 B [2024-10-26 21:54:50] 2024-10-27 01:53:51,729 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-26 21:54:50] 2024-10-27 01:53:51,843 | INFO | sufficient remaining disk space (102025396224 B) [2024-10-26 21:54:50] 2024-10-27 01:53:51,844 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-26 21:54:50] 2024-10-27 01:53:51,845 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6379826799: 71652550 B (within 5767168000 B limit) [2024-10-26 21:54:50] 2024-10-27 01:53:51,846 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-26 21:54:50] 2024-10-27 01:53:51,846 | INFO | total size of present files: 0 B (workdir size: 71652550 B) [2024-10-26 21:54:50] 2024-10-27 01:53:51,846 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/HITS.41780302._000824.pool.root.1 since it does not exist [2024-10-26 21:54:50] 2024-10-27 01:53:52,556 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:53:52,557 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:53:55,065 | INFO | monitor loop #698: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:54:05,847 | INFO | (instant) CPU consumption time for pid=87445: 293.03999999999996 (rounded to 293) [2024-10-26 21:54:50] 2024-10-27 01:54:05,847 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:54:05,847 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:54:05,851 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-26 21:54:50] 2024-10-27 01:54:06,504 | INFO | number of running child processes to parent process 87445: 13 [2024-10-26 21:54:50] 2024-10-27 01:54:06,526 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:54:09,032 | INFO | monitor loop #699: job 0:6379826799 is in state 'running' [2024-10-26 21:54:50] 2024-10-27 01:54:10,027 | INFO | running: iteration=160 pid=87445 exit_code=None [2024-10-26 21:54:50] 2024-10-27 01:54:15,921 | CRITICAL | max running time (10000s) minus grace time (180s) has been exceeded - time to abort pilot [2024-10-26 21:54:50] 2024-10-27 01:54:15,922 | INFO | setting REACHED_MAXTIME and graceful stop [2024-10-26 21:54:50] 2024-10-27 01:54:15,925 | INFO | [monitor] control thread has ended [2024-10-26 21:54:50] 2024-10-27 01:54:16,165 | INFO | breaking -- sending SIGTERM to pid=87445 [2024-10-26 21:54:50] 2024-10-27 01:54:16,169 | INFO | breaking -- sleep 10 s before sending SIGKILL pid=87445 [2024-10-26 21:54:50] 2024-10-27 01:54:16,777 | INFO | all payload control threads have been joined [2024-10-26 21:54:50] 2024-10-27 01:54:16,982 | INFO | all job control threads have been joined [2024-10-26 21:54:50] 2024-10-27 01:54:17,005 | INFO | all data control threads have been joined [2024-10-26 21:54:50] 2024-10-27 01:54:17,057 | INFO | [payload] validate_post thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,146 | INFO | [data] copytool_in thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,285 | INFO | [job] retrieve thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,587 | INFO | job.realtimelogging is not enabled [2024-10-26 21:54:50] 2024-10-27 01:54:17,782 | INFO | [payload] control thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,790 | INFO | [payload] validate_pre thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,920 | INFO | [job] validate thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:17,988 | INFO | [job] control thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:18,011 | INFO | [data] control thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:18,178 | WARNING | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s [2024-10-26 21:54:50] 2024-10-27 01:54:18,268 | INFO | [job] create_data_payload thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:18,447 | INFO | [payload] failed_post thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:18,593 | INFO | [payload] run_realtimelog thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:18,909 | INFO | [data] copytool_out thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:19,694 | INFO | (instant) CPU consumption time for pid=87445: 4.75 (rounded to 5) [2024-10-26 21:54:50] 2024-10-27 01:54:19,694 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-26 21:54:50] 2024-10-27 01:54:19,694 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_output.txt (trf name=prmon) [2024-10-26 21:54:50] 2024-10-27 01:54:19,698 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 0 [2024-10-26 21:54:50] 2024-10-27 01:54:20,341 | INFO | number of running child processes to parent process 87445: 1 [2024-10-26 21:54:50] 2024-10-27 01:54:20,341 | INFO | maximum number of monitored processes: 16 [2024-10-26 21:54:50] 2024-10-27 01:54:20,341 | INFO | will abort loop [2024-10-26 21:54:50] 2024-10-27 01:54:21,347 | INFO | [job] job monitor thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:22,489 | INFO | [data] queue_monitor thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:26,221 | INFO | [2024-10-26 21:54:50] [2024-10-26 21:54:50] finished pid=87445 exit_code=None state=failed [2024-10-26 21:54:50] [2024-10-26 21:54:50] 2024-10-27 01:54:26,221 | WARNING | detected unset exit_code from wait_graceful - reset to -1 [2024-10-26 21:54:50] 2024-10-27 01:54:26,228 | INFO | using pid=5005 to kill prmon [2024-10-26 21:54:50] 2024-10-27 01:54:26,229 | INFO | stopping utility process 'MemoryMonitor' with signal 10 [2024-10-26 21:54:50] 2024-10-27 01:54:26,229 | INFO | process 5005 no longer exists [2024-10-26 21:54:50] 2024-10-27 01:54:26,229 | INFO | utility process 87453 cleanup finished with status=True [2024-10-26 21:54:50] 2024-10-27 01:54:26,229 | INFO | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20) [2024-10-26 21:54:50] 2024-10-27 01:54:29,244 | INFO | copied /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/memory_monitor_summary.json to /tmp/boinchome/slots/4 [2024-10-26 21:54:50] 2024-10-27 01:54:29,244 | INFO | CPU consumption time: 229.92000000000002 s (rounded to 230 s) [2024-10-26 21:54:50] 2024-10-27 01:54:29,255 | WARNING | main payload execution returned non-zero exit code: -1 [2024-10-26 21:54:50] 2024-10-27 01:54:29,259 | INFO | scanning dmesg message for subprocess=5380 for memory errors [2024-10-26 21:54:50] 2024-10-27 01:54:29,259 | INFO | executing command: dmesg|grep 5380 [2024-10-26 21:54:50] 2024-10-27 01:54:29,586 | INFO | scanning dmesg message for subprocess=5382 for memory errors [2024-10-26 21:54:50] 2024-10-27 01:54:29,587 | INFO | executing command: dmesg|grep 5382 [2024-10-26 21:54:50] 2024-10-27 01:54:29,902 | WARNING | job report does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/jobReport.json [2024-10-26 21:54:50] 2024-10-27 01:54:29,902 | WARNING | metadata does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/metadata.xml [2024-10-26 21:54:50] 2024-10-27 01:54:29,902 | WARNING | file does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6379826799/metadata.xml [2024-10-26 21:54:50] 2024-10-27 01:54:29,902 | INFO | generated guid for lfn=HITS.41780302._000824.pool.root.1: 0FDDDA78-6E20-477A-A51A-57BBFD687AE6 [2024-10-26 21:54:50] 2024-10-27 01:54:29,903 | WARNING | aborting payload error diagnosis since an error has already been set: [1315, 1187] [2024-10-26 21:54:50] 2024-10-27 01:54:31,262 | INFO | [payload] execute_payloads thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:33,147 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:35,159 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:37,173 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:39,185 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:41,197 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:41,290 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140208265267008)>', '<ExcThread(queue_monitor, started 140207957452544)>'] [2024-10-26 21:54:50] 2024-10-27 01:54:41,291 | INFO | [job] queue monitor thread has finished [2024-10-26 21:54:50] 2024-10-27 01:54:43,210 | INFO | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140208265267008)>']) [2024-10-26 21:54:50] 2024-10-27 01:54:48,235 | INFO | all workflow threads have been joined [2024-10-26 21:54:50] 2024-10-27 01:54:48,235 | INFO | end of generic workflow (traces error code: 0) [2024-10-26 21:54:50] 2024-10-27 01:54:48,236 | INFO | traces error code: 0 [2024-10-26 21:54:50] 2024-10-27 01:54:48,236 | INFO | pilot has finished (exit code=0, shell exit code=0) [2024-10-26 21:54:50] 2024-10-27 01:54:48,463 [wrapper] ==== pilot stdout END ==== [2024-10-26 21:54:50] 2024-10-27 01:54:48,468 [wrapper] ==== wrapper stdout RESUME ==== [2024-10-26 21:54:50] 2024-10-27 01:54:48,509 [wrapper] pilotpid: 56249 [2024-10-26 21:54:50] 2024-10-27 01:54:48,541 [wrapper] Pilot exit status: 0 [2024-10-26 21:54:50] 2024-10-27 01:54:48,848 [wrapper] pandaids: 6379826799 [2024-10-26 21:54:50] 2024-10-27 01:54:49,266 [wrapper] cleanup supervisor_pilot 77296 56250 [2024-10-26 21:54:50] 2024-10-27 01:54:49,307 [wrapper] Test setup, not cleaning [2024-10-26 21:54:50] 2024-10-27 01:54:49,332 [wrapper] ==== wrapper stdout END ==== [2024-10-26 21:54:50] 2024-10-27 01:54:49,369 [wrapper] ==== wrapper stderr END ==== [2024-10-26 21:54:50] 2024-10-27 01:54:49,374 [wrapper] apfmon messages muted [2024-10-26 21:54:50] *** Error codes and diagnostics *** [2024-10-26 21:54:50] *** Listing of results directory *** [2024-10-26 21:54:50] total 350408 [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 486088 Oct 26 18:22 pilot3.tar.gz [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 5653 Oct 26 18:29 queuedata.json [2024-10-26 21:54:50] -rwx------ 1 boincer umatlas 34589 Oct 26 18:29 runpilot2-wrapper.sh [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 100 Oct 26 19:09 wrapper_26015_x86_64-pc-linux-gnu [2024-10-26 21:54:50] -rwxr-xr-x 1 boincer umatlas 7986 Oct 26 19:09 run_atlas [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 105 Oct 26 19:09 job.xml [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 352446235 Oct 26 19:09 EVNT.41780299._000013.pool.root.1 [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 499906 Oct 26 19:09 input.tar.gz [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 17569 Oct 26 19:09 start_atlas.sh [2024-10-26 21:54:50] drwxrwx--x 2 boincer umatlas 4096 Oct 26 19:09 shared [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 0 Oct 26 19:09 boinc_lockfile [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 2856 Oct 26 19:09 pandaJob.out [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 467 Oct 26 19:09 setup.sh.local [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 1022139 Oct 26 19:10 agis_schedconf.cvmfs.json [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 1611897 Oct 26 19:10 agis_ddmendpoints.agis.ALL.json [2024-10-26 21:54:50] drwx------ 4 boincer umatlas 4096 Oct 26 19:10 pilot3 [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 909 Oct 26 21:44 heartbeat.json [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 6387 Oct 26 21:49 init_data.xml [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 95 Oct 26 21:53 pilot_heartbeat.json [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 529 Oct 26 21:54 boinc_task_state.xml [2024-10-26 21:54:50] drwxrwx--- 3 boincer umatlas 4096 Oct 26 21:54 PanDA_Pilot-6379826799 [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 1085 Oct 26 21:54 memory_monitor_summary.json [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 8192 Oct 26 21:54 boinc_mmap_file [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 27 Oct 26 21:54 wrapper_checkpoint.txt [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 829356 Oct 26 21:54 pilotlog.txt [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 846305 Oct 26 21:54 log.41780302._000824.job.log.1 [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 571 Oct 26 21:54 runtime_log [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 860160 Oct 26 21:54 result.tar.gz [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 10745 Oct 26 21:54 runtime_log.err [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 649 Oct 26 21:54 9khNDmNT4O6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmdWmLDmkgI4Do.diag [2024-10-26 21:54:50] -rw-r--r-- 1 boincer umatlas 28215 Oct 26 21:54 stderr.txt [2024-10-26 21:54:50] No HITS result produced [2024-10-26 21:54:50] *** Contents of shared directory: *** [2024-10-26 21:54:50] total 345544 [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 352446235 Oct 26 19:09 ATLAS.root_0 [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 499906 Oct 26 19:09 input.tar.gz [2024-10-26 21:54:50] -rw-r--r-- 2 boincer umatlas 17569 Oct 26 19:09 start_atlas.sh [2024-10-26 21:54:50] -rw------- 1 boincer umatlas 860160 Oct 26 21:54 result.tar.gz 21:54:52 (38704): run_atlas exited; CPU time 264.909071 21:54:52 (38704): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN