| Name | aDgLDmJTTO6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmp1ZLDmDv0nmm_0 |
| Workunit | 226274629 |
| Created | 25 Oct 2024, 7:00:16 UTC |
| Sent | 25 Oct 2024, 7:03:55 UTC |
| Report deadline | 2 Nov 2024, 7:03:55 UTC |
| Received | 25 Oct 2024, 12:36:35 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10804956 |
| Run time | 2 hours 46 min 52 sec |
| CPU time | 4 hours 20 min 0 sec |
| Validate state | Valid |
| Credit | 631.46 |
| Device peak FLOPS | 43.24 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 1.52 GB |
| Peak swap size | 2.18 GB |
| Peak disk usage | 744.65 MB |
Stderr output
<core_client_version>7.7.0</core_client_version> <![CDATA[ <stderr_txt> 05:49:32 (51804): wrapper (7.7.26015): starting 05:49:32 (51804): wrapper: running run_atlas (--nthreads 8) [2024-10-25 05:49:32] Arguments: --nthreads 8 [2024-10-25 05:49:32] Threads: 8 [2024-10-25 05:49:32] Checking for CVMFS [2024-10-25 05:49:33] Probing /cvmfs/atlas.cern.ch... OK [2024-10-25 05:49:33] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-25 05:49:33] Running cvmfs_config stat atlas.cern.ch [2024-10-25 05:49:34] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-25 05:49:34] 2.11.2.0 85160 2217 116232 138715 1 528 12984214 18432001 32371 130560 0 7701610 99.416 8180582 54474 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-25 05:49:34] CVMFS is ok [2024-10-25 05:49:34] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-25 05:49:34] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-25 05:49:34] Further information can be found at the LHC@home message board. [2024-10-25 05:49:34] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-25 05:49:34] Checking for apptainer binary... [2024-10-25 05:49:34] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-25 05:49:34] Running /usr/bin/apptainer --version [2024-10-25 05:49:34] apptainer version 1.3.2-1.el7 [2024-10-25 05:49:34] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-25 05:49:56] c-212-11.aglt2.org [2024-10-25 05:49:56] apptainer works [2024-10-25 05:49:56] Set ATHENA_PROC_NUMBER=8 [2024-10-25 05:49:56] Set ATHENA_CORE_NUMBER=8 [2024-10-25 05:49:56] Starting ATLAS job with PandaID=6378051502 [2024-10-25 05:49:56] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/4 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh [2024-10-25 08:36:22] *** The last 200 lines of the pilot log: *** [2024-10-25 08:36:22] 2024-10-25 12:33:00,836 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:33:01,385 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:33:01,385 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:33:03,890 | INFO | monitor loop #668: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:33:11,296 | INFO | 9661s have passed since pilot start [2024-10-25 08:36:22] 2024-10-25 12:33:14,566 | INFO | (instant) CPU consumption time for pid=20752: 277.17999999999995 (rounded to 277) [2024-10-25 08:36:22] 2024-10-25 12:33:14,566 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:33:14,567 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:33:14,570 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:33:14,570 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:33:14,573 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2668197 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 08:36:22] 2024-10-25 12:33:14,573 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:33:14,574 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651233 B [2024-10-25 08:36:22] 2024-10-25 12:33:14,574 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 08:36:22] 2024-10-25 12:33:14,574 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 08:36:22] 2024-10-25 12:33:14,674 | INFO | sufficient remaining disk space (99941875712 B) [2024-10-25 08:36:22] 2024-10-25 12:33:14,675 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 08:36:22] 2024-10-25 12:33:14,676 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6378051502: 65398974 B (within 5767168000 B limit) [2024-10-25 08:36:22] 2024-10-25 12:33:14,677 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 08:36:22] 2024-10-25 12:33:14,677 | INFO | total size of present files: 0 B (workdir size: 65398974 B) [2024-10-25 08:36:22] 2024-10-25 12:33:14,677 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 since it does not exist [2024-10-25 08:36:22] 2024-10-25 12:33:15,148 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:33:15,156 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:33:17,661 | INFO | monitor loop #669: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:33:28,372 | INFO | (instant) CPU consumption time for pid=20752: 277.17999999999995 (rounded to 277) [2024-10-25 08:36:22] 2024-10-25 12:33:28,373 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:33:28,373 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:33:28,382 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:33:28,909 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:33:28,909 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:33:31,414 | INFO | monitor loop #670: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:33:42,079 | INFO | (instant) CPU consumption time for pid=20752: 277.87 (rounded to 278) [2024-10-25 08:36:22] 2024-10-25 12:33:42,080 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:33:42,080 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:33:42,083 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:33:42,942 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:33:42,942 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:33:45,449 | INFO | monitor loop #671: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:33:56,322 | INFO | (instant) CPU consumption time for pid=20752: 277.87 (rounded to 278) [2024-10-25 08:36:22] 2024-10-25 12:33:56,322 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:33:56,322 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:33:56,326 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:33:57,055 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:33:57,055 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:33:59,560 | INFO | monitor loop #672: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:34:10,508 | INFO | (instant) CPU consumption time for pid=20752: 277.88 (rounded to 278) [2024-10-25 08:36:22] 2024-10-25 12:34:10,509 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:34:10,509 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:34:10,530 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:34:11,275 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:34:11,275 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:34:13,782 | INFO | monitor loop #673: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:34:24,763 | INFO | (instant) CPU consumption time for pid=20752: 278.63 (rounded to 279) [2024-10-25 08:36:22] 2024-10-25 12:34:24,763 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:34:24,764 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:34:24,768 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:34:25,675 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:34:25,676 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:34:28,179 | INFO | monitor loop #674: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:34:38,868 | INFO | (instant) CPU consumption time for pid=20752: 278.64 (rounded to 279) [2024-10-25 08:36:22] 2024-10-25 12:34:38,868 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:34:38,869 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:34:38,872 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:34:38,872 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:34:38,876 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2671458 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 08:36:22] 2024-10-25 12:34:38,876 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:34:38,877 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651233 B [2024-10-25 08:36:22] 2024-10-25 12:34:38,877 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 08:36:22] 2024-10-25 12:34:38,877 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 08:36:22] 2024-10-25 12:34:38,973 | INFO | sufficient remaining disk space (99935584256 B) [2024-10-25 08:36:22] 2024-10-25 12:34:38,974 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 08:36:22] 2024-10-25 12:34:38,976 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6378051502: 66215245 B (within 5767168000 B limit) [2024-10-25 08:36:22] 2024-10-25 12:34:38,976 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 08:36:22] 2024-10-25 12:34:38,976 | INFO | total size of present files: 0 B (workdir size: 66215245 B) [2024-10-25 08:36:22] 2024-10-25 12:34:38,976 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 since it does not exist [2024-10-25 08:36:22] 2024-10-25 12:34:39,406 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:34:39,406 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:34:41,932 | INFO | monitor loop #675: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:34:52,755 | INFO | (instant) CPU consumption time for pid=20752: 279.36 (rounded to 279) [2024-10-25 08:36:22] 2024-10-25 12:34:52,756 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:34:52,756 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:34:52,761 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:34:53,181 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:34:53,181 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:34:55,686 | INFO | monitor loop #676: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:35:06,044 | WARNING | exception caught: invalid literal for int() with base 10: '81-' [2024-10-25 08:36:22] 2024-10-25 12:35:06,150 | INFO | (instant) CPU consumption time for pid=20752: 279.37 (rounded to 279) [2024-10-25 08:36:22] 2024-10-25 12:35:06,150 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:35:06,150 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:35:06,152 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:35:06,564 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:35:06,565 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:35:09,068 | INFO | monitor loop #677: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:35:11,840 | INFO | 9782s have passed since pilot start [2024-10-25 08:36:22] 2024-10-25 12:35:19,877 | INFO | (instant) CPU consumption time for pid=20752: 279.37 (rounded to 279) [2024-10-25 08:36:22] 2024-10-25 12:35:19,878 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:35:19,878 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:35:19,882 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:35:20,760 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:35:20,761 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:35:23,265 | INFO | monitor loop #678: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:35:33,905 | INFO | (instant) CPU consumption time for pid=20752: 280.13000000000005 (rounded to 280) [2024-10-25 08:36:22] 2024-10-25 12:35:33,905 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:35:33,905 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:35:33,909 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:35:34,628 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:35:34,629 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:35:37,142 | INFO | monitor loop #679: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:35:48,332 | INFO | (instant) CPU consumption time for pid=20752: 280.13000000000005 (rounded to 280) [2024-10-25 08:36:22] 2024-10-25 12:35:48,349 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:35:48,349 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:35:48,352 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 08:36:22] 2024-10-25 12:35:48,805 | INFO | number of running child processes to parent process 20752: 13 [2024-10-25 08:36:22] 2024-10-25 12:35:48,805 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:35:51,316 | INFO | monitor loop #680: job 0:6378051502 is in state 'running' [2024-10-25 08:36:22] 2024-10-25 12:35:51,998 | CRITICAL | max running time (10000s) minus grace time (180s) has been exceeded - time to abort pilot [2024-10-25 08:36:22] 2024-10-25 12:35:51,999 | INFO | setting REACHED_MAXTIME and graceful stop [2024-10-25 08:36:22] 2024-10-25 12:35:51,999 | INFO | [monitor] control thread has ended [2024-10-25 08:36:22] 2024-10-25 12:35:52,434 | WARNING | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s [2024-10-25 08:36:22] 2024-10-25 12:35:52,740 | INFO | all payload control threads have been joined [2024-10-25 08:36:22] 2024-10-25 12:35:52,839 | INFO | all job control threads have been joined [2024-10-25 08:36:22] 2024-10-25 12:35:52,858 | INFO | breaking -- sending SIGTERM to pid=20752 [2024-10-25 08:36:22] 2024-10-25 12:35:52,858 | INFO | breaking -- sleep 10 s before sending SIGKILL pid=20752 [2024-10-25 08:36:22] 2024-10-25 12:35:53,047 | INFO | all data control threads have been joined [2024-10-25 08:36:22] 2024-10-25 12:35:53,090 | INFO | [payload] failed_post thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,170 | INFO | [job] create_data_payload thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,703 | INFO | [data] copytool_in thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,743 | INFO | [job] validate thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,746 | INFO | [payload] control thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,797 | INFO | [payload] validate_pre thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:53,845 | INFO | [job] control thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:54,006 | INFO | [data] copytool_out thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:54,053 | INFO | [data] control thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:54,284 | INFO | [job] retrieve thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:54,413 | INFO | [payload] validate_post thread has finished [2024-10-25 08:36:22] 2024-10-25 12:35:57,370 | INFO | [data] queue_monitor thread has finished [2024-10-25 08:36:22] 2024-10-25 12:36:00,748 | INFO | job.realtimelogging is not enabled [2024-10-25 08:36:22] 2024-10-25 12:36:01,729 | INFO | (instant) CPU consumption time for pid=20752: 4.720000000000001 (rounded to 5) [2024-10-25 08:36:22] 2024-10-25 12:36:01,729 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:36:01,729 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:36:01,733 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 08:36:22] 2024-10-25 12:36:01,733 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_output.txt (trf name=prmon) [2024-10-25 08:36:22] 2024-10-25 12:36:01,737 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2671458 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 08:36:22] 2024-10-25 12:36:01,737 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 0 [2024-10-25 08:36:22] 2024-10-25 12:36:01,738 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651233 B [2024-10-25 08:36:22] 2024-10-25 12:36:01,738 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 08:36:22] 2024-10-25 12:36:01,738 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 08:36:22] 2024-10-25 12:36:01,754 | INFO | [payload] run_realtimelog thread has finished [2024-10-25 08:36:22] 2024-10-25 12:36:01,832 | INFO | sufficient remaining disk space (99931389952 B) [2024-10-25 08:36:22] 2024-10-25 12:36:01,832 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 08:36:22] 2024-10-25 12:36:01,833 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6378051502: 66975433 B (within 5767168000 B limit) [2024-10-25 08:36:22] 2024-10-25 12:36:01,833 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 08:36:22] 2024-10-25 12:36:01,834 | INFO | total size of present files: 0 B (workdir size: 66975433 B) [2024-10-25 08:36:22] 2024-10-25 12:36:01,834 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/HITS.41779991._001412.pool.root.1 since it does not exist [2024-10-25 08:36:22] 2024-10-25 12:36:02,252 | INFO | number of running child processes to parent process 20752: 1 [2024-10-25 08:36:22] 2024-10-25 12:36:02,252 | INFO | maximum number of monitored processes: 16 [2024-10-25 08:36:22] 2024-10-25 12:36:02,252 | INFO | will abort loop [2024-10-25 08:36:22] 2024-10-25 12:36:02,909 | INFO | [2024-10-25 08:36:22] [2024-10-25 08:36:22] finished pid=20752 exit_code=None state=failed [2024-10-25 08:36:22] [2024-10-25 08:36:22] 2024-10-25 12:36:02,910 | WARNING | detected unset exit_code from wait_graceful - reset to -1 [2024-10-25 08:36:22] 2024-10-25 12:36:02,911 | INFO | using pid=45402 to kill prmon [2024-10-25 08:36:22] 2024-10-25 12:36:02,911 | INFO | stopping utility process 'MemoryMonitor' with signal 10 [2024-10-25 08:36:22] 2024-10-25 12:36:02,911 | INFO | process 45402 no longer exists [2024-10-25 08:36:22] 2024-10-25 12:36:02,911 | INFO | utility process 20754 cleanup finished with status=True [2024-10-25 08:36:22] 2024-10-25 12:36:02,911 | INFO | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20) [2024-10-25 08:36:22] 2024-10-25 12:36:03,258 | INFO | [job] job monitor thread has finished [2024-10-25 08:36:22] 2024-10-25 12:36:05,927 | INFO | copied /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/memory_monitor_summary.json to /tmp/boinchome/slots/4 [2024-10-25 08:36:22] 2024-10-25 12:36:05,927 | INFO | CPU consumption time: 262.12 s (rounded to 262 s) [2024-10-25 08:36:22] 2024-10-25 12:36:05,927 | WARNING | main payload execution returned non-zero exit code: -1 [2024-10-25 08:36:22] 2024-10-25 12:36:05,928 | INFO | scanning dmesg message for subprocess=45442 for memory errors [2024-10-25 08:36:22] 2024-10-25 12:36:05,928 | INFO | executing command: dmesg|grep 45442 [2024-10-25 08:36:22] 2024-10-25 12:36:06,147 | WARNING | job report does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/jobReport.json [2024-10-25 08:36:22] 2024-10-25 12:36:06,147 | WARNING | metadata does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/metadata.xml [2024-10-25 08:36:22] 2024-10-25 12:36:06,147 | WARNING | file does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6378051502/metadata.xml [2024-10-25 08:36:22] 2024-10-25 12:36:06,147 | INFO | generated guid for lfn=HITS.41779991._001412.pool.root.1: 3396B0AC-E257-483E-B8C3-D614B3ED9C2F [2024-10-25 08:36:22] 2024-10-25 12:36:06,147 | WARNING | aborting payload error diagnosis since an error has already been set: [1315, 1187] [2024-10-25 08:36:22] 2024-10-25 12:36:07,224 | INFO | [payload] execute_payloads thread has finished [2024-10-25 08:36:22] 2024-10-25 12:36:07,832 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140413444605760)>', '<ExcThread(queue_monitor, started 140412849207040)>'] [2024-10-25 08:36:22] 2024-10-25 12:36:09,844 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140413444605760)>', '<ExcThread(queue_monitor, started 140412849207040)>'] [2024-10-25 08:36:22] 2024-10-25 12:36:11,856 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140413444605760)>', '<ExcThread(queue_monitor, started 140412849207040)>'] [2024-10-25 08:36:22] 2024-10-25 12:36:13,868 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140413444605760)>', '<ExcThread(queue_monitor, started 140412849207040)>'] [2024-10-25 08:36:22] 2024-10-25 12:36:15,548 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140413444605760)>', '<ExcThread(queue_monitor, started 140412849207040)>'] [2024-10-25 08:36:22] 2024-10-25 12:36:15,550 | INFO | [job] queue monitor thread has finished [2024-10-25 08:36:22] 2024-10-25 12:36:15,880 | INFO | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140413444605760)>']) [2024-10-25 08:36:22] 2024-10-25 12:36:20,905 | INFO | all workflow threads have been joined [2024-10-25 08:36:22] 2024-10-25 12:36:20,905 | INFO | end of generic workflow (traces error code: 0) [2024-10-25 08:36:22] 2024-10-25 12:36:20,906 | INFO | traces error code: 0 [2024-10-25 08:36:22] 2024-10-25 12:36:20,906 | INFO | pilot has finished (exit code=0, shell exit code=0) [2024-10-25 08:36:22] 2024-10-25 12:36:21,165 [wrapper] ==== pilot stdout END ==== [2024-10-25 08:36:22] 2024-10-25 12:36:21,200 [wrapper] ==== wrapper stdout RESUME ==== [2024-10-25 08:36:22] 2024-10-25 12:36:21,216 [wrapper] pilotpid: 81642 [2024-10-25 08:36:22] 2024-10-25 12:36:21,225 [wrapper] Pilot exit status: 0 [2024-10-25 08:36:22] 2024-10-25 12:36:21,290 [wrapper] pandaids: 6378051502 [2024-10-25 08:36:22] 2024-10-25 12:36:21,408 [wrapper] cleanup supervisor_pilot 7599 81643 [2024-10-25 08:36:22] 2024-10-25 12:36:21,423 [wrapper] Test setup, not cleaning [2024-10-25 08:36:22] 2024-10-25 12:36:21,429 [wrapper] ==== wrapper stdout END ==== [2024-10-25 08:36:22] 2024-10-25 12:36:21,454 [wrapper] ==== wrapper stderr END ==== [2024-10-25 08:36:22] 2024-10-25 12:36:21,500 [wrapper] apfmon messages muted [2024-10-25 08:36:22] *** Error codes and diagnostics *** [2024-10-25 08:36:22] *** Listing of results directory *** [2024-10-25 08:36:22] total 351088 [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 5653 Oct 25 02:58 queuedata.json [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 486088 Oct 25 02:58 pilot3.tar.gz [2024-10-25 08:36:22] -rwx------ 1 boincer umatlas 33946 Oct 25 02:59 runpilot2-wrapper.sh [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 100 Oct 25 05:49 wrapper_26015_x86_64-pc-linux-gnu [2024-10-25 08:36:22] -rwxr-xr-x 1 boincer umatlas 7986 Oct 25 05:49 run_atlas [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 105 Oct 25 05:49 job.xml [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 353256411 Oct 25 05:49 EVNT.41779988._000003.pool.root.1 [2024-10-25 08:36:22] drwxrwx--x 2 boincer umatlas 4096 Oct 25 05:49 shared [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 499785 Oct 25 05:49 input.tar.gz [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 17569 Oct 25 05:49 start_atlas.sh [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 0 Oct 25 05:49 boinc_lockfile [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 2853 Oct 25 05:49 pandaJob.out [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 467 Oct 25 05:50 setup.sh.local [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 1019392 Oct 25 05:52 agis_schedconf.cvmfs.json [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 1611897 Oct 25 05:52 agis_ddmendpoints.agis.ALL.json [2024-10-25 08:36:22] drwx------ 4 boincer umatlas 4096 Oct 25 05:52 pilot3 [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 900 Oct 25 08:27 heartbeat.json [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 6386 Oct 25 08:31 init_data.xml [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 95 Oct 25 08:35 pilot_heartbeat.json [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 529 Oct 25 08:35 boinc_task_state.xml [2024-10-25 08:36:22] drwxrwx--- 3 boincer umatlas 4096 Oct 25 08:36 PanDA_Pilot-6378051502 [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 1074 Oct 25 08:36 memory_monitor_summary.json [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 8192 Oct 25 08:36 boinc_mmap_file [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 27 Oct 25 08:36 wrapper_checkpoint.txt [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 794587 Oct 25 08:36 pilotlog.txt [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 811536 Oct 25 08:36 log.41779991._001412.job.log.1 [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 571 Oct 25 08:36 runtime_log [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 819200 Oct 25 08:36 result.tar.gz [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 10745 Oct 25 08:36 runtime_log.err [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 647 Oct 25 08:36 aDgLDmJTTO6nsSi4ap6QjLDmwznN0nGgGQJmDC4LDmp1ZLDmDv0nmm.diag [2024-10-25 08:36:22] -rw-r--r-- 1 boincer umatlas 28151 Oct 25 08:36 stderr.txt [2024-10-25 08:36:22] No HITS result produced [2024-10-25 08:36:22] *** Contents of shared directory: *** [2024-10-25 08:36:22] total 346296 [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 353256411 Oct 25 05:49 ATLAS.root_0 [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 499785 Oct 25 05:49 input.tar.gz [2024-10-25 08:36:22] -rw-r--r-- 2 boincer umatlas 17569 Oct 25 05:49 start_atlas.sh [2024-10-25 08:36:22] -rw------- 1 boincer umatlas 819200 Oct 25 08:36 result.tar.gz 08:36:23 (51804): run_atlas exited; CPU time 300.803876 08:36:23 (51804): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN