| Name | cx2KDmiwQO6n9Rq4apOajLDm4fhM0noT9bVorHsSDm1L4KDmkBZGXn_0 |
| Workunit | 226268297 |
| Created | 25 Oct 2024, 4:17:54 UTC |
| Sent | 25 Oct 2024, 4:18:17 UTC |
| Report deadline | 2 Nov 2024, 4:18:17 UTC |
| Received | 25 Oct 2024, 9:49:38 UTC |
| Server state | Over |
| Outcome | Success |
| Client state | Done |
| Exit status | 0 (0x00000000) |
| Computer ID | 10804956 |
| Run time | 2 hours 45 min 51 sec |
| CPU time | 4 hours 14 min 45 sec |
| Validate state | Valid |
| Credit | 629.67 |
| Device peak FLOPS | 43.24 GFLOPS |
| Application version | ATLAS Simulation v3.01 (native_mt) x86_64-pc-linux-gnu |
| Peak working set size | 1.56 GB |
| Peak swap size | 2.18 GB |
| Peak disk usage | 807.67 MB |
Stderr output
<core_client_version>7.7.0</core_client_version> <![CDATA[ <stderr_txt> 03:03:36 (65195): wrapper (7.7.26015): starting 03:03:36 (65195): wrapper: running run_atlas (--nthreads 8) [2024-10-25 03:03:36] Arguments: --nthreads 8 [2024-10-25 03:03:36] Threads: 8 [2024-10-25 03:03:36] Checking for CVMFS [2024-10-25 03:03:37] Probing /cvmfs/atlas.cern.ch... OK [2024-10-25 03:03:37] Probing /cvmfs/atlas-condb.cern.ch... OK [2024-10-25 03:03:37] Running cvmfs_config stat atlas.cern.ch [2024-10-25 03:03:38] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE [2024-10-25 03:03:38] 2.11.2.0 85160 2051 122832 138711 3 557 10360040 18432000 32598 130560 0 6770692 99.376 7741325 54165 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.237.109:6081 1 [2024-10-25 03:03:38] CVMFS is ok [2024-10-25 03:03:38] Efficiency of ATLAS tasks can be improved by the following measure(s): [2024-10-25 03:03:38] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io. [2024-10-25 03:03:38] Further information can be found at the LHC@home message board. [2024-10-25 03:03:38] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 [2024-10-25 03:03:38] Checking for apptainer binary... [2024-10-25 03:03:38] Using apptainer found in PATH at /usr/bin/apptainer [2024-10-25 03:03:38] Running /usr/bin/apptainer --version [2024-10-25 03:03:38] apptainer version 1.3.2-1.el7 [2024-10-25 03:03:38] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname [2024-10-25 03:03:51] c-212-11.aglt2.org [2024-10-25 03:03:51] apptainer works [2024-10-25 03:03:51] Set ATHENA_PROC_NUMBER=8 [2024-10-25 03:03:51] Set ATHENA_CORE_NUMBER=8 [2024-10-25 03:03:51] Starting ATLAS job with PandaID=6377922651 [2024-10-25 03:03:51] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/4 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh [2024-10-25 05:49:24] *** The last 200 lines of the pilot log: *** [2024-10-25 05:49:24] 2024-10-25 09:45:39,054 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:45:39,057 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:45:40,546 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:45:40,547 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:45:43,053 | INFO | monitor loop #646: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:45:53,805 | INFO | (instant) CPU consumption time for pid=14386: 278.47 (rounded to 278) [2024-10-25 05:49:24] 2024-10-25 09:45:53,805 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:45:53,805 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:45:53,809 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:45:54,732 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:45:54,732 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:45:57,237 | INFO | monitor loop #647: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:46:07,532 | INFO | 9657s have passed since pilot start [2024-10-25 05:49:24] 2024-10-25 09:46:08,750 | INFO | (instant) CPU consumption time for pid=14386: 279.21999999999997 (rounded to 279) [2024-10-25 05:49:24] 2024-10-25 09:46:08,750 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:46:08,751 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:46:08,775 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:46:08,776 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:46:08,794 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2693349 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 05:49:24] 2024-10-25 09:46:08,794 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:46:08,795 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-25 05:49:24] 2024-10-25 09:46:08,795 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 05:49:24] 2024-10-25 09:46:08,795 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 05:49:24] 2024-10-25 09:46:08,927 | INFO | sufficient remaining disk space (100325654528 B) [2024-10-25 05:49:24] 2024-10-25 09:46:08,927 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 05:49:24] 2024-10-25 09:46:08,929 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6377922651: 62385720 B (within 5767168000 B limit) [2024-10-25 05:49:24] 2024-10-25 09:46:08,929 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 05:49:24] 2024-10-25 09:46:08,930 | INFO | total size of present files: 0 B (workdir size: 62385720 B) [2024-10-25 05:49:24] 2024-10-25 09:46:08,930 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 since it does not exist [2024-10-25 05:49:24] 2024-10-25 09:46:09,955 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:46:09,955 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:46:12,465 | INFO | monitor loop #648: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:46:24,615 | INFO | (instant) CPU consumption time for pid=14386: 279.21999999999997 (rounded to 279) [2024-10-25 05:49:24] 2024-10-25 09:46:24,616 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:46:24,616 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:46:24,636 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:46:26,239 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:46:26,240 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:46:28,744 | INFO | monitor loop #649: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:46:39,570 | INFO | (instant) CPU consumption time for pid=14386: 279.96 (rounded to 280) [2024-10-25 05:49:24] 2024-10-25 09:46:39,570 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:46:39,570 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:46:39,574 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:46:40,371 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:46:40,371 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:46:42,875 | INFO | monitor loop #650: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:46:53,792 | INFO | (instant) CPU consumption time for pid=14386: 279.96 (rounded to 280) [2024-10-25 05:49:24] 2024-10-25 09:46:53,793 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:46:53,793 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:46:53,796 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:46:54,943 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:46:54,944 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:46:57,451 | INFO | monitor loop #651: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:47:08,723 | INFO | (instant) CPU consumption time for pid=14386: 280.18 (rounded to 280) [2024-10-25 05:49:24] 2024-10-25 09:47:08,728 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:47:08,728 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:47:08,742 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:47:09,965 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:47:09,984 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:47:12,489 | INFO | monitor loop #652: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:47:23,499 | INFO | (instant) CPU consumption time for pid=14386: 280.7 (rounded to 281) [2024-10-25 05:49:24] 2024-10-25 09:47:23,499 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:47:23,500 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:47:23,503 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:47:23,503 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:47:23,506 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2696268 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 05:49:24] 2024-10-25 09:47:23,506 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:47:23,507 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-25 05:49:24] 2024-10-25 09:47:23,507 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 05:49:24] 2024-10-25 09:47:23,507 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 05:49:24] 2024-10-25 09:47:23,648 | INFO | sufficient remaining disk space (100320411648 B) [2024-10-25 05:49:24] 2024-10-25 09:47:23,649 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 05:49:24] 2024-10-25 09:47:23,650 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6377922651: 64652264 B (within 5767168000 B limit) [2024-10-25 05:49:24] 2024-10-25 09:47:23,651 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 05:49:24] 2024-10-25 09:47:23,651 | INFO | total size of present files: 0 B (workdir size: 64652264 B) [2024-10-25 05:49:24] 2024-10-25 09:47:23,651 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 since it does not exist [2024-10-25 05:49:24] 2024-10-25 09:47:24,698 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:47:24,699 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:47:27,203 | INFO | monitor loop #653: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:47:38,547 | INFO | (instant) CPU consumption time for pid=14386: 280.71000000000004 (rounded to 281) [2024-10-25 05:49:24] 2024-10-25 09:47:38,547 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:47:38,547 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:47:38,551 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:47:39,506 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:47:39,507 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:47:42,015 | INFO | monitor loop #654: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:47:53,554 | INFO | (instant) CPU consumption time for pid=14386: 281.48 (rounded to 281) [2024-10-25 05:49:24] 2024-10-25 09:47:53,555 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:47:53,555 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:47:53,558 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:47:54,810 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:47:54,810 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:47:57,316 | INFO | monitor loop #655: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:48:08,133 | INFO | 9777s have passed since pilot start [2024-10-25 05:49:24] 2024-10-25 09:48:09,215 | INFO | (instant) CPU consumption time for pid=14386: 281.48 (rounded to 281) [2024-10-25 05:49:24] 2024-10-25 09:48:09,215 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:48:09,215 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:48:09,241 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:48:10,426 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:48:10,427 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:48:12,931 | INFO | monitor loop #656: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:48:24,257 | INFO | (instant) CPU consumption time for pid=14386: 282.24 (rounded to 282) [2024-10-25 05:49:24] 2024-10-25 09:48:24,258 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:48:24,258 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:48:24,261 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:48:25,498 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:48:25,499 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:48:28,005 | INFO | monitor loop #657: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:48:39,511 | INFO | (instant) CPU consumption time for pid=14386: 282.25 (rounded to 282) [2024-10-25 05:49:24] 2024-10-25 09:48:39,511 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:48:39,520 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:48:39,529 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:48:39,530 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:48:39,533 | INFO | max memory (maxPSS) used by the payload is within the allowed limit: 2696343 B (2 * maxRSS = 65536000 B, memkillgrace = 100%) [2024-10-25 05:49:24] 2024-10-25 09:48:39,543 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 1000 [2024-10-25 05:49:24] 2024-10-25 09:48:39,544 | INFO | payload log (log.EVNTtoHITS) within allowed size limit (2147483648 B): 651232 B [2024-10-25 05:49:24] 2024-10-25 09:48:39,544 | INFO | payload log (payload.stdout) within allowed size limit (2147483648 B): 6610 B [2024-10-25 05:49:24] 2024-10-25 09:48:39,544 | INFO | executing command: df -mP /tmp/boinchome/slots/4 [2024-10-25 05:49:24] 2024-10-25 09:48:39,770 | INFO | sufficient remaining disk space (100319363072 B) [2024-10-25 05:49:24] 2024-10-25 09:48:39,770 | INFO | work directory size check will use 5767168000 B as a max limit (10% grace limit added) [2024-10-25 05:49:24] 2024-10-25 09:48:39,772 | INFO | size of work directory /tmp/boinchome/slots/4/PanDA_Pilot-6377922651: 65688950 B (within 5767168000 B limit) [2024-10-25 05:49:24] 2024-10-25 09:48:39,772 | INFO | pfn file=/tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 does not exist (skip from workdir size calculation) [2024-10-25 05:49:24] 2024-10-25 09:48:39,772 | INFO | total size of present files: 0 B (workdir size: 65688950 B) [2024-10-25 05:49:24] 2024-10-25 09:48:39,773 | INFO | output file size check: skipping output file /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/HITS.41779574._000629.pool.root.1 since it does not exist [2024-10-25 05:49:24] 2024-10-25 09:48:40,874 | INFO | number of running child processes to parent process 14386: 13 [2024-10-25 05:49:24] 2024-10-25 09:48:40,875 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:48:43,379 | INFO | monitor loop #658: job 0:6377922651 is in state 'running' [2024-10-25 05:49:24] 2024-10-25 09:48:52,361 | CRITICAL | max running time (10000s) minus grace time (180s) has been exceeded - time to abort pilot [2024-10-25 05:49:24] 2024-10-25 09:48:52,362 | INFO | setting REACHED_MAXTIME and graceful stop [2024-10-25 05:49:24] 2024-10-25 09:48:52,362 | INFO | [monitor] control thread has ended [2024-10-25 05:49:24] 2024-10-25 09:48:52,362 | WARNING | since job:queue_monitor is responsible for sending job updates, we sleep for 20 s [2024-10-25 05:49:24] 2024-10-25 09:48:52,586 | INFO | all payload control threads have been joined [2024-10-25 05:49:24] 2024-10-25 09:48:52,730 | INFO | all job control threads have been joined [2024-10-25 05:49:24] 2024-10-25 09:48:53,116 | INFO | all data control threads have been joined [2024-10-25 05:49:24] 2024-10-25 09:48:53,224 | INFO | breaking -- sending SIGTERM to pid=14386 [2024-10-25 05:49:24] 2024-10-25 09:48:53,225 | INFO | breaking -- sleep 10 s before sending SIGKILL pid=14386 [2024-10-25 05:49:24] 2024-10-25 09:48:53,427 | INFO | [payload] validate_pre thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:53,512 | INFO | [job] validate thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:53,592 | INFO | [payload] control thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:53,736 | INFO | [job] control thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:53,883 | INFO | [payload] failed_post thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:54,122 | INFO | [data] control thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:54,271 | INFO | [job] retrieve thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:54,363 | INFO | [data] copytool_in thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:54,430 | INFO | (instant) CPU consumption time for pid=14386: 4.55 (rounded to 5) [2024-10-25 05:49:24] 2024-10-25 09:48:54,430 | INFO | neither /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json, nor /tmp/boinchome/slots/4/memory_monitor_summary.json exist [2024-10-25 05:49:24] 2024-10-25 09:48:54,430 | INFO | using path: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_output.txt (trf name=prmon) [2024-10-25 05:49:24] 2024-10-25 09:48:54,434 | INFO | oom_score(pilot) = 1000, oom_score(payload) = 0 [2024-10-25 05:49:24] 2024-10-25 09:48:54,474 | INFO | [job] create_data_payload thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:54,544 | INFO | [payload] validate_post thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:55,168 | INFO | number of running child processes to parent process 14386: 1 [2024-10-25 05:49:24] 2024-10-25 09:48:55,168 | INFO | maximum number of monitored processes: 16 [2024-10-25 05:49:24] 2024-10-25 09:48:55,168 | INFO | will abort loop [2024-10-25 05:49:24] 2024-10-25 09:48:55,785 | INFO | [data] copytool_out thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:56,198 | INFO | [job] job monitor thread has finished [2024-10-25 05:49:24] 2024-10-25 09:48:59,320 | INFO | job.realtimelogging is not enabled [2024-10-25 05:49:24] 2024-10-25 09:48:59,362 | INFO | [data] queue_monitor thread has finished [2024-10-25 05:49:24] 2024-10-25 09:49:00,332 | INFO | [payload] run_realtimelog thread has finished [2024-10-25 05:49:24] 2024-10-25 09:49:03,278 | INFO | [2024-10-25 05:49:24] [2024-10-25 05:49:24] finished pid=14386 exit_code=None state=failed [2024-10-25 05:49:24] [2024-10-25 05:49:24] 2024-10-25 09:49:03,278 | WARNING | detected unset exit_code from wait_graceful - reset to -1 [2024-10-25 05:49:24] 2024-10-25 09:49:03,283 | INFO | using pid=29077 to kill prmon [2024-10-25 05:49:24] 2024-10-25 09:49:03,284 | INFO | stopping utility process 'MemoryMonitor' with signal 10 [2024-10-25 05:49:24] 2024-10-25 09:49:03,284 | INFO | process 29077 no longer exists [2024-10-25 05:49:24] 2024-10-25 09:49:03,284 | INFO | utility process 14390 cleanup finished with status=True [2024-10-25 05:49:24] 2024-10-25 09:49:03,284 | INFO | taking a short nap (3 s) to allow the memory monitor to finish writing to the summary file (#0/#20) [2024-10-25 05:49:24] 2024-10-25 09:49:06,300 | INFO | copied /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/memory_monitor_summary.json to /tmp/boinchome/slots/4 [2024-10-25 05:49:24] 2024-10-25 09:49:06,300 | INFO | CPU consumption time: 243.17999999999998 s (rounded to 243 s) [2024-10-25 05:49:24] 2024-10-25 09:49:06,300 | WARNING | main payload execution returned non-zero exit code: -1 [2024-10-25 05:49:24] 2024-10-25 09:49:06,314 | INFO | scanning dmesg message for subprocess=28594 for memory errors [2024-10-25 05:49:24] 2024-10-25 09:49:06,315 | INFO | executing command: dmesg|grep 28594 [2024-10-25 05:49:24] 2024-10-25 09:49:06,734 | WARNING | job report does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/jobReport.json [2024-10-25 05:49:24] 2024-10-25 09:49:06,735 | WARNING | metadata does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/metadata.xml [2024-10-25 05:49:24] 2024-10-25 09:49:06,735 | WARNING | file does not exist: /tmp/boinchome/slots/4/PanDA_Pilot-6377922651/metadata.xml [2024-10-25 05:49:24] 2024-10-25 09:49:06,735 | INFO | generated guid for lfn=HITS.41779574._000629.pool.root.1: 8F060B95-DEC3-40B3-BEF9-AEB532528F46 [2024-10-25 05:49:24] 2024-10-25 09:49:06,735 | WARNING | aborting payload error diagnosis since an error has already been set: [1315, 1187] [2024-10-25 05:49:24] 2024-10-25 09:49:07,939 | INFO | [payload] execute_payloads thread has finished [2024-10-25 05:49:24] 2024-10-25 09:49:09,134 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140335118382912)>', '<ExcThread(queue_monitor, started 140334164080384)>'] [2024-10-25 05:49:24] 2024-10-25 09:49:11,147 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140335118382912)>', '<ExcThread(queue_monitor, started 140334164080384)>'] [2024-10-25 05:49:24] 2024-10-25 09:49:13,159 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140335118382912)>', '<ExcThread(queue_monitor, started 140334164080384)>'] [2024-10-25 05:49:24] 2024-10-25 09:49:15,171 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140335118382912)>', '<ExcThread(queue_monitor, started 140334164080384)>'] [2024-10-25 05:49:24] 2024-10-25 09:49:15,473 | INFO | waiting for thread to finish: ['<_MainThread(MainThread, started 140335118382912)>', '<ExcThread(queue_monitor, started 140334164080384)>'] [2024-10-25 05:49:24] 2024-10-25 09:49:15,474 | INFO | [job] queue monitor thread has finished [2024-10-25 05:49:24] 2024-10-25 09:49:17,183 | INFO | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 140335118382912)>']) [2024-10-25 05:49:24] 2024-10-25 09:49:22,209 | INFO | all workflow threads have been joined [2024-10-25 05:49:24] 2024-10-25 09:49:22,210 | INFO | end of generic workflow (traces error code: 0) [2024-10-25 05:49:24] 2024-10-25 09:49:22,210 | INFO | traces error code: 0 [2024-10-25 05:49:24] 2024-10-25 09:49:22,210 | INFO | pilot has finished (exit code=0, shell exit code=0) [2024-10-25 05:49:24] 2024-10-25 09:49:22,538 [wrapper] ==== pilot stdout END ==== [2024-10-25 05:49:24] 2024-10-25 09:49:22,542 [wrapper] ==== wrapper stdout RESUME ==== [2024-10-25 05:49:24] 2024-10-25 09:49:22,557 [wrapper] pilotpid: 88122 [2024-10-25 05:49:24] 2024-10-25 09:49:22,592 [wrapper] Pilot exit status: 0 [2024-10-25 05:49:24] 2024-10-25 09:49:22,720 [wrapper] pandaids: 6377922651 [2024-10-25 05:49:24] 2024-10-25 09:49:23,180 [wrapper] cleanup supervisor_pilot 80736 88124 [2024-10-25 05:49:24] 2024-10-25 09:49:23,216 [wrapper] Test setup, not cleaning [2024-10-25 05:49:24] 2024-10-25 09:49:23,260 [wrapper] ==== wrapper stdout END ==== [2024-10-25 05:49:24] 2024-10-25 09:49:23,301 [wrapper] ==== wrapper stderr END ==== [2024-10-25 05:49:24] 2024-10-25 09:49:23,407 [wrapper] apfmon messages muted [2024-10-25 05:49:24] *** Error codes and diagnostics *** [2024-10-25 05:49:24] *** Listing of results directory *** [2024-10-25 05:49:24] total 384132 [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 486088 Oct 25 00:07 pilot3.tar.gz [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 5653 Oct 25 00:14 queuedata.json [2024-10-25 05:49:24] -rwx------ 1 boincer umatlas 33946 Oct 25 00:17 runpilot2-wrapper.sh [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 100 Oct 25 03:03 wrapper_26015_x86_64-pc-linux-gnu [2024-10-25 05:49:24] -rwxr-xr-x 1 boincer umatlas 7986 Oct 25 03:03 run_atlas [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 105 Oct 25 03:03 job.xml [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 387125417 Oct 25 03:03 EVNT.41779572._000007.pool.root.1 [2024-10-25 05:49:24] drwxrwx--x 2 boincer umatlas 4096 Oct 25 03:03 shared [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 499796 Oct 25 03:03 input.tar.gz [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 17569 Oct 25 03:03 start_atlas.sh [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 0 Oct 25 03:03 boinc_lockfile [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 2892 Oct 25 03:03 pandaJob.out [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 467 Oct 25 03:04 setup.sh.local [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 1019384 Oct 25 03:05 agis_schedconf.cvmfs.json [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 1611897 Oct 25 03:05 agis_ddmendpoints.agis.ALL.json [2024-10-25 05:49:24] drwx------ 4 boincer umatlas 4096 Oct 25 03:05 pilot3 [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 899 Oct 25 05:40 heartbeat.json [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 6387 Oct 25 05:47 init_data.xml [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 95 Oct 25 05:48 pilot_heartbeat.json [2024-10-25 05:49:24] drwxrwx--- 3 boincer umatlas 4096 Oct 25 05:49 PanDA_Pilot-6377922651 [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 1084 Oct 25 05:49 memory_monitor_summary.json [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 529 Oct 25 05:49 boinc_task_state.xml [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 8192 Oct 25 05:49 boinc_mmap_file [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 27 Oct 25 05:49 wrapper_checkpoint.txt [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 783426 Oct 25 05:49 pilotlog.txt [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 798625 Oct 25 05:49 log.41779574._000629.job.log.1 [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 571 Oct 25 05:49 runtime_log [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 808960 Oct 25 05:49 result.tar.gz [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 10745 Oct 25 05:49 runtime_log.err [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 648 Oct 25 05:49 cx2KDmiwQO6n9Rq4apOajLDm4fhM0noT9bVorHsSDm1L4KDmkBZGXn.diag [2024-10-25 05:49:24] -rw-r--r-- 1 boincer umatlas 28177 Oct 25 05:49 stderr.txt [2024-10-25 05:49:24] No HITS result produced [2024-10-25 05:49:24] *** Contents of shared directory: *** [2024-10-25 05:49:24] total 379364 [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 387125417 Oct 25 03:03 ATLAS.root_0 [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 499796 Oct 25 03:03 input.tar.gz [2024-10-25 05:49:24] -rw-r--r-- 2 boincer umatlas 17569 Oct 25 03:03 start_atlas.sh [2024-10-25 05:49:24] -rw------- 1 boincer umatlas 808960 Oct 25 05:49 result.tar.gz 05:49:25 (65195): run_atlas exited; CPU time 274.627321 05:49:25 (65195): called boinc_finish(0) </stderr_txt> ]]>
©2024 CERN