Stderr output

<core_client_version>7.16.20</core_client_version>
<![CDATA[
<stderr_txt>
2025-02-18 02:56:47 (5156): Detected: vboxwrapper 26206
2025-02-18 02:56:47 (5156): Detected: BOINC client v7.16.20
2025-02-18 02:56:47 (5156): Detected: VirtualBox VboxManage Interface (Version: 6.0.14)
2025-02-18 02:56:48 (5156): Successfully copied 'init_data.xml' to the shared directory.
2025-02-18 02:56:48 (5156): Create VM. (boinc_4b213d9bfa348918, slot#0)
2025-02-18 02:56:49 (5156): Setting Memory Size for VM. (4400MB)
2025-02-18 02:56:49 (5156): Setting CPU Count for VM. (4)
2025-02-18 02:56:51 (5156): Setting Chipset Options for VM.
2025-02-18 02:56:52 (5156): Setting Graphics Controller Options for VM.
2025-02-18 02:56:53 (5156): Setting Boot Options for VM.
2025-02-18 02:56:53 (5156): Setting Network Configuration for NAT.
2025-02-18 02:56:55 (5156): Enabling VM Network Access.
2025-02-18 02:56:56 (5156): Disabling USB Support for VM.
2025-02-18 02:56:58 (5156): Disabling COM Port Support for VM.
2025-02-18 02:56:58 (5156): Disabling LPT Port Support for VM.
2025-02-18 02:56:59 (5156): Disabling Audio Support for VM.
2025-02-18 02:57:00 (5156): Disabling Clipboard Support for VM.
2025-02-18 02:57:01 (5156): Disabling Drag and Drop Support for VM.
2025-02-18 02:57:01 (5156): Adding storage controller(s) to VM.
2025-02-18 02:57:02 (5156): Adding virtual disk drive to VM. (ATLAS_vbox_3.01_image.vdi)
2025-02-18 02:57:06 (5156): Adding VirtualBox Guest Additions to VM.
2025-02-18 02:57:07 (5156): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB)
2025-02-18 02:57:08 (5156): forwarding host port 55651 to guest port 80
2025-02-18 02:57:09 (5156): Enabling remote desktop for VM.
2025-02-18 02:57:09 (5156): Required extension pack not installed, remote desktop not enabled.
2025-02-18 02:57:09 (5156): Enabling shared directory for VM.
2025-02-18 02:57:10 (5156): Starting VM using VBoxManage interface. (boinc_4b213d9bfa348918, slot#0)
2025-02-18 02:57:15 (5156): Successfully started VM. (PID = '1684')
2025-02-18 02:57:15 (5156): Reporting VM Process ID to BOINC.
2025-02-18 02:57:15 (5156): Guest Log: BIOS: VirtualBox 6.0.14
2025-02-18 02:57:15 (5156): Guest Log: CPUID EDX: 0x178bfbff
2025-02-18 02:57:15 (5156): Guest Log: BIOS: AHCI 0-P#0: PCHS=16383/16/63 LCHS=1024/255/63 0x0000000002800000 sectors
2025-02-18 02:57:15 (5156): VM state change detected. (old = 'poweredoff', new = 'running')
2025-02-18 02:57:15 (5156): Detected: Web Application Enabled (http://localhost:55651)
2025-02-18 02:57:16 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 02:58:19 (5156): Preference change detected
2025-02-18 02:58:19 (5156): Setting CPU throttle for VM. (100%)
2025-02-18 02:58:19 (5156): Setting checkpoint interval to 900 seconds. (Higher value of (Preference: 60 seconds) or (Vbox_job.xml: 900 seconds))
2025-02-18 02:58:20 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 02:58:21 (5156): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032
2025-02-18 02:58:21 (5156): Guest Log: BIOS: Booting from Hard Disk...
2025-02-18 02:58:24 (5156): Guest Log: BIOS: KBD: unsupported int 16h function 03
2025-02-18 02:58:24 (5156): Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000 
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=81
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=81
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=82
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=82
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=83
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=83
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=84
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=84
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=85
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=85
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=86
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=86
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=87
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=87
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=88
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=88
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=89
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=89
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8a
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8a
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8b
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8b
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8c
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8c
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8d
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8d
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8e
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8e
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8f
2025-02-18 02:58:24 (5156): Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8f
2025-02-18 02:58:30 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 03:00:40 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 03:00:42 (5156): Guest Log: vgdrvHeartbeatInit: Setting up heartbeat to trigger every 2000 milliseconds
2025-02-18 03:00:42 (5156): Guest Log: vboxguest: misc device minor 58, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
2025-02-18 03:00:50 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 03:14:20 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 03:14:24 (5156): Guest Log: VBoxService 5.2.32 r132073 (verbosity: 0) linux.amd64 (Jul 12 2019 10:32:28) release log
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000149 main     Log opened 2025-02-18T02:57:39.291330000Z
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000260 main     OS Product: Linux
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000317 main     OS Release: 3.10.0-957.27.2.el7.x86_64
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000352 main     OS Version: #1 SMP Mon Jul 29 17:46:05 UTC 2019
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000386 main     Executable: /opt/VBoxGuestAdditions-5.2.32/sbin/VBoxService
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000386 main     Process ID: 1305
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.000387 main     Package type: LINUX_64BITS_GENERIC
2025-02-18 03:14:24 (5156): Guest Log: 00:00:00.001190 main     5.2.32 r132073 started. Verbose level = 0
2025-02-18 03:14:25 (5156): Guest Log: [INFO] Probing /cvmfs/atlas.cern.ch...
2025-02-18 03:14:25 (5156): Guest Log: [INFO] Mounting shared directory
2025-02-18 03:14:25 (5156): Guest Log: [INFO] Checking for init_data.xml
2025-02-18 03:14:30 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 03:17:40 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 03:17:44 (5156): Guest Log: 00:00:10.003769 timesync vgsvcTimeSyncWorker: Radical guest time change: 19 015 147 656 000ns (GuestNow=1 739 866 474 441 773 000 ns GuestLast=1 739 847 459 294 117 000 ns fSetTimeLastLoop=true )
2025-02-18 03:17:50 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 03:19:50 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 03:20:01 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 03:52:52 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 03:52:56 (5156): Guest Log: 00:00:30.006508 timesync vgsvcTimeSyncWorker: Radical host time change: 1 981 260 000 000ns (HostNow=1 739 868 775 857 000 000 ns HostLast=1 739 866 794 597 000 000 ns)
2025-02-18 03:53:02 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:00:32 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:00:36 (5156): Guest Log: 00:00:40.007557 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 291 413 616 000ns (GuestNow=1 739 868 785 857 860 000 ns GuestLast=1 739 866 494 444 244 000 ns fSetTimeLastLoop=true )
2025-02-18 04:00:42 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:02:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:02:32 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:02:52 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:03:02 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:04:02 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:04:12 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:32:24 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:32:35 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:33:24 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:33:27 (5156): Guest Log: 00:01:30.013197 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 332 360 147 000ns (GuestNow=1 739 871 158 223 459 000 ns GuestLast=1 739 868 825 863 312 000 ns fSetTimeLastLoop=true )
2025-02-18 04:33:34 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:33:44 (5156): Guest Log: [INFO] Probing /cvmfs/atlas.cern.ch... OK
2025-02-18 04:33:44 (5156): Guest Log: [INFO] Detected branch: prod
2025-02-18 04:33:44 (5156): Guest Log: This is the prod version of the ATLAS job wrapper
2025-02-18 04:33:44 (5156): Guest Log: Copying input files
2025-02-18 04:33:44 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:33:48 (5156): Guest Log: Copied input files into RunAtlas.
2025-02-18 04:33:48 (5156): Guest Log: This VM did not configure a local http proxy via BOINC.
2025-02-18 04:33:48 (5156): Guest Log: Small home clusters do not require a local http proxy but it is suggested if
2025-02-18 04:33:48 (5156): Guest Log: more than 10 cores throughout the same LAN segment are regularly running ATLAS like tasks.
2025-02-18 04:33:48 (5156): Guest Log: Further information can be found at the LHC@home message board.
2025-02-18 04:33:54 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:50:45 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:50:56 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 04:52:26 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 04:52:36 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:00:16 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:00:26 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:00:36 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:00:46 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:13:06 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:13:16 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:14:16 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:14:26 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:50:48 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:50:49 (5156): Guest Log: 00:02:50.043340 timesync vgsvcTimeSyncWorker: Radical host time change: 2 191 331 000 000ns (HostNow=1 739 875 848 803 000 000 ns HostLast=1 739 873 657 472 000 000 ns)
2025-02-18 05:50:58 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:52:18 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:52:19 (5156): Guest Log: 00:03:00.044921 timesync vgsvcTimeSyncWorker: Radical guest time change: 3 001 675 206 000ns (GuestNow=1 739 875 858 804 375 000 ns GuestLast=1 739 872 857 129 169 000 ns fSetTimeLastLoop=true )
2025-02-18 05:52:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 05:55:19 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 05:55:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:00:38 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:00:43 (5156): Guest Log: Running cvmfs_config stat atlas.cern.ch
2025-02-18 06:00:43 (5156): Guest Log: VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
2025-02-18 06:00:43 (5156): Guest Log: 2.6.3.0 1648 118 32316 142782 4 1 3119916 4096000 0 65024 0 0 n/a 1169 28 http://s1bnl-cvmfs.openhtc.io/cvmfs/atlas.cern.ch DIRECT 1
2025-02-18 06:00:45 (5156): Guest Log: copied the webapp to /var/www
2025-02-18 06:00:45 (5156): Guest Log: ATHENA_PROC_NUMBER=4
2025-02-18 06:00:45 (5156): Guest Log: ATHENA_CORE_NUMBER=4
2025-02-18 06:00:46 (5156): Guest Log: Apptainer command /cvmfs/atlas.cern.ch/repo/containers/sw/apptainer/x86_64-el7/current/bin/apptainer exec -B /cvmfs,/data,/home/atlas/RunAtlas /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 
2025-02-18 06:00:46 (5156): Guest Log:  *** Starting ATLAS job. (PandaID=6517864717 taskID=43205075) ***
2025-02-18 06:00:49 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:04:49 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:04:59 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:05:39 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:05:49 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:07:19 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:07:29 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:09:40 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:09:50 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:51:51 (5156): Guest Log: 00:04:10.069129 timesync vgsvcTimeSyncWorker: Radical host time change: 2 531 548 000 000ns (HostNow=1 739 879 510 912 000 000 ns HostLast=1 739 876 979 364 000 000 ns)
2025-02-18 06:51:51 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:52:02 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 06:53:22 (5156): Guest Log: 00:04:21.190127 timesync vgsvcTimeSyncWorker: Radical guest time change: 3 593 204 562 000ns (GuestNow=1 739 879 522 033 086 000 ns GuestLast=1 739 875 928 828 524 000 ns fSetTimeLastLoop=true )
2025-02-18 06:53:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 06:53:32 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:00:32 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:00:42 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:01:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:01:32 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:14:13 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:14:23 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:23:03 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:23:12 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:49:45 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:49:55 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 07:57:35 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 07:57:36 (5156): Guest Log: 00:05:21.367190 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 122 576 496 000ns (GuestNow=1 739 882 995 146 975 000 ns GuestLast=1 739 880 872 570 479 000 ns fSetTimeLastLoop=true )
2025-02-18 07:57:45 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 20:00:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 20:00:23 (5156): Guest Log: 00:05:31.368976 timesync vgsvcTimeSyncWorker: Radical host time change: 43 367 610 000 000ns (HostNow=1 739 926 823 042 000 000 ns HostLast=1 739 883 455 432 000 000 ns)
2025-02-18 20:00:33 (5156): Guest Log: 00:05:41.372446 timesync vgsvcTimeSyncWorker: Radical guest time change: 43 827 897 295 000ns (GuestNow=1 739 926 833 045 564 000 ns GuestLast=1 739 883 005 148 269 000 ns fSetTimeLastLoop=true )
2025-02-18 20:00:42 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 20:43:24 (5156): Guest Log: 00:05:51.379177 timesync vgsvcTimeSyncWorker: Radical host time change: 2 571 544 000 000ns (HostNow=1 739 929 404 588 000 000 ns HostLast=1 739 926 833 044 000 000 ns)
2025-02-18 20:43:24 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 20:43:34 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 20:51:44 (5156): Guest Log: 00:06:01.381635 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 571 538 471 000ns (GuestNow=1 739 929 414 590 616 000 ns GuestLast=1 739 926 843 052 145 000 ns fSetTimeLastLoop=true )
2025-02-18 20:51:44 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 20:51:54 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 20:53:25 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 20:53:35 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:00:35 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:00:45 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:01:25 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:01:35 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:09:25 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:09:35 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:14:15 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:14:26 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:15:15 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:15:25 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:17:46 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:17:55 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:50:58 (5156): Guest Log: 00:07:21.397777 timesync vgsvcTimeSyncWorker: Radical host time change: 2 133 430 000 000ns (HostNow=1 739 933 457 604 000 000 ns HostLast=1 739 931 324 174 000 000 ns)
2025-02-18 21:50:58 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:51:08 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:55:48 (5156): Guest Log: 00:07:31.398947 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 462 239 259 000ns (GuestNow=1 739 933 467 604 852 000 ns GuestLast=1 739 931 005 365 593 000 ns fSetTimeLastLoop=true )
2025-02-18 21:55:48 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:55:58 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 21:57:18 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 21:57:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:00:38 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:00:48 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:34:10 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:34:19 (5156): Guest Log: 00:08:11.419454 timesync vgsvcTimeSyncWorker: Radical host time change: 2 011 315 000 000ns (HostNow=1 739 936 059 158 000 000 ns HostLast=1 739 934 047 843 000 000 ns)
2025-02-18 22:34:20 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:43:10 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:43:19 (5156): Guest Log: 00:08:21.419710 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 561 532 619 000ns (GuestNow=1 739 936 069 158 327 000 ns GuestLast=1 739 933 507 625 708 000 ns fSetTimeLastLoop=true )
2025-02-18 22:43:20 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:44:40 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:44:50 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:52:51 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:53:01 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 22:55:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 22:55:42 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:00:42 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:00:52 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:02:22 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:02:32 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:14:12 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:14:22 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:45:44 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:45:53 (5156): Guest Log: 00:09:41.449638 timesync vgsvcTimeSyncWorker: Radical host time change: 1 891 261 000 000ns (HostNow=1 739 940 352 181 000 000 ns HostLast=1 739 938 460 920 000 000 ns)
2025-02-18 23:45:54 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:47:24 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:47:33 (5156): Guest Log: 00:09:51.450371 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 981 422 974 000ns (GuestNow=1 739 940 362 181 794 000 ns GuestLast=1 739 937 380 758 820 000 ns fSetTimeLastLoop=true )
2025-02-18 23:47:34 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:55:44 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:55:54 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-18 23:57:24 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-18 23:57:34 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 00:00:34 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 00:00:44 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 00:33:17 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 00:33:25 (5156): Guest Log: 00:10:31.462404 timesync vgsvcTimeSyncWorker: Radical host time change: 1 962 167 000 000ns (HostNow=1 739 943 204 844 000 000 ns HostLast=1 739 941 242 677 000 000 ns)
2025-02-19 00:33:27 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 00:43:07 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 00:43:15 (5156): Guest Log: 00:10:41.462636 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 812 650 863 000ns (GuestNow=1 739 943 214 844 345 000 ns GuestLast=1 739 940 402 193 482 000 ns fSetTimeLastLoop=true )
2025-02-19 00:43:17 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 00:52:47 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 00:52:57 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 01:00:38 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 01:00:48 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 01:02:18 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 01:02:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 01:07:18 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 01:07:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 01:14:18 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 01:14:28 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 01:52:51 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 01:52:58 (5156): Guest Log: 00:11:41.475982 timesync vgsvcTimeSyncWorker: Radical host time change: 2 312 010 000 000ns (HostNow=1 739 947 977 941 000 000 ns HostLast=1 739 945 665 931 000 000 ns)
2025-02-19 01:53:01 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 02:00:41 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 02:00:48 (5156): Guest Log: 00:11:51.476220 timesync vgsvcTimeSyncWorker: Radical guest time change: 3 102 198 139 000ns (GuestNow=1 739 947 987 941 322 000 ns GuestLast=1 739 944 885 743 183 000 ns fSetTimeLastLoop=true )
2025-02-19 02:00:51 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 02:02:21 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 02:02:31 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 02:35:33 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 02:35:40 (5156): Guest Log: 00:12:11.484562 timesync vgsvcTimeSyncWorker: Radical host time change: 1 991 114 000 000ns (HostNow=1 739 950 539 186 000 000 ns HostLast=1 739 948 548 072 000 000 ns)
2025-02-19 02:35:43 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 02:48:53 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 02:49:00 (5156): Guest Log: 00:12:21.496518 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 541 249 446 000ns (GuestNow=1 739 950 549 198 365 000 ns GuestLast=1 739 948 007 948 919 000 ns fSetTimeLastLoop=true )
2025-02-19 02:49:03 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 02:58:23 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 02:58:33 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:00:34 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:00:44 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:14:14 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:14:24 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:17:45 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:17:55 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:21:25 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:21:35 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:21:45 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:21:55 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:47:46 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:47:56 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:49:46 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:49:56 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:50:06 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:50:16 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:58:07 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:58:17 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 03:58:47 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 03:58:57 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:00:37 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:00:47 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:02:17 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:02:27 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:08:27 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:08:37 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:45:50 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:45:55 (5156): Guest Log: 00:14:51.544755 timesync vgsvcTimeSyncWorker: Radical host time change: 2 242 332 000 000ns (HostNow=1 739 958 354 720 000 000 ns HostLast=1 739 956 112 388 000 000 ns)
2025-02-19 04:46:00 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:47:20 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:47:25 (5156): Guest Log: 00:15:01.545273 timesync vgsvcTimeSyncWorker: Radical guest time change: 3 412 471 639 000ns (GuestNow=1 739 958 364 720 625 000 ns GuestLast=1 739 954 952 248 986 000 ns fSetTimeLastLoop=true )
2025-02-19 04:47:30 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 04:55:20 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 04:55:30 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 05:00:20 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 05:00:30 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 05:00:40 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 05:00:50 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 05:14:21 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 05:14:31 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 05:49:52 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 05:49:57 (5156): Guest Log: 00:15:51.610938 timesync vgsvcTimeSyncWorker: Radical host time change: 2 131 335 000 000ns (HostNow=1 739 962 196 767 000 000 ns HostLast=1 739 960 065 432 000 000 ns)
2025-02-19 05:50:02 (5156): VM state change detected. (old = 'running', new = 'paused')
2025-02-19 05:58:53 (5156): VM state change detected. (old = 'paused', new = 'running')
2025-02-19 05:58:58 (5156): Guest Log: 00:16:01.650046 timesync vgsvcTimeSyncWorker: Radical guest time change: 2 131 364 295 000ns (GuestNow=1 739 962 206 806 227 000 ns GuestLast=1 739 960 075 441 932 000 ns fSetTimeLastLoop=true )
2025-02-19 05:58:59 (5156): Guest Log:  *** Job finished ***
2025-02-19 05:58:59 (5156): Guest Log:  *** The last 20 lines of the pilot log: ***
2025-02-19 05:58:59 (5156): Guest Log: details: error code: 1317, message: Exceeded maximum waiting time
2025-02-19 05:58:59 (5156): Guest Log: details: ('reached maximum waiting time - threads should have finished (ignore exception)',)
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:49:58,199 | WARNING  | job:queue_monitor:received graceful stop - abort after this iteration
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:49:59,206 | INFO     | [job] queue monitor thread has finished
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:49:59,465 | INFO     | [data] queue_monitor thread has finished
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:01,323 | INFO     | only monitor.control thread still running - safe to abort: ['<_MainThread(MainThread, started 140531290982208)>', '<ExcThread(monitor, started 140530700764928)>']
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,325 | INFO     | all workflow threads have been joined
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,326 | INFO     | end of generic workflow (traces error code: 0)
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,326 | INFO     | traces error code: 0
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,326 | INFO     | pilot has finished (exit code=0, shell exit code=0)
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,962 [wrapper] ==== pilot stdout END ====
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,968 [wrapper] ==== wrapper stdout RESUME ====
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,974 [wrapper] pilotpid: 6163
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:06,980 [wrapper] Pilot exit status: 0
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,328 [wrapper] pandaids: 6517864717
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,374 [wrapper] cleanup supervisor_pilot  6169 6164
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,385 [wrapper] Test setup, not cleaning
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,398 [wrapper] apfmon messages muted
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,404 [wrapper] ==== wrapper stdout END ====
2025-02-19 05:58:59 (5156): Guest Log: 2025-02-19 10:50:07,410 [wrapper] ==== wrapper stderr END ====
2025-02-19 05:58:59 (5156): Guest Log:  *** Error codes and diagnostics ***
2025-02-19 05:58:59 (5156): Guest Log:     "exeErrorCode": 65,
2025-02-19 05:58:59 (5156): Guest Log:     "exeErrorDiag": "Non-zero return code from EVNTtoHITS (1); Logfile error in log.EVNTtoHITS: \"XMLCatalog                                                       FATAL UNKNOWN XML parse error in file /cvmfs/atlas-condb.cern.ch/repo/conditions/poolcond/PoolFileCatalog.xml\"",
2025-02-19 05:58:59 (5156): Guest Log:     "pilotErrorCode": 1305,
2025-02-19 05:58:59 (5156): Guest Log:     "pilotErrorDiag": "Failed to execute payload:PyJobTransforms.transform.execute 2025-02-19 08:49:04,859 CRITICAL Transform executor raised TransformValidationException: Non-zero return code from EVNTtoHITS (1); Logfile error in log.EVNTtoHITS: \"XMLCatalog                                                       FATAL UNKNOWN XML ",
2025-02-19 05:58:59 (5156): Guest Log:  *** Listing of results directory ***
2025-02-19 05:58:59 (5156): Guest Log: total 3964
2025-02-19 05:58:59 (5156): Guest Log: -rw-r--r--. 1 atlas atlas  491065 Feb 18 03:16 pilot3.tar.gz
2025-02-19 05:58:59 (5156): Guest Log: -rwx------. 1 atlas atlas   35862 Feb 18 03:40 runpilot2-wrapper.sh
2025-02-19 05:58:59 (5156): Guest Log: -rw-r--r--. 1 atlas atlas    5118 Feb 18 03:40 queuedata.json
2025-02-19 05:58:59 (5156): Guest Log: -rwxr-xr-x. 1 atlas atlas    8615 Feb 18 09:32 init_data.xml
2025-02-19 05:58:59 (5156): Guest Log: -rwxr-xr-x. 1 atlas atlas  503562 Feb 18 09:32 input.tar.gz
2025-02-19 05:58:59 (5156): Guest Log: -rwxr-xr-x. 1 atlas atlas   17569 Feb 18 09:32 start_atlas.sh
2025-02-19 05:58:59 (5156): Guest Log: lrwxrwxrwx. 1 atlas atlas      20 Feb 18 09:32 EVNT.43092757._000058.pool.root.1 -> /data/./ATLAS.root_0
2025-02-19 05:58:59 (5156): Guest Log: -rw-r--r--. 1 atlas atlas    2554 Feb 18 10:51 pandaJob.out
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas     424 Feb 18 10:51 setup.sh.local
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas  989842 Feb 19 05:33 agis_schedconf.cvmfs.json
2025-02-19 05:58:59 (5156): Guest Log: drwx------. 4 atlas atlas    4096 Feb 19 05:33 pilot3
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas    1044 Feb 19 08:49 memory_monitor_summary.json
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas   46343 Feb 19 09:45 log.43205075._043671.job.log.tgz.1
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas 1595357 Feb 19 09:45 agis_ddmendpoints.agis.ALL.json
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas     114 Feb 19 10:14 pilot_heartbeat.json
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas   12717 Feb 19 10:14 heartbeat.json
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas    4113 Feb 19 10:50 pilotlog.txt
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas  100382 Feb 19 10:50 log.43205075._043671.job.log.1
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas     188 Feb 19 10:50 output.list
2025-02-19 05:58:59 (5156): Guest Log: -rw-r--r--. 1 atlas atlas     620 Feb 19 10:50 runtime_log
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas  174080 Feb 19 10:50 result.tar.gz
2025-02-19 05:58:59 (5156): Guest Log: -rw-r--r--. 1 atlas atlas   10802 Feb 19 10:50 runtime_log.err
2025-02-19 05:58:59 (5156): Guest Log: -rw-------. 1 atlas atlas     649 Feb 19 10:50 kHWKDmeeT46nsSi4ap6QjLDmwznN0nGgGQJmkKkKDmBXlKDmY7RQ8m.diag
2025-02-19 05:58:59 (5156): Guest Log: Looking for outputfile HITS.43205075._043671.pool.root.1
2025-02-19 05:58:59 (5156): Guest Log: No HITS file was produced
2025-02-19 05:58:59 (5156): Guest Log: Successfully finished the ATLAS job!
2025-02-19 05:58:59 (5156): Guest Log: Copying the results back to the shared directory!
2025-02-19 05:58:59 (5156): Guest Log:  *** Contents of shared directory: ***
2025-02-19 05:58:59 (5156): Guest Log: total 333368
2025-02-19 05:58:59 (5156): Guest Log: -rwxrwxrwx. 1 root root 340652135 Feb 18 07:56 ATLAS.root_0
2025-02-19 05:58:59 (5156): Guest Log: -rwxrwxrwx. 1 root root      8615 Feb 18 07:56 init_data.xml
2025-02-19 05:58:59 (5156): Guest Log: -rwxrwxrwx. 1 root root    503562 Feb 18 07:47 input.tar.gz
2025-02-19 05:58:59 (5156): Guest Log: -rwxrwxrwx. 1 root root    174080 Feb 19  2025 result.tar.gz
2025-02-19 05:58:59 (5156): Guest Log: -rwxrwxrwx. 1 root root     17569 Feb 18 07:47 start_atlas.sh
2025-02-19 05:58:59 (5156): Guest Log:  *** Success! Shutting down the machine. ***
2025-02-19 05:58:59 (5156): VM Completion File Detected.
2025-02-19 05:58:59 (5156): Powering off VM.
2025-02-19 05:59:01 (5156): Successfully stopped VM.
2025-02-19 05:59:01 (5156): Deregistering VM. (boinc_4b213d9bfa348918, slot#0)
2025-02-19 05:59:01 (5156): Removing network bandwidth throttle group from VM.
2025-02-19 05:59:01 (5156): Removing VM from VirtualBox.
05:59:06 (5156): called boinc_finish(0)

</stderr_txt>
]]>

©2025 CERN