Message boards : ATLAS application : Validate errors all of a sudden
Message board moderation

To post messages, you must log in.

AuthorMessage
Azmodes

Send message
Joined: 26 Sep 17
Posts: 6
Credit: 1,190,866
RAC: 0
Message 37829 - Posted: 26 Jan 2019, 19:18:31 UTC
Last modified: 26 Jan 2019, 19:25:11 UTC

After months without a hitch, I've been getting tons of validate errors on both my machines running ATLAS since yesterday/today. There is no apparent reason for this, I have not changed anything on my side (not that I'm aware of anyway). Has something changed server-side with the tasks?

Stderr output for one of the invalid tasks:
<core_client_version>7.8.3</core_client_version>
<![CDATA[
<stderr_txt>
2019-01-26 12:05:54 (14976): vboxwrapper (7.7.26196): starting
2019-01-26 12:05:54 (14976): Feature: Checkpoint interval offset (434 seconds)
2019-01-26 12:05:54 (14976): Detected: VirtualBox COM Interface (Version: 5.1.26)
2019-01-26 12:05:54 (14976): Detected: Minimum checkpoint interval (900.000000 seconds)
2019-01-26 12:05:54 (14976): Successfully copied 'init_data.xml' to the shared directory.
2019-01-26 12:05:54 (14976): Create VM. (boinc_282685bbe2cf5c9b, slot#10)
2019-01-26 12:05:54 (14976): Setting Memory Size for VM. (6600MB)
2019-01-26 12:05:54 (14976): Setting CPU Count for VM. (4)
2019-01-26 12:05:54 (14976): Setting Chipset Options for VM.
2019-01-26 12:05:54 (14976): Setting Boot Options for VM.
2019-01-26 12:05:54 (14976): Enabling VM Network Access.
2019-01-26 12:05:54 (14976): Setting Network Configuration for NAT.
2019-01-26 12:05:54 (14976): Disabling USB Support for VM.
2019-01-26 12:05:54 (14976): Disabling COM Port Support for VM.
2019-01-26 12:05:54 (14976): Disabling LPT Port Support for VM.
2019-01-26 12:05:54 (14976): Disabling Audio Support for VM.
2019-01-26 12:05:54 (14976): Disabling Clipboard Support for VM.
2019-01-26 12:05:54 (14976): Disabling Drag and Drop Support for VM.
2019-01-26 12:05:54 (14976): Adding storage controller(s) to VM.
2019-01-26 12:05:54 (14976): Adding virtual disk drive to VM. (vm_image.vdi)
2019-01-26 12:05:55 (14976): Adding VirtualBox Guest Additions to VM.
2019-01-26 12:05:55 (14976): Adding network bandwidth throttle group to VM. (Defaulting to 1024GB)
2019-01-26 12:05:55 (14976): forwarding host port 54290 to guest port 80
2019-01-26 12:05:55 (14976): Enabling remote desktop for VM.
2019-01-26 12:05:55 (14976): Required extension pack not installed, remote desktop not enabled.
2019-01-26 12:05:55 (14976): Enabling shared directory for VM.
2019-01-26 12:05:55 (14976): Starting VM. (boinc_282685bbe2cf5c9b, slot#10)
2019-01-26 12:06:09 (14976): Guest Log: BIOS: VirtualBox 5.1.26
2019-01-26 12:06:09 (14976): Guest Log: BIOS: ata0-0: PCHS=16383/16/63 LCHS=1024/255/63
2019-01-26 12:06:09 (14976): Guest Log: BIOS: Boot : bseqnr=1, bootseq=0032
2019-01-26 12:06:09 (14976): Guest Log: BIOS: Booting from Hard Disk...
2019-01-26 12:06:09 (14976): Guest Log: BIOS: KBD: unsupported int 16h function 03
2019-01-26 12:06:09 (14976): Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000 
2019-01-26 12:06:09 (14976): Successfully started VM. (PID = '9856')
2019-01-26 12:06:09 (14976): Reporting VM Process ID to BOINC.
2019-01-26 12:06:19 (14976): VM state change detected. (old = 'poweroff', new = 'running')
2019-01-26 12:06:29 (14976): Detected: Web Application Enabled (http://localhost:54290)
2019-01-26 12:06:39 (14976): Guest Log: vboxguest: major 0, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
2019-01-26 12:06:39 (14976): Preference change detected
2019-01-26 12:06:39 (14976): Setting CPU throttle for VM. (100%)
2019-01-26 12:06:39 (14976): Setting checkpoint interval to 900 seconds. (Higher value of (Preference: 60 seconds) or (Vbox_job.xml: 900 seconds))
2019-01-26 12:06:49 (14976): Guest Log: VBoxGuest: VBoxGuestCommonGuestCapsAcquire: pSession(0xffff8801acb01e10), OR(0x0), NOT(0xffffffff), flags(0x0)
2019-01-26 12:06:49 (14976): Guest Log: VBoxGuest: VBoxGuestCommonGuestCapsAcquire: pSession(0xffff8801ae67e210), OR(0x0), NOT(0xffffffff), flags(0x0)
2019-01-26 12:06:49 (14976): Guest Log: VBoxGuest: VBoxGuestCommonGuestCapsAcquire: pSession(0xffff8801ae678810), OR(0x0), NOT(0xffffffff), flags(0x0)
2019-01-26 12:06:49 (14976): Guest Log: VBoxGuest: VBoxGuestCommonGuestCapsAcquire: pSession(0xffff8801ae67e210), OR(0x0), NOT(0xffffffff), flags(0x0)
2019-01-26 12:07:39 (14976): Guest Log: Copying input files into RunAtlas.
2019-01-26 12:07:39 (14976): Guest Log: Copied input files into RunAtlas.
2019-01-26 12:08:20 (14976): Guest Log: copied the webapp to /var/www
2019-01-26 12:08:20 (14976): Guest Log: This vm does not need to setup http proxy
2019-01-26 12:08:20 (14976): Guest Log: ATHENA_PROC_NUMBER=4
2019-01-26 12:08:20 (14976): Guest Log: Starting ATLAS job. (PandaID=4223075552 taskID=16756637)
2019-01-26 12:18:00 (14976): VM state change detected. (old = 'running', new = 'paused')
2019-01-26 12:20:10 (14976): VM state change detected. (old = 'paused', new = 'running')
2019-01-26 13:20:55 (14976): Guest Log: The last 10 lines of the pilot log.
2019-01-26 13:20:55 (14976): Guest Log:     <metadata att_name="surl" att_value="srm://srm.ndgf.org:8443/srm/managerv2?SFN=/atlas/disk/atlasdatadisk/rucio/mc16_13TeV/cf/ee/HITS.16756637._016930.pool.root.1"/>
2019-01-26 13:20:55 (14976): Guest Log:     <metadata att_name="fsize" att_value="51795579"/>
2019-01-26 13:20:55 (14976): Guest Log:     <metadata att_name="adler32" att_value="848cc5f1"/>
2019-01-26 13:20:55 (14976): Guest Log:   </File>
2019-01-26 13:20:55 (14976): Guest Log: </POOLFILECATALOG>
2019-01-26 13:20:55 (14976): Guest Log: ---------
2019-01-26 13:20:55 (14976): Guest Log: output list
2019-01-26 13:20:55 (14976): Guest Log: HITS.16756637._016930.pool.root.1 srm://srm.ndgf.org:8443;autodir=no;spacetoken=ATLASDATADISK/srm/managerv2?SFN=/atlas/disk/atlasdatadisk/rucio/mc16_13TeV/cf/ee/HITS.16756637._016930.pool.root.1:checksumtype=adler32:checksumvalue=848cc5f1
2019-01-26 13:20:55 (14976): Guest Log: log.16756637._016930.job.log.tgz.1 srm://srm.ndgf.org:8443;autodir=no;spacetoken=ATLASDATADISK/srm/managerv2?SFN=/atlas/disk/atlasdatadisk/rucio/mc16_13TeV/52/66/log.16756637._016930.job.log.tgz.1:checksumtype=adler32:checksumvalue=e0ad8069
2019-01-26 13:20:55 (14976): Guest Log: Listing of results directory
2019-01-26 13:20:55 (14976): Guest Log: total 200152
2019-01-26 13:20:55 (14976): Guest Log: -rw-r--r-- 1 atlas01 atlas01      4436 Jan 25 23:38 queuedata.pilot.json
2019-01-26 13:20:55 (14976): Guest Log: -rwx------ 1 atlas01 atlas01      2441 Jan 25 23:38 ARCpilot
2019-01-26 13:20:55 (14976): Guest Log: -rwxr-xr-x 1 atlas01 atlas01 136357100 Jan 26 12:07 EVNT.16513725._000322.pool.root.1
2019-01-26 13:20:55 (14976): Guest Log: -rwxr-xr-x 1 atlas01 atlas01      8929 Jan 26 12:07 init_data.xml
2019-01-26 13:20:55 (14976): Guest Log: -rwxr-xr-x 1 atlas01 atlas01   1088716 Jan 26 12:07 input.tar.gz
2019-01-26 13:20:55 (14976): Guest Log: -rwxr-xr-x 1 atlas01 atlas01       606 Jan 26 12:07 RTE.tar.gz
2019-01-26 13:20:55 (14976): Guest Log: -rwxr-xr-x 1 atlas01 atlas01     14356 Jan 26 12:07 start_atlas.sh
2019-01-26 13:20:55 (14976): Guest Log: drwxr-xr-x 3 atlas01 atlas01      4096 Jan 26 12:08 APPS
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01      2922 Jan 26 12:08 pandaJobData.out
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01        22 Jan 26 12:08 PILOT_INITDIR
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01        50 Jan 26 12:08 CURRENT_SITEWORKDIR
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01     11430 Jan 26 12:08 pilotlog.txt
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01         1 Jan 26 12:08 STATUSCODE
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01        11 Jan 26 12:08 pandaIDs.out
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01        19 Jan 26 12:08 START_TIME_4223075552
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01      3801 Jan 26 12:08 queuedata.json
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01      2922 Jan 26 12:08 pandaJobData_1.out
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01   7117981 Jan 26 12:11 agis_ddmendpoints.cvmfs.json
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01   5394756 Jan 26 12:11 agis_schedconf.cvmfs.json
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01  51795579 Jan 26 13:17 HITS.16756637._016930.pool.root.1
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       308 Jan 26 13:18 memory_monitor_summary.json
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       165 Jan 26 13:19 workdir_size-4223075552.json
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01    779304 Jan 26 13:19 log.16756637._016930.job.log.tgz.1
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       933 Jan 26 13:19 OutputFiles-4223075552.xml
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01      1023 Jan 26 13:19 metadata-surl.xml
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01     32448 Jan 26 13:19 panda_node_struct.pickle
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       137 Jan 26 13:20 pilotlog-last.txt
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01    469039 Jan 26 13:20 log.16756637._016930.job.log.1
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01      5704 Jan 26 13:20 jobSmallFiles.tgz
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       497 Jan 26 13:20 1hyMDmT1N7tnyYickojUe11pABFKDmABFKDmflXYDmABFKDmOQaOTn.diag
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01       463 Jan 26 13:20 output.list
2019-01-26 13:20:55 (14976): Guest Log: -rw-r--r-- 1 atlas01 atlas01     10283 Jan 26 13:20 runtime_log.err
2019-01-26 13:20:55 (14976): Guest Log: -rw-r--r-- 1 atlas01 atlas01       604 Jan 26 13:20 runtime_log
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01   1740800 Jan 26 13:20 result.tar.gz
2019-01-26 13:20:55 (14976): Guest Log: HITS file was successfully produced
2019-01-26 13:20:55 (14976): Guest Log: -rw------- 1 atlas01 atlas01 51795579 Jan 26 13:17 /home/atlas01/RunAtlas/HITS.16756637._016930.pool.root.1
2019-01-26 13:20:55 (14976): VM Completion File Detected.
2019-01-26 13:20:55 (14976): Powering off VM.
2019-01-26 13:20:58 (14976): Successfully stopped VM.
2019-01-26 13:21:03 (14976): Deregistering VM. (boinc_282685bbe2cf5c9b, slot#10)
2019-01-26 13:21:03 (14976): Removing virtual disk drive(s) from VM.
2019-01-26 13:21:03 (14976): Removing network bandwidth throttle group from VM.
2019-01-26 13:21:03 (14976): Removing storage controller(s) from VM.
2019-01-26 13:21:03 (14976): Removing VM from VirtualBox.
13:21:08 (14976): called boinc_finish(0)

</stderr_txt>
]]>
ID: 37829 · Report as offensive     Reply Quote
Azmodes

Send message
Joined: 26 Sep 17
Posts: 6
Credit: 1,190,866
RAC: 0
Message 37830 - Posted: 26 Jan 2019, 19:24:27 UTC
Last modified: 26 Jan 2019, 19:24:48 UTC

Related to this?
ID: 37830 · Report as offensive     Reply Quote

Message boards : ATLAS application : Validate errors all of a sudden


©2024 CERN