Message boards : ATLAS application : Atlas 2.0 task stalled (again)
Message board moderation

To post messages, you must log in.

AuthorMessage
greg_be

Send message
Joined: 28 Dec 08
Posts: 294
Credit: 2,538,287
RAC: 2,272
Message 43760 - Posted: 1 Dec 2020, 10:18:20 UTC
Last modified: 1 Dec 2020, 10:20:22 UTC

I've been through this once before but don't remember the cause.
This is the second task to stall out in the last stages of running.
Here is lower section of the stderr file:

2020-11-28 21:24:05 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 21:24:27 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 21:35:08 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 21:35:45 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 21:45:58 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 21:46:40 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 21:59:27 (29528): Status Report: Elapsed Time: '6000.825943'
2020-11-28 21:59:27 (29528): Status Report: CPU Time: '151.531250'
2020-11-28 22:07:49 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 22:08:27 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 22:19:06 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 22:19:40 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 22:30:05 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 22:30:43 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 22:41:05 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 22:41:38 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 22:51:40 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 22:52:18 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:02:20 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:03:01 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:13:04 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:13:45 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:23:59 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:24:43 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:34:44 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:35:26 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:46:02 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:46:55 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-28 23:56:56 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-28 23:57:41 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 00:07:47 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 00:08:40 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 00:14:46 (29528): Status Report: Elapsed Time: '12000.864647'
2020-11-29 00:14:46 (29528): Status Report: CPU Time: '256.609375'
2020-11-29 00:18:50 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 00:19:32 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 00:29:57 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 00:30:41 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 00:40:59 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 00:41:44 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 00:52:08 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 00:52:53 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:03:27 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:04:11 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:14:39 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:15:21 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:25:40 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:26:33 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:36:50 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:38:08 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:48:14 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:49:10 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:51:34 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:52:17 (29528): VM state change detected. (old = 'Paused', new = 'Running')
2020-11-29 01:57:03 (29528): VM state change detected. (old = 'Running', new = 'Paused')
2020-11-29 01:57:07 (29528): Stopping VM.
2020-11-30 21:44:01 (25756): Detected: vboxwrapper 26197
2020-11-30 21:44:01 (25756): Detected: BOINC client v7.7
2020-11-30 21:44:01 (25756): Detected: VirtualBox VboxManage Interface (Version: 6.1.16)
2020-11-30 21:44:02 (25756): Starting VM using VBoxManage interface. (boinc_2b3d522e2ddfa7f3, slot#14)
2020-11-30 21:44:16 (25756): Successfully started VM. (PID = '26240')
2020-11-30 21:44:16 (25756): Reporting VM Process ID to BOINC.
2020-11-30 21:44:16 (25756): VM state change detected. (old = 'PoweredOff', new = 'Running')
2020-11-30 21:44:16 (25756): Detected: Web Application Enabled (http://localhost:50289)
2020-11-30 21:44:16 (25756): Detected: Remote Desktop Enabled (localhost:50290)
2020-11-30 21:44:16 (25756): Status Report: Elapsed Time: '16393.666878'
2020-11-30 21:44:16 (25756): Status Report: CPU Time: '340.625000'
2020-11-30 21:44:16 (25756): Preference change detected
2020-11-30 21:44:16 (25756): Setting CPU throttle for VM. (100%)
2020-11-30 21:44:16 (25756): Setting checkpoint interval to 900 seconds. (Higher value of (Preference: 180 seconds) or (Vbox_job.xml: 900 seconds))
2020-11-30 21:44:18 (25756): Guest Log: 05:40:54.142949 timesync vgsvcTimeSyncWorker: Radical host time change: 157 642 087 000 000ns (HostNow=1 606 769 022 052 000 000 ns HostLast=1 606 611 379 965 000 000 ns)

2020-11-30 21:44:28 (25756): Guest Log: 05:41:04.145673 timesync vgsvcTimeSyncWorker: Radical guest time change: 158 778 142 897 000ns (GuestNow=1 606 769 032 054 985 000 ns GuestLast=1 606 610 253 912 088 000 ns fSetTimeLastLoop=true )

2020-11-30 23:24:26 (25756): Status Report: Elapsed Time: '22393.666878'
2020-11-30 23:24:26 (25756): Status Report: CPU Time: '376.609375'
2020-12-01 00:26:20 (25756): Preference change detected
2020-12-01 00:26:20 (25756): Setting CPU throttle for VM. (100%)
2020-12-01 00:26:21 (25756): Setting checkpoint interval to 900 seconds. (Higher value of (Preference: 180 seconds) or (Vbox_job.xml: 900 seconds))
2020-12-01 01:04:35 (25756): Status Report: Elapsed Time: '28393.666878'
2020-12-01 01:04:35 (25756): Status Report: CPU Time: '402.875000'
2020-12-01 02:44:42 (25756): Status Report: Elapsed Time: '34393.666878'
2020-12-01 02:44:42 (25756): Status Report: CPU Time: '429.000000'
2020-12-01 04:24:48 (25756): Status Report: Elapsed Time: '40393.666878'
2020-12-01 04:24:48 (25756): Status Report: CPU Time: '453.656250'
2020-12-01 06:04:54 (25756): Status Report: Elapsed Time: '46393.666878'
2020-12-01 06:04:54 (25756): Status Report: CPU Time: '479.343750'
2020-12-01 07:45:01 (25756): Status Report: Elapsed Time: '52393.666878'
2020-12-01 07:45:01 (25756): Status Report: CPU Time: '505.531250'
2020-12-01 09:25:07 (25756): Status Report: Elapsed Time: '58393.666878'
2020-12-01 09:25:07 (25756): Status Report: CPU Time: '531.218750'
2020-12-01 11:05:13 (25756): Status Report: Elapsed Time: '64393.666878'
2020-12-01 11:05:13 (25756): Status Report: CPU Time: '556.343750'


What's going on? Need to update Vbox or something or clear it?
I'll do that now. But otherwise I have no idea.
BOINC Mgr is set for 9 hrs run time.

CPU usage is .06 and the advance rate is .0010% every 2 seconds.
In one hour the completion estimated time has moved only 1 minute if that.
Task aborted.
ID: 43760 · Report as offensive     Reply Quote

Message boards : ATLAS application : Atlas 2.0 task stalled (again)


©2022 CERN