Message boards : News : Heavy I/O on Windows WUs
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Profile Tom95134

Send message
Joined: 4 May 07
Posts: 250
Credit: 826,541
RAC: 0
Message 26994 - Posted: 15 Nov 2014, 19:03:12 UTC

John,

I noticed the same slowdown when "w-" Tasks start to overwhelm the system. The approach I took was to use BOINC Manager Tasks to selectively suspend two or three of them which allows other Projects to run Tasks. I'll Resume these Suspended Tasks after a few hours and non-"w" Tasks are running on LHC.

Yeah, it's a pain to have to manually manage these things but it got me through the terrible "w"s.

Tom95134
ID: 26994 · Report as offensive     Reply Quote
Eric Mcintosh
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 12 Jul 11
Posts: 851
Credit: 1,612,697
RAC: 181
Message 26995 - Posted: 17 Nov 2014, 5:21:39 UTC - in response to Message 26981.  

Fascinating Carolina; I believe we buffer and flush only when
doing a checkpoint. 30 files is correct but we have a plan
to use only one file. The actual records are small about
one hundred bytes. Thanks. Eric.
(NOT a Windows expert.)
ID: 26995 · Report as offensive     Reply Quote
Profile Tom95134

Send message
Joined: 4 May 07
Posts: 250
Credit: 826,541
RAC: 0
Message 27000 - Posted: 18 Nov 2014, 3:57:14 UTC

Eric,

FYI. The new version of BOINC (7.4.27) includes the ability to set which debug information is included in the BOINC Log. I haven't tried it but it looks like there are a number of settings that include disk info.

I just loaded 7.4.27 late this afternoon to see how it runs. vLHC and Enigma seem to be okay. Hopefully we'll have some more LHC work soon. SETI has gone quiet.

Tom
ID: 27000 · Report as offensive     Reply Quote
Carolina Calling

Send message
Joined: 9 May 07
Posts: 10
Credit: 783,966
RAC: 0
Message 27001 - Posted: 19 Nov 2014, 11:03:43 UTC - in response to Message 26995.  

I'm glad that I got the number of "fort" files per task correct. The problem I have is that my system tends to be running more than one SixTrack job at a time and the impact of the I/O is crippling. The "fort" files are not normally soooo chopped up either. I was astounded to find five SixTrack tasks had ~240 file in over 360,000 fragments. It particularly impacts job creation (when there is heavy use of the swap file).

How often does SixTrack checkpoint? Is it possible that it is somehow checkpointing inappropriately? What controls when SixTrack checkpoints?

Thanks for the response!

-- CCW
ID: 27001 · Report as offensive     Reply Quote
Profile yo2013
Avatar

Send message
Joined: 16 Oct 13
Posts: 50
Credit: 253,113
RAC: 0
Message 27002 - Posted: 19 Nov 2014, 17:03:00 UTC - in response to Message 27001.  


How often does SixTrack checkpoint?


It depends on your settings.

If you refer to disk writing instead (not all disk writing is due to checkpointing), in my Linux boxes it writes every 10 seconds or so (too much, IMO).
ID: 27002 · Report as offensive     Reply Quote
Leo Hoodak

Send message
Joined: 12 Aug 05
Posts: 2
Credit: 162,587
RAC: 0
Message 27007 - Posted: 25 Nov 2014, 1:31:06 UTC

I just started runing these wu on my 8 core home computer with Win7. I can still use my computer if I cut back to 2 cpu's. If I use 4 of 8, you go nowhere. I have 16gb ram, and a Nvidia 275 gpu. I can run Seti on 6 cpu's, and no real slowdown on anything else I do.
ID: 27007 · Report as offensive     Reply Quote
pls

Send message
Joined: 22 Oct 07
Posts: 25
Credit: 808,821
RAC: 0
Message 27035 - Posted: 11 Dec 2014, 5:30:42 UTC

Maybe I can provide a bit more information.

I am running on Windows XP Sp3, running SixTrack 451.07 on a "w-" work unit.

As I write this, BOINC shows that the WU has been running for 14:32 and is 93.491% complete. Attaching a monitor to the process shows that it has used the following

CPU
User time: 00:14:00
Kernel time: 11:22:18

I/O
Reads: 1,990 and not increasing
Writes: 1,805,531
Write Delta: 82 per 5 secs
Write Bytes Delta: 7000
Calculated Write Size: 85 bytes
Other: 5,228,924
Other Delta: 241 per 5 secs
Other Delta Bytes: 320

It appears that an ioctl is doing a lot of 1 byte writes

1 active thread accounts for almost all of the CPU time.
Pulling a stack traces for this thread gives (3 examples)

ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntdll.dll!KiFastSystemCallRet
sixtrack_win32_4517_pni.exe+0x24b141
sixtrack_win32_4517_pni.exe+0x190069
sixtrack_win32_4517_pni.exe+0x1796b4
sixtrack_win32_4517_pni.exe+0x1790e4
sixtrack_win32_4517_pni.exe+0x62c6d
sixtrack_win32_4517_pni.exe+0x2cc842
sixtrack_win32_4517_pni.exe+0x295970
kernel32.dll!RegisterWaitForInputIdle+0x49


ntkrnlpa.exe!ZwYieldExecution+0x1d3a
sixtrack_win32_4517_pni.exe+0xa12254
ntkrnlpa.exe!IoBuildPartialMdl+0xed
fltMgr.sys!FltPerformSynchronousIo+0xb9
fltMgr.sys!FltFlushBuffers+0x314
fltMgr.sys!FltFsControlFile+0x26
fltMgr.sys!FltRequestOperationStatusCallback+0x5bd
fltMgr.sys!FltGetIrpName+0x57a
fltMgr.sys!FltGetIrpName+0xaa9
fltMgr.sys!FltGetIrpName+0x113b
fltMgr.sys!FltGetIrpName+0x12ad
ntkrnlpa.exe!IoBuildPartialMdl+0xed
ntkrnlpa.exe!KeReleaseInStackQueuedSpinLockFromDpcLevel+0xb80
ntdll.dll!KiFastSystemCallRet
sixtrack_win32_4517_pni.exe+0x25cb8b Seen this address a lot
sixtrack_win32_4517_pni.exe+0x23bece
sixtrack_win32_4517_pni.exe+0x19003e
sixtrack_win32_4517_pni.exe+0x1796b4
sixtrack_win32_4517_pni.exe+0x1790e4
sixtrack_win32_4517_pni.exe+0x62c6d
sixtrack_win32_4517_pni.exe+0x2cc842
sixtrack_win32_4517_pni.exe+0x295970
kernel32.dll!RegisterWaitForInputIdle+0x49


ntkrnlpa.exe!KiUnexpectedInterrupt+0x121
ntkrnlpa.exe!ZwYieldExecution+0x1d3a
ntkrnlpa.exe!KiDeliverApc+0x12c
ntdll.dll!KiFastSystemCallRet
sixtrack_win32_4517_pni.exe+0x24b141
sixtrack_win32_4517_pni.exe+0x190069
sixtrack_win32_4517_pni.exe+0x1796b4
sixtrack_win32_4517_pni.exe+0x1790e4
sixtrack_win32_4517_pni.exe+0x62c6d
sixtrack_win32_4517_pni.exe+0x2cc842
sixtrack_win32_4517_pni.exe+0x295970
kernel32.dll!RegisterWaitForInputIdle+0x49


I haven't dug deeper. I can if it's useful but I'll need the program source.

To be honest, I think running this is a waste of time. My CPUs are spinning their wheels in a system call, not doing computation. I've set this project to "no new work" until I hear there is a fix. Please let me know if you want me to do more.

++PLS
ID: 27035 · Report as offensive     Reply Quote
Previous · 1 · 2

Message boards : News : Heavy I/O on Windows WUs


©2019 CERN