Message boards :
Sixtrack Application :
Sixtrack Error while computing on machine with many cores and "slow" storage.
Message board moderation
Author | Message |
---|---|
Send message Joined: 30 Jul 05 Posts: 7 Credit: 18,326,959 RAC: 0 |
Hi! I have had issues with computational errors on LHC-classic (not running with virtual-box) on a 64 core Opteron machine when many work-units have been ready at the same time, or when suspending/resuming operations. I have suspected some bug in the Linux-numa kernel but not being able to pinpoint the problem (other applications have been running fine). Recently i have got a Threadripper 16 core/32thread machine. When testing with LHC on that machine i noticed that the disk was accessed continuously (standard HDD, no SSD), giving very poor response time for starting other programs. I also got some errors while computing on this machine. So I moved the BOINC work folders to ram-disk (/dev/shm) and since then i have not seen any "Errors while computing" on either the Threadripper or the Opteron machine. Machines are: ID: 10359398 ID: 10515966 So I guess there is some timing-issue between the boinc client and the six-track application. Would be good if the team could look into this. |
Send message Joined: 15 Jun 08 Posts: 2411 Credit: 226,413,747 RAC: 131,862 |
You may check your "load average" values ("top" or "cat /proc/loadavg"). They shouldn't be much higher than your core numbers, i.e. 32 on your TR and 64 on your Opteron. Much higher values indicate some bottleneck, in your case, I guess, not enough RAM. If the values are too high, reduce the number of concurrently running tasks. In addition you may check if (and how much and how often) your systems use swap space. |
Send message Joined: 30 Jul 05 Posts: 7 Credit: 18,326,959 RAC: 0 |
The data below is with all the boinc-stuff residing in memory: On the TR machine with 32 sixtrack + 1 setiathome-cuda + firefox:: top - 09:06:51 up 1 day, 9:06, 13 users, load average: 33.42, 33.81, 33.90 Tasks: 521 total, 36 running, 304 sleeping, 0 stopped, 2 zombie %Cpu(s): 1.9 us, 0.1 sy, 93.1 ni, 4.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16355028 total, 2138648 free, 12150888 used, 2065492 buff/cache KiB Swap: 33554428 total, 33344764 free, 209664 used. 2268360 avail Mem On the opteron machine, 64 sixtrack: top - 09:08:56 up 5 days, 16:24, 4 users, load average: 66.14, 66.75, 66.65 Tasks: 772 total, 65 running, 395 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 0.0%sy, 97.4%ni, 1.9%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 32904536k total, 26145756k used, 6758780k free, 378784k buffers Swap: 33554428k total, 267776k used, 33286652k free, 2594904k cached So seems OK. When having the boinc-stuff on disk memory usage on the TR machine was almost 100% (more active disk-cache), but load was about the same. I also tried to increase the checkpoint time from 60 seconds to 600, but no big difference, should maybe try to increase to one hour to see if it helps. |
Send message Joined: 15 Jun 08 Posts: 2411 Credit: 226,413,747 RAC: 131,862 |
There's nothing wrong with the numbers you posted. Your machines operate around 100% but are not yet overloaded. However in special situations (see your first post) more RAM and/or faster disks could be helpful. I suspect it's not the application itself but an "unpatient" BOINC client that causes your problems. My suggestions: - Install more RAM. - Avoid using a RAM disk (it's fast but using the RAM for the disk cache is more efficient). - Reduce your swappiness "sysctl vm.swappiness=1". - Install a 2nd disk. - Setup a 2nd BOINC instance on the 2nd disk and split the work between both instances. - Stagger your shutdown, i.e. do not request a shutdown with 32/64 running apps. |
Send message Joined: 15 Nov 14 Posts: 602 Credit: 24,371,321 RAC: 0 |
In addition to those suggestions, you could also set up a write cache, or actually increase the size, since it is there by default anyway in Linux. For 16 GB memory I would suggest starting with these (which work in Ubuntu): Set write cache to 4 GB for 16 GB main memory: sudo sysctl vm.dirty_background_bytes=4000000000 (size of cache when the pages start being flushed to disk) I actually use larger values with 32 GB memory, but these should help reduce the disk thrashing. One advantage of a cache over a ramdisk is that Linux will automatically reclaim cache space if it is needed for working memory (it is said), though I don't normally have to put that to the test. |
Send message Joined: 30 Jul 05 Posts: 7 Credit: 18,326,959 RAC: 0 |
Tanks, will test this when i start running again, right now there is som other stuff running. Further, the sixtrack servers seems overloaded as there is so many uploads that downloads was disabled. |
Send message Joined: 30 Jul 05 Posts: 7 Credit: 18,326,959 RAC: 0 |
[quote] Tanks, will test this when i start running again, right now there is som other stuff running. Further, the sixtrack servers seems overloaded as there is so many uploads that downloads was disabled. Update: I have been testing some more for a time... Setting up the cache did help for a while, but when the cache was full the disk could not cope with the writes, with random writes to a mechanical disk from 32 threads I get a write speed of less than 1 MByte/sec. So 16 threads seems to be about what a single disk can handle. But this should be no problem for a SSD. Also tried an USB-stick, no problem during normal run, but with the 10-second work units errors started to occur. As a side note, with an USB-stick and the settings as above it took about 5 minutes to sync when doing a reboot. So I am back to having the BOINC-folder on /dev/shm, takes about 1GB of RAM, so less than the cache and only expands as needed. Main drawback as I see it is you have to remember to copy it to/from disk when restarting the machine. Would be good if the timing issue could be fixed in either the sixtrack application or the boinc client. |
Send message Joined: 29 Feb 16 Posts: 157 Credit: 2,659,975 RAC: 0 |
Hello Thord, Thanks for the feedback on short SixTrack tasks - also other volunteers reported the same issue. On the SixTrack side we are working on a filtering process, so that the very short tasks (few seconds only) are not sent to volunteers, wasting resources. Cheers, A. |
Send message Joined: 15 Jun 08 Posts: 2411 Credit: 226,413,747 RAC: 131,862 |
... So I am back to having the BOINC-folder on /dev/shm ... If you use shm or tmpfs to shadow your BOINC folders you may save some bytes of RAM if you only place the slots folder there and leave the remaining folders on the normal storage. In addition it needs a strict monitoring of the RAM usage. |
©2024 CERN