LHC@home

Hi!

I have had issues with computational errors on LHC-classic (not running with virtual-box) on a 64 core Opteron machine when many work-units have been ready at the same time, or when suspending/resuming operations. I have suspected some bug in the Linux-numa kernel but not being able to pinpoint the problem (other applications have been running fine).

Recently i have got a Threadripper 16 core/32thread machine. When testing with LHC on that machine i noticed that the disk was accessed continuously (standard HDD, no SSD), giving very poor response time for starting other programs.

I also got some errors while computing on this machine. So I moved the BOINC work folders to ram-disk (/dev/shm) and since then i have not seen any "Errors while computing" on either the Threadripper or the Opteron machine.

Machines are:
ID: 10359398
ID: 10515966

So I guess there is some timing-issue between the boinc client and the six-track application.

Would be good if the team could look into this.

---

You may check your "load average" values ("top" or "cat /proc/loadavg").
They shouldn't be much higher than your core numbers, i.e. 32 on your TR and 64 on your Opteron.
Much higher values indicate some bottleneck, in your case, I guess, not enough RAM.

If the values are too high, reduce the number of concurrently running tasks.

In addition you may check if (and how much and how often) your systems use swap space.

The data below is with all the boinc-stuff residing in memory:

On the TR machine with 32 sixtrack + 1 setiathome-cuda + firefox::

top - 09:06:51 up 1 day, 9:06, 13 users, load average: 33.42, 33.81, 33.90
Tasks: 521 total, 36 running, 304 sleeping, 0 stopped, 2 zombie
%Cpu(s): 1.9 us, 0.1 sy, 93.1 ni, 4.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16355028 total, 2138648 free, 12150888 used, 2065492 buff/cache
KiB Swap: 33554428 total, 33344764 free, 209664 used. 2268360 avail Mem

On the opteron machine, 64 sixtrack:

top - 09:08:56 up 5 days, 16:24, 4 users, load average: 66.14, 66.75, 66.65
Tasks: 772 total, 65 running, 395 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.4%us, 0.0%sy, 97.4%ni, 1.9%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32904536k total, 26145756k used, 6758780k free, 378784k buffers
Swap: 33554428k total, 267776k used, 33286652k free, 2594904k cached

So seems OK. When having the boinc-stuff on disk memory usage on the TR machine was almost 100% (more active disk-cache), but load was about the same.

I also tried to increase the checkpoint time from 60 seconds to 600, but no big difference, should maybe try to increase to one hour to see if it helps.

---

There's nothing wrong with the numbers you posted.
Your machines operate around 100% but are not yet overloaded.

However in special situations (see your first post) more RAM and/or faster disks could be helpful.
I suspect it's not the application itself but an "unpatient" BOINC client that causes your problems.

My suggestions:
- Install more RAM.
- Avoid using a RAM disk (it's fast but using the RAM for the disk cache is more efficient).
- Reduce your swappiness "sysctl vm.swappiness=1".
- Install a 2nd disk.
- Setup a 2nd BOINC instance on the 2nd disk and split the work between both instances.
- Stagger your shutdown, i.e. do not request a shutdown with 32/64 running apps.

My suggestions:
- Install more RAM.
- Avoid using a RAM disk (it's fast but using the RAM for the disk cache is more efficient).
- Reduce your swappiness "sysctl vm.swappiness=1".

In addition to those suggestions, you could also set up a write cache, or actually increase the size, since it is there by default anyway in Linux.

For 16 GB memory I would suggest starting with these (which work in Ubuntu):

Set write cache to 4 GB for 16 GB main memory:

sudo sysctl vm.dirty_background_bytes=4000000000 (size of cache when the pages start being flushed to disk)
sudo sysctl vm.dirty_bytes=5000000000 (maximum size of cache before halting further writes)
sudo sysctl vm.dirty_writeback_centisecs=500 (checks the cache every 5 seconds)
sudo sysctl vm.dirty_expire_centisecs=360000 (pages older than 1 hour are flushed)

I actually use larger values with 32 GB memory, but these should help reduce the disk thrashing. One advantage of a cache over a ramdisk is that Linux will automatically reclaim cache space if it is needed for working memory (it is said), though I don't normally have to put that to the test.

My suggestions:
- Install more RAM.
- Avoid using a RAM disk (it's fast but using the RAM for the disk cache is more efficient).
- Reduce your swappiness "sysctl vm.swappiness=1".

In addition to those suggestions, you could also set up a write cache, or actually increase the size, since it is there by default anyway in Linux.

For 16 GB memory I would suggest starting with these (which work in Ubuntu):

Set write cache to 4 GB for 16 GB main memory:

sudo sysctl vm.dirty_background_bytes=4000000000 (size of cache when the pages start being flushed to disk)
sudo sysctl vm.dirty_bytes=5000000000 (maximum size of cache before halting further writes)
sudo sysctl vm.dirty_writeback_centisecs=500 (checks the cache every 5 seconds)
sudo sysctl vm.dirty_expire_centisecs=360000 (pages older than 1 hour are flushed)

I actually use larger values with 32 GB memory, but these should help reduce the disk thrashing. One advantage of a cache over a ramdisk is that Linux will automatically reclaim cache space if it is needed for working memory (it is said), though I don't normally have to put that to the test.

Tanks, will test this when i start running again, right now there is som other stuff running. Further, the sixtrack servers seems overloaded as there is so many uploads that downloads was disabled.

---

[quote]
My suggestions:
- Install more RAM.
- Avoid using a RAM disk (it's fast but using the RAM for the disk cache is more efficient).
- Reduce your swappiness "sysctl vm.swappiness=1".

In addition to those suggestions, you could also set up a write cache, or actually increase the size, since it is there by default anyway in Linux.

For 16 GB memory I would suggest starting with these (which work in Ubuntu):

Set write cache to 4 GB for 16 GB main memory:

sudo sysctl vm.dirty_background_bytes=4000000000 (size of cache when the pages start being flushed to disk)
sudo sysctl vm.dirty_bytes=5000000000 (maximum size of cache before halting further writes)
sudo sysctl vm.dirty_writeback_centisecs=500 (checks the cache every 5 seconds)
sudo sysctl vm.dirty_expire_centisecs=360000 (pages older than 1 hour are flushed)

I actually use larger values with 32 GB memory, but these should help reduce the disk thrashing. One advantage of a cache over a ramdisk is that Linux will automatically reclaim cache space if it is needed for working memory (it is said), though I don't normally have to put that to the test.

Tanks, will test this when i start running again, right now there is som other stuff running. Further, the sixtrack servers seems overloaded as there is so many uploads that downloads was disabled.

Update:
I have been testing some more for a time... Setting up the cache did help for a while, but when the cache was full the disk could not cope with the writes, with random writes to a mechanical disk from 32 threads I get a write speed of less than 1 MByte/sec. So 16 threads seems to be about what a single disk can handle. But this should be no problem for a SSD. Also tried an USB-stick, no problem during normal run, but with the 10-second work units errors started to occur. As a side note, with an USB-stick and the settings as above it took about 5 minutes to sync when doing a reboot.

So I am back to having the BOINC-folder on /dev/shm, takes about 1GB of RAM, so less than the cache and only expands as needed.
Main drawback as I see it is you have to remember to copy it to/from disk when restarting the machine.

Would be good if the timing issue could be fixed in either the sixtrack application or the boinc client.

---

Hello Thord,
Thanks for the feedback on short SixTrack tasks - also other volunteers reported the same issue.
On the SixTrack side we are working on a filtering process, so that the very short tasks (few seconds only) are not sent to volunteers, wasting resources.
Cheers,
A.

... So I am back to having the BOINC-folder on /dev/shm ...

If you use shm or tmpfs to shadow your BOINC folders you may save some bytes of RAM if you only place the slots folder there and leave the remaining folders on the normal storage.
In addition it needs a strict monitoring of the RAM usage.