LHC@home

Recently I have watched the recordings from the BOINC workshop this year and in a video from LHC@Home it was mentioned that the initial code base for the Sixtrack application goes back almost 30 yrs if I am not mistaken. Furthermore, Laurence Field mentioned in his talk that the application had already successfully been developed/deployed and tested server side at LHC and was found to be performing at a level roughly equivalent of 250 CPU cores.

As this was referred to as an ongoing investigation into the possibility of bringing this app to GPUs here on LHC@Home, I was wondering if so from the project lead / management could provide some information about the plans for the future in the sense of a roadmap for Q3/4 2021 for the Sixtrack application.

Thank you

GPU tasks on OpenPandemics at WorldCommunityGrid take about ten minutes on my GTX 1060, I believe a Pascal chip board. On GPUGRID I have a task running since two days on a GTX 1650 board, which uses a Turing chip. GPU tasks running times depend on many variables, they are not always a solution.
Tullio

---

That much I am aware of. I see your point but don't get really get your reference in regard to my question. I am just curious in the potential energy efficiency boost and core allocation of my machine if this idea were to materialise into a BOINC application. And I wouldn't have brought it up, if it wasn't for the explicit note on the slides in the mentioned BOINC workshop talk.

Would highly appreciate any insight into LHC@H's plans on this!

I cannot Hyper-V my Win 10 PC, so only sixtrack will work for me. I'd love to continue to crunch for little green men/women!

---

Let's crunch for our future.

That much I am aware of. I see your point but don't get really get your reference in regard to my question. I am just curious in the potential energy efficiency boost and core allocation of my machine if this idea were to materialise into a BOINC application. And I wouldn't have brought it up, if it wasn't for the explicit note on the slides in the mentioned BOINC workshop talk.

Would highly appreciate any insight into LHC@H's plans on this!

+1

Here is the link of the YouTube video I referred to in my initial post. You can watch it here: 2021 BOINC Workshop - Day 02 - Laurence Field

By the way, here is a recent news post (27 May, 2021), that mentioned the video recordings of the 2 talks given by LHC representatives at the BOINC workshop. Both very worthwhile watching IMHO. Cheers

In 2016 when a it was investigated sixtrack uses FP64 compute which consumer GPU are very weak on and the memory bandwidth that GPU have are not needed.

https://indico.cern.ch/event/506317/contributions/2017945/attachments/1241758/1826458/SixTrackGPU.pdf

in 2018 it was revisited, you can see that a GTX 1080 is similar same speed as a 20 core low clock speed Xeon CPU, where as a Tesla V100 is 16x quicker.

https://bt.pa.msu.edu/CPO-10/talks/23Tue/AM1/S2F/23Tue_AM1_0915_S2F_De_Maria.pdf

Given that most people don't have a Tesla V100, even if there was a GPU version the real world increase performance would be low, maybe double assuming people have high end GPU's like 1080.

Given that sixtrack work is bursty, it does not seem like CERN even needs more calculations in sixtrack, so why develop a faster method if its not needed

Since sixtrack is open source then I would propose that you can build your on GPU version and test if it gives a performance boost yourself, maybe then you can pitch this to the CERN team for integration or could be deployed via the BOINC anonymous platform from your own hosted source.

Here is also a newer paper from 2020 that has newer CPU's in.

https://indico.cern.ch/event/929467/attachments/2060626/3456407/introduction_sixtracklib_schwinzerl.pdf

Thanks Toby for referencing this helpful information. I will make time tonight to carefully read all about the Sixtrack app. I definitely see why this wasn't enforced/followed up on as nearly no volunteer would meet the requirements that would allow for this performance boost. Just was curious after it was explicitly mentioned in the talk and stated on the slides to know about the application and potential plans surrounding it. And I also see the point of infrequent Sixtrack work submission.

Unfortunately, I am not proficient in the programming languages needed to develop such a CUDA/opencl app. I'll be happy to continue contributing with my little machines to the CPU app version whenever there is work. But I'll take a closer look at the open source code as you sparked my interest to dig a little deeper.

Thx again for your sources! (The last paper/slide deck mentioned from 2020 definitely looks intimidating...)

I definitely see why this wasn't enforced/followed up on as nearly no volunteer would meet the requirements that would allow for this performance boost. Just was curious after it was explicitly mentioned in the talk and stated on the slides to know about the application and potential plans surrounding it. And I also see the point of infrequent Sixtrack work submission.

Maybe will be more wus when LHC will restart (September/October).
I hope gpu will be published...

Kronos Group released OpenCL 3.0.8, with some fixes, clarifications and a "useful extension for accelerated inferencing using quantized neural networks".

Any update?

---

Any update?

Yep, with 8 hrs cpu wus, a gpu app will be welcome :-)

HL-LHC is on the way (2022).
So, no time for new software, to reduce the runtime.
Only the success of proven.

HL-LHC is on the way (2022).

?? HL-LHC will start in 2025. This is the Run3 of LHC.

Hello,

Currently our teams are developing a new tracking code, named XTrack, as an addition/alternative to SixTrack.
As it is written in python with JIT to C, the possibility to run on GPUs was a logical step (also leveraging the experience done by writing sixtracklib, the code referenced above). The code hence runs natively on CPU and GPUs alike.
It is still in its early stages (less than a year old), but is showing remarkable stability and reliance, so much that it is already being used in production research by some people.

The aim is to make our BOINC environment compatible with it, implying that we would be able to start running on BOINC GPUs as well. However this is by far not an easy task, as the current implementation is completely tailored to the SixTrack executables. Hence getting XTrack on BOINC implies a total overhaul of our infrastructure. We will almost certainly do it, but we will need some time. I'd estimate this to be in the order of 1-1.5 years. Difficult to pin point, as it also depends on division of work - we have new people coming in, so the timeframe is rather flexible.

Cheers

Frederik

Exciting news! I'll read through the whole message again later to digest all of the information you relayed to us!

I am curious though as to what your affilition to LHC is as you mention "your teams" a couple times. Your structured and informed message lets me only speculate but I'd guess that you are part of a team of scienctiests working directly with LHC, am I right? Is there already a public Github or Gitlab repo for the XTrack code where we can take a look and track the progress? Can you give us a ballpark number of what performance improvement we could expect with a GPU client version?

Surely, your plan sounds ambitious to deploy the new application here on LHC@Home in the future but I am excited to see the sparked interest in improving the SixTrack code! I'll keep tabs on you and would appreciate if you could occasionally update us on the overall progress.

Have a great week

Back in Oct 2016 milkyway@home had a huge server side performance problem mainly caused by very short runtimes in connection with modern GPUs getting faster and faster.
Details can be seen here together with a suggestion that helps to attenuate the issue:
https://milkyway.cs.rpi.edu/milkyway/forum_thread.php?id=4042#65537

Since then GPUs have reached much higher performance levels, so I'm sure other measures might also be required to ensure the project servers and the DB behind them can handle the expected throughput.
I remember LHC@home already had DB performance issues although it doesn't yet have a GPU app.
The mentioned time frame of 1-1.5 years should be used to develop a solution.

Indeed, I work at CERN on the LHC. I'm Alessio's successor (in being the link between the beams and IT departments concerning BOINC).
Also, as until last year I was the main user of BOINC, you've probably been crunching my jobs :-)