LHC@home

I thought an update of my inconclusives is warranted here. After the old validator was reimplemented my inconclusives immediately dropped back to 5 from around 18. Since then, they have slowly climbed back to the current level of 19. And, as I stated below (relative to the new validator), the new group is different. That is, I am not only getting inconclusives when paired with x86_64-pc-linux-gnu hosts but now, also with some Windows hosts: see Validation inconclusive tasks for Stick

Don't know if this is good or bad news, but immediately after the validator change, my inconclusive count jumped from 6 to 11. And the new group is very different. Prior to the change, all 6 of my inconclusives were paired against tasks done by x86_64-pc-linux-gnu machines. Now, 4 out of the 5 new ones were pairings between my SixTrack v451.07 (sse2) windows_x86_64 tasks and a variety of machines running SixTrack v451.07 (pni) windows_x86_64.

Inconclusives rising again ?

Situation here is as follows:-
Host 10414945
Total WU 181
In progress 56
Pending 50
Inconclusive 38
Valid 37
Invalid 0
Error 0

---

For all my hosts the numbers are:


Pending 66.03%
Inconclusive 15.31%
Vaild 18.66%

There is still a big backlog on the validator it appears. I will make detailed analysis this weekend.

I took my last 100 Inconclusive results. Some trends

I have all windows, 87% was with a Linux Wingman

Most common CPU's each with 10 AMD FX-8300 (3.19.0-32-generic), E5-2630 v3 (2.6.32-642.15.1.el6.x86_64, Windows 8.1 & Windows 10) & E5-2699C v4 (4.1.12-94.3.5.el7uek.x86_64)

43% have CPU time less than 50sec, 25% above 1000sec on the wingman

Of the short one there is high probabilty that the linux host fails in much shorter time than the windows host. For long one they are much closer together.

Right now I am at......

23 Valid
22 Validation inconclusive
179 Validation pending

Yes I've noticed the my inconclusives rising. Currently sitting at

Validation Pending - 236
Validation Inconclusive - 168
Valid - down from 100 yesterday to currently 83.

https://lhcathome.cern.ch/lhcathome/results.php?userid=250933

Any light at the end of tunnel yet?

Situation here is as follows:-
Host 10414945
Total WU 513
In progress 77
Pending 80
Inconclusive 190
Valid 166
Invalid 0
Error 0

---

The number to look at is the number of Invalid results rather than Inconclusives. If you have zero Invalids then there is nothing wrong with your host. You will need to wait a little longer for the resent job to be returned but this is just similar to having a job Pending from a slower wingman or someone with a larger cache of work than their machine can return in reasonable time.
Eric has been working hard at isolating rogue hosts and optimising the validation process. Better to get an inconclusive result and a short delay than having a bad result falsely validated as good.

I am now waiting for management/support to take action. Eric.

---

Thanks Ray, "short" delay is too long for me, but I am hoping for some management/support action tomorrow latest. Eric.

The number to look at is the number of Invalid results rather than Inconclusives. If you have zero Invalids then there is nothing wrong with your host. You will need to wait a little longer for the resent job to be returned but this is just similar to having a job Pending from a slower wingman or someone with a larger cache of work than their machine can return in reasonable time.
Eric has been working hard at isolating rogue hosts and optimising the validation process. Better to get an inconclusive result and a short delay than having a bad result falsely validated as good.

---

Things are looking much better:

Error 0.02%
Invalid 0.05%
Valid 43.12%
inconclusive 25.58%
Pending 31.23%

Looks like the re-runs are comming though and a 2nd wingman validates.

Thanks Toby, and I can also report that overnight "No consensus"
have dropped 20,516 -> 16,660 -> 12,281 (last 24 hours)
and 445,068 -> 444,151 -> 443,080 (last 7 days). Still some way to
go but patience is called for.

I should also finally, after3 years!, be able to make a proper analysis
of the real invalid results and of other problems.

Things are looking much better:

Error 0.02%
Invalid 0.05%
Valid 43.12%
inconclusive 25.58%
Pending 31.23%

Looks like the re-runs are coming though and a 2nd wingman validates.

---

What's wrong with my statistics?

#https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=70891665
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10483219&offset=80&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10415523&offset=0&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10482091&offset=160&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10450260&offset=20&show_names=0&state=3&appid=

Will look at this soonest. Very interesting and useful. Eric.

What's wrong with my statistics?
#https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=70891665
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10483219&offset=80&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10415523&offset=0&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10482091&offset=160&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10450260&offset=20&show_names=0&state=3&appid=

---

most are waiting for a 3rd host to re-run WU and validate or not.

for 70891665, no one could agree so it's marked as can't validate as the maxiumium number of tries is 5. In this case it's probally a bad WU as the probabilty of 4 different computers being bad is very low.

Eric could comment on how a WU could be bad from a fundemental perspective

Well this is the BIG problem. We had a three hour discussion today, but not
everyone, if anyone, agrees with me. We are seeing too many bad/null results, mixed up
results, but still getting a lot done. I think this a good thing on the long term BUT we must identify the source of these bad/null results. I myself suspect corrupted input and/or
output with good reason. Sadly I guess after max tries the WU will be abandoned. The
work unit is fine though. So no credits for these, and the CERN user will have to re-submit.

Nonetheless I now see a few cases with 3 runs, one rejected and two good results.
I also see 553,615 valid results in the last 7 days. Great, but not good enough.

I plead my case, I am not responsible for these errors nor for all the transient errors we have seen. (I did stop/start the validator on24th June and problems started around 03:00
on 25th.) The sixtrack_validator has been much improved since then by my colleagues.
A side effect is the problem you, and I, and others have seen. If we are lucky and you don't get a 3rd bad result all will be well.

The number of inconclusive, "No consensus" is decreasing but too slowly. I think we must
identify the source of these bad/null/invalid results. (Just my opinion.) The much improved
validator at least allows us to clearly identify the bad tasks.
(Atlas has the same or very similar problem; not sure they know that!)

I have NEVER found a Fundamental problem with a Work Unit. They may exist but
the CERN users are responsible people who check their initial conditions before
submitting to BOINC/LHC@home.

At least the number of inconclusive is down to 5,939 for last 24 hours and while decreasing
is still at 406573 for the last 7 days. but also decreasing. This is in fact good news I think as we are down by almost 40,000. Patience, which I am sadly lacking, is required all round. We live in hope and are working hard to sort this out. Eric.

most are waiting for a 3rd host to re-run WU and validate or not.

for 70891665, no one could agree so it's marked as can't validate as the maxiumium number of tries is 5. In this case it's probally a bad WU as the probability of 4 different computers being bad is very low.

Eric could comment on how a WU could be bad from a fundemental perspective

---

Please see my reply http://lhcathomeclassic.cern.ch/sixtrack/forum_thread.php?id=4306&postid=31311

Is that a hyperlink????

I have the same problem myself with some wzero/jtbb2cm1 cases.

Will look at this soonest. Very interesting and useful. Eric.

What's wrong with my statistics?
#https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=70891665
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10483219&offset=80&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10415523&offset=0&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10482091&offset=160&show_names=0&state=3&appid=
#https://lhcathome.cern.ch/lhcathome/results.php?hostid=10450260&offset=20&show_names=0&state=3&appid=

---

Eric, thank you so much for your huge work you do for all of us.
Did not expect that this simple question would cause such heated debate.

db4m2

( wherein :
db - [D]istributed [B]oinc
4 - [F]or
m - [M]e
2 - [T]o )

errors Too many total results
https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=72453472

Today is Friday.
Need to change
db - [D]istributed [B]oinc
to
db - [D]ouble [B]eer

I have finally understood what you and my colleagues have been trying to tell me! We are in the unfortunate situation where we are unable to distinguish between an infrastructure failure and a case where all particles are lost before completing (typically) a thousand turns. I am a bit slow sometimes and I am obsessed by the unexplained failures of which there are currently too many.

To try and be clear, if I can:

SixTrack has three phases, pre-processing, tracking and post-processing.
Under the present setup, and at the request of a colleague more than ten
years ago, a SixTrack run on LHC@home returns only one file fort.10 as
a result from the post-processing phase.
This was done in order to reduce network traffic and the load on the server.

fort.10, the result, has one very long line for each particle, 60 double
precision floating-point numbers per line, normally 60 lines around 40 KiloBytes.

IF

we have a bad set of initial conditions (very unlikely, but possible) leading to a pre-processing failure

OR

more likely all particles are lost in tracking before completing typically a thousand turns

OR

we never perform post-processing for some other reason

THEN

SixTrack stops and returns an empty fort.10 result.

ENDIF

BUT

Infrastructure failures or run time errors may also produce a null empty result file.

The sixtrack_validator now rejects such null results but clearly identifies
it has done so.

We cannot distinguish between a genuine all particles lost or some other
failure, like a run time crash, segment violation, etc.

These cases will now never be Validated and after 5 attempts the Work Unit
will be scrapped. That is the bad news. Random infrastructure failures should normally be rejected and invalidated as we expect 2 other Valid results.

The relatively good news is that since all particles are lost, little CPU
time, real time, or credits are wasted.

PHEW.... my colleagues are currently trying to test a new SixTrack version
which will allow us to clearly distinguish the reason for a null/empty
result file and our problem will be solved and we should in particular
be able to identify the source of empty results which are not due to
all particles lost.

Summary, No action is required by you. A new SixTrack will be released as
soon as possible. Eric.

---

Is it possible to do the following workaround?

IF

we have a bad set of initial conditions (very unlikely, but possible) leading to a pre-processing failure
Insert an error code in the fort.10

OR

more likely all particles are lost in tracking before completing typically a thousand turns
Insert an error code in the fort.10

OR

we never perform post-processing for some other reason
Insert an error code in the fort.10

THEN

SixTrack stops and returns an empty fort.10 result.
Now SixTrack stops and returns fort.10 with error code(s).

ENDIF

BUT

Infrastructure failures or run time errors may also produce a null empty result file.
Now they can be identified.

The sixtrack_validator now rejects such null results but clearly identifies
it has done so.
The validator (or a separate script) strips the error code lines if there are valid result lines.