LHC@home

Trying out 3 rights sounds like a good plan to test. However, don't you still need to have a total of 5, like we used to have? Right now (if I understand it correctly) having a total of 3 means that if 1 fails, it will not send out new and noone will get credit? Or is it just me reading it wrong?

So it will be 3/5/3 (3 wrong, 5 total, 3 right). Then the least resource will be lost and it will still have a chance to validate even though 1 of the 3 are wrong.

Almost, if I am right, if the three don't match an additional WU is put to
the end of the queue :-( and should be assigned to a "reliable" host.
So unless I get three empty/null results, I should get a good result, or by
a miracle I may get the same bad result from three hosts and that would be
a really good clue. Eric.
P.S. I seem to remember a comment about available work that I can't find
(03:00 here!). Work tends to come in batches and often different customers
at the same time. Since we have first in first out the different customers
then slow each other down in real time. I think we should rather run three
different studies consecutively, 1st user waits three days say, 2nd 6 days,
and 3rd 9 days. Is this better than 3 cutomers waiting 9 days? A classic
scheduling issue. I am thinking of introducing an additional layer of
control and monitoring between the customer and BOINC. This would allow,
customer to enquire about study progress, CERN speciific scheduling, and
better management of CERN local resources like disk space.
FIRST solve the annoying result differences!!!

---

Interestingly I see significant drop in "inconclusive" results last days - only 2-4 per day, not scores as before.

Remember I suspended a big culprit, but it takes time to become
effective. Expect a real fix soonest. Eric.

---

P.S. I seem to remember a comment about available work that I can't find
(03:00 here!). Work tends to come in batches and often different customers
at the same time. Since we have first in first out the different customers
then slow each other down in real time. I think we should rather run three
different studies consecutively, 1st user waits three days say, 2nd 6 days,
and 3rd 9 days. Is this better than 3 cutomers waiting 9 days? A classic
scheduling issue. I am thinking of introducing an additional layer of
control and monitoring between the customer and BOINC. This would allow,
customer to enquire about study progress, CERN speciific scheduling, and
better management of CERN local resources like disk space.
FIRST solve the annoying result differences!!!

I remember making a remark like that last time we had Lots of Work!

It was a fairly trivial throw-away comment at the time, but a few days later we had quite a big problem with Error reported by file upload server: Server is out of disk space. I can't help thinking the two events were related, and precisely for that reason - if a tie-breaker needs to be issued and goes to the end of a very long 'all researchers' combined queue, then all the result files for the workunit have to hang around for that much longer and clog up all the pipes.

Right; in the meantime I have just found a bug with my gfortzran
version, either in gcc or the gcc/gfortran interface. I am not pursuing
it because production is with ifort and all works fine with older
versions of gcc/gfortran. I am really praying that my current Easter
tests will give insight to the current ifort production invalid results.
"Only" a very small percentage of WUs, but really annoying. Eric.

(One day I hope to get SixTrack into SpecFP tests again or at least
make it freely available as a compiler/hardware test program.)

---

... having a total of 3 means that if 1 fails, it will not send out new and noone will get credit?

I think you mean about this: 30565542

I am one of the two computers with 17000 seconds. The faulty computer 10353061 nearly broken all tasks. This computer has a "EXIT_TIME_LIMIT_EXCEEDED" error after exactly 252.06 or 2520.60 seconds on all tasks. Unusual.

Bye, Grubix.


PS: I'm not so interested in credits, but in science. :-)

Jep. That was what I was thinking about. Same here, in it for the Science and not the credits. I just fear that if a task gets rejected it's lost work even though 2 computers came with the correct answer.

---

Well thanks for your patience and support. I reckon I messed up
again. The error limit of 3 is clearly wrong. Should be much bigger,
like 10 say. I just tried to change it and I am not allowed to.
I don't know why. I changed the file just yesterday. However,
ls -l shows it hasn't changed since last October. i
Need to try again later. Eric.

---

Just an example...
Einstein@home uses minimum quorum 2, maximum tasks - 20...
Believe or not, by this extremely high reserve, rarely but sometimes several WUs are getting status "to many errors"... ;-)

---

Just an example...
Einstein@home uses minimum quorum 2, maximum tasks - 20...
Believe or not, by this extremely high reserve, rarely but sometimes several WUs are getting status "to many errors"... ;-)

That approach works for Einstein, because they are performing long searches (many months) through large but consistent datasets. The preparation of individual workunits from those datasets can be automated, and very rarely results in significant numbers of 'impossible' tasks. When they do happen, it's usually because the staff are preparing a new search or application, and are paying close attention so they can catch problems with a small test batch quickly.

Here, I get the impression that there's more direct human input into the preparation of smaller production batches. It's still rare, but it's more likely that one whole batch will go wrong. With a very high maximum replication number, and occasional very long queues, any bad batch would recirculate through the system for a long time until the very last WU met its maker.

We certainly need a 'maximum tasks' limit high enough to allow for some errors to happen before quorum is reached, but as in life - moderation in all things.

Thanks Richard; for production we use as shown below.
I really goofed up with my test runs. Eric.

# The number of redundant calculations. Set to two or more to achieve redudancy.
redundancy=2

# The number of copies of the workunit to issue to clients. Must be at least the
# number of redundant calculations or higher if a loss of results is expected
# or if the result should to be obtained fast.
copies=2

# The number of errors from clients before the workunit is declared to have
# an error.
errors=5

# The total number of clients to issue the workunit to before it is declared
# to have an error.
numIssues=5

# The total number of returned results without a concensus is found before the
# workunit is declared to have an error.
resultsWithoutConcensus=3

---

Yes, it's a complicated juggling act - made more so for users, because we only see the simplified "max # of error/total/success tasks", and I wasn't aware of some of the nuances of definition of some of those entries until today.

It looks as if https://boinc.berkeley.edu/trac/wiki/JobIn#scheduling is the place to be. I think I come to the conclusion - but I could well be wrong - that:

max_success_results (your resultsWithoutConcensus) should always be above min_quorum (your redundancy), but perhaps only by one - that would allow one inconclusive result, but not go on to allow an increased risk of two bad hosts validating each other. So these two values should march in step - if one changes, they both should be changed.

max_error_results (your errors) could perhaps come down a bit, depending on your observed sporadic error rate across the population of hosts. This value would be the one which kills a 'bad WU', in most cases, so it should be larger than the sporadic error rate by enough to avoid too many false triggers, but not by enough to allow bad WUs to clog the database.

If max_error_results and max_success_results are set properly, one or other of them will always come into play before max_total_results (your numIssues), so maybe Einstein's value of 20 isn't so far wrong after all. But I think max_total_results should be set >= max_error + max_success, else it will kick in prematurely at, say, one success plus four errors, or two inconclusive plus three errors.

I have of course stopped submitting WUs. My cases are intermixed
with regular production and while I could identify mw WUs I don't
know how to delete them other than one at a time (or by range) :-(.
On the other hand as planned I am getting valuable information
back about Validated but wrong results as well as more info about the
empty results files. Tomorrow I will try and identify my WUs and
perhaps find a finite number of ranges to delete.....

With the help of my colleague Alex we are very close to identifying
the gcc problem. Eric.

---

Well, first beams circulated today!

We currently have some 300,000 WUs queued. (Obviously my CERN customers
are not all on vacation, or at least they took their laptop!)

There are about 300,000 WUs queued of which about 53,000 maximum are mine.
I already got back about 30,000 reults and I have in addition about
1087 wrong but Validated results. Many of these will be the *null"
file problem results but I have already identified some real new
issues. Should have fun analysing all this. Eric.

---

New tasks?
Shed some light on us, we are starving

New tasks?
Shed some light on us, we are starving

How can you be "starving" when you participate in 21 Projects. I'm doing work for 4 and I always have enough work to keep busy. Granted, it's not always LHC Tasks but I just keep crunching away.

LHC Tasks tend to be feast or famine. It's the nature of the work they are doing for the LHC.

New tasks?
Shed some light on us, we are starving

How can you be "starving" when you participate in 21 Projects. I'm doing work for 4 and I always have enough work to keep busy. Granted, it's not always LHC Tasks but I just keep crunching away.

LHC Tasks tend to be feast or famine. It's the nature of the work they are doing for the LHC.

all boinc projects are ranged for me and LHC is one of top ones.

je n'arrive pas à avoir de travail depuis environ le 26 juin, mais je vois que le nombre de Wus est très petit ces jour-ci. on veut du travail, on veut du travail!
i can't get work these days, and specialy from the 26th of june. i can se than the amount of wus is low those days. we want work, we want work !

greatings to everybody

---

Geez, folks. Get off Eric's back. It isn't like he sits up all night with a hammer and chisel making tiny rocks out of big ones. I remember not too many years ago before the LHC came on-line when there were no work units at all for many months at a stretch. WU will come when they come. Surely you all are working on other projects to pass the time.

---

HERE! HERE!

Geez, folks. Get off Eric's back. It isn't like he sits up all night with a hammer and chisel making tiny rocks out of big ones. I remember not too many years ago before the LHC came on-line when there were no work units at all for many months at a stretch. WU will come when they come. Surely you all are working on other projects to pass the time.