LHC@home

Since the last WMAgent update we have had a large number of job failures, as reported on the job graphs and the overall number of tasks running is higher than normal. However, the overall number of failures reported by WMStats is not unusual.
There are, though, a high number of jobs reporting as success with exit code 0 which are re-runs after an initial failure. Most of the jobs failures I see are code 8002 -- fatal exception. I can only look at a small number of these, but they all failed because of an inability to connect to the conditions database (frontier) servers via IPv6:

An exception of category 'StdException' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing ESSource: class=PoolDBESSource label='GlobalTag'
Exception Message:
A std::exception was thrown.
Can not get data (Additional Information: [frontier.c:1159]: No more servers/proxies. Last error was: Request 131 on chan 1 failed at Sun Oct 25 10:00:01 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:3032::681c:94c: Network is unreachable) ( CORAL : "coral::FrontierAccess::Statement::execute" from "CORAL/RelationalPlugins/frontier" )

Now, many of these failures are coming from the same users -- and indeed machines. Their task reports are characterised by only lasting 30-60 minutes and only consuming a few minutes of CPU; nevertheless, they are flagged as valid.
Properly running CMS tasks should run for 12-18 hours and calculate several of our jobs in that time.
My assumption is that the 8002 failures are resubmitted to another machine and properly run the second second time, getting a zero exit code (success)
I managed to contact one of the users who was getting many 8002 errors and he reported that his IPv6 connection had inadvertently been turned off at his router. However his jobs are still running short with little CPU usage, but about 50% of them now report an error while computing.
I'm still not sure exactly where the problem lies, but I would ask that you check your CMS task reports and note if many are running for less than 12 hours, with very low CPU usage. If you find that, please check that your router and firewall are letting through IPv6 requests on the ports listed in our FAQ.
Thanks.

---

Hello Ivan,
saw this task from last night. Stopped at changing from CEST to CET:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=288254714

IPv6 not found so long.

Hello Ivan,
saw this task from last night. Stopped at changing from CEST to CET:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=288254714

IPv6 not found so long.

I got a bunch of these as well. All CMS and Theory tasks that were running failed when Daylight Savings Time ended and computer moved clock one hour back. Failure was 'VM Heartbeat file specified, but missing heartbeat.' Atlas tasks seem to have survived.

---

Interesting. I would have hoped that such checks replied on the UNIX epoch rather than local time.

---

Interesting. I would have hoped that such checks replied on the UNIX epoch rather than local time.

We are hoping this was the last change in the EU from CEST to CET.
No one need it in the future.

We are hoping this was the last change in the EU from CEST to CET.

Unfortunately, European politicians cannot decide what will be the standard time in the future (CET or CEST),

so "Joe Bloggs" and "Fred Bloggs" have to switch their watches for the time being.

Btw: I removed the heartbeat mechanism check from my CMS-job xml-file, so I cannot be surprised anymore by a killed VM.

My Italian colleague has checked some of the condor logs and finds that the vast majority (98%) of jobs with 8002 errors come from the two machines I have already identified, so this is not a general problem, and not really "ours" except that it makes our performance plots look bad.

---

We have had that time change problem here many times over the years.
(Fall back to Pacific Standard Time: November 1, 2020 for me)
(Sunday, October 25, 2020 for the Geneva time change)

CMS queue is dry again.

It looks like computers suffering from those or other problems quickly eat up all available subtasks in the queues.
It also looks like the automatic host banishment implemented in BOINC is too slow to cover that.

The question is whether it is possible to manually exclude the most affected hosts from getting tasks until their volunteers contact the project admins or the forum.

CMS queue is dry again.

It looks like computers suffering from those or other problems quickly eat up all available subtasks in the queues.
It also looks like the automatic host banishment implemented in BOINC is too slow to cover that.

The question is whether it is possible to manually exclude the most affected hosts from getting tasks until their volunteers contact the project admins or the forum.

It shouldn't have been, at least from my monitoring. (I did have a monitoring problem over the weekend, turned out that my browser queue was corrupted -- clearing all history data eventually cleared it.)
I'm just waiting on Federica to analyse the latest condor logs to see if our "rogue" volunteer is still the main problem with jobs being flagged as failures in the graphs. Unfortunately I don't have BOINC admin rights at LHC@Home (I used to when CMS@Home was a separate entity). If we confirm that our Finnish friend is still the main culprit, I can ask Nils to email him directly (he's not answered my private message); failing that, I can then ask for him to be banned. Unfortunately he has a 64-processor computer and somehow manages to get over 150 tasks queued at any one time...

---

To get 150 CMS tasks is easy, all you have to do is enable 'If no work for selected applications is available, accept work from other applications?' and within an hour you get that amount despite of your cache size settings.

---

To get 150 CMS tasks is easy, all you have to do is enable 'If no work for selected applications is available, accept work from other applications?' and within an hour you get that amount despite of your cache size settings.

Ah, thanks; I wasn't aware of that loophole.

---

To get 150 CMS tasks is easy, all you have to do is enable 'If no work for selected applications is available, accept work from other applications?' and within an hour you get that amount despite of your cache size settings.

Ah, thanks; I wasn't aware of that loophole.

I haven't tested all the settings, but every time I enable CMS I get swamped with a lot of tasks.

---

To get 150 CMS tasks is easy, all you have to do is enable 'If no work for selected applications is available, accept work from other applications?' and within an hour you get that amount despite of your cache size settings.

Ah, thanks; I wasn't aware of that loophole.

This is explainable BOINC behaviour.

On the 1st of November you had several short running tasks (700 seconds) after each other, exiting cause of NO_SUB_TASKS.
For BOINC they were OK and BOINC calculates that your machine is able to run that application within a certain amount of time.
When you have a big cache buffer to fill (days) and that application is available again, BOINC is expecting you can do that application in about 700 seconds and needs a lot of tasks to fill your cache.

I thought that tasks that error out are omitted from APR calculation?

---

My 4-core box will readily get swamped by 100+ CMS tasks, to the extent that I will only tick the CMS box when down to the last few CMS tasks if I'm around to watch what happens and untick it in a hurry! This happens even when CMS is running fine - no correlation with job failures. Set to 0.5 days + 1.5 days. "accept work from other applications?" unticked. It might be trying to fill the local queue wrt the CMS deadline, but since letting it do this can force the other sub-projects off the machine I've never left it get that far.

My pet theory is that this is some side-effect of how Atlas and Theory artificially limit their task numbers based on core numbers (such that I can't get 2-day's-worth work between them). That, and the usual chaos one gets from trying to micro-manage the BOINC client (see above) rather than just let it get on with it.

p.s. This discussion should probably be in this thread?

---

My 4-core box will readily get swamped by 100+ CMS tasks, to the extent that I will only tick the CMS box when down to the last few CMS tasks if I'm around to watch what happens and untick it in a hurry! This happens even when CMS is running fine - no correlation with job failures. Set to 0.5 days + 1.5 days. "accept work from other applications?" unticked. It might be trying to fill the local queue wrt the CMS deadline, but since letting it do this can force the other sub-projects off the machine I've never left it get that far.

That just happened to me on a newly-attached machine, with 0.5 + 0.5 days buffer setting.
It ran fine for a few days, and then went bonkers. It stopped after filling the buffer with 7 days of work, and I have set NNW.
https://lhcathome.cern.ch/lhcathome/results.php?hostid=10671473

I have noticed it on various projects (including WCG) ever since the recent versions of BOINC came out. They made a change to the scheduler (always a dangerous thing to do), and no one knows what is happening.

I have noticed it on various projects (including WCG) ever since the recent versions of BOINC came out. They made a change to the scheduler (always a dangerous thing to do), and no one knows what is happening.

The same happened to me on PrimeGrid on 23 Oct. using BOINC version 7.16.11
Quoting myself from PrimeGrid's message board:

I'm responsible for 171 aborted tasks.
In spite of having the cache buffer set to 0.04 + 0.01 additional, tasks kept on flowing to one of my machines. No idea why.
From the about 750 tasks, I crunched the certificate ones (about 375) and aborted 171 from the normal long runners (more aborts to come tomorrow).

My 4-core box will readily get swamped by 100+ CMS tasks, to the extent that I will only tick the CMS box when down to the last few CMS tasks if I'm around to watch what happens and untick it in a hurry! This happens even when CMS is running fine - no correlation with job failures. Set to 0.5 days + 1.5 days. "accept work from other applications?" unticked. It might be trying to fill the local queue wrt the CMS deadline, but since letting it do this can force the other sub-projects off the machine I've never left it get that far.

My pet theory is that this is some side-effect of how Atlas and Theory artificially limit their task numbers based on core numbers (such that I can't get 2-day's-worth work between them). That, and the usual chaos one gets from trying to micro-manage the BOINC client (see above) rather than just let it get on with it.

p.s. This discussion should probably be in this thread?

This is all a bit strange to me. As far as I recall I've never got more CMS tasks than the physical number of CPUs in my machines -- I have one 4-core machine set to work preferences of Max 6 tasks, Max 6 CPUs; it only ever gets 4 tasks. 6-core and 40 core "work" servers get no more than 6 tasks at any one time.
Perhaps something's changed since I stopped running them intensively during lockdown(s).

---

My 4-core box will readily get swamped by 100+ CMS tasks, to the extent that I will only tick the CMS box when down to the last few CMS tasks if I'm around to watch what happens and untick it in a hurry! This happens even when CMS is running fine - no correlation with job failures. Set to 0.5 days + 1.5 days. "accept work from other applications?" unticked. It might be trying to fill the local queue wrt the CMS deadline, but since letting it do this can force the other sub-projects off the machine I've never left it get that far.

My pet theory is that this is some side-effect of how Atlas and Theory artificially limit their task numbers based on core numbers (such that I can't get 2-day's-worth work between them). That, and the usual chaos one gets from trying to micro-manage the BOINC client (see above) rather than just let it get on with it.

p.s. This discussion should probably be in this thread?

This is all a bit strange to me. As far as I recall I've never got more CMS tasks than the physical number of CPUs in my machines -- I have one 4-core machine set to work preferences of Max 6 tasks, Max 6 CPUs; it only ever gets 4 tasks. 6-core and 40 core "work" servers get no more than 6 tasks at any one time.
Perhaps something's changed since I stopped running them intensively during lockdown(s).

Could you try how many you get if you don't limit the number of tasks? Those of us that have more than 8 cores the 8 max tasks is not enough to keep our machines fully loaded. Atlas and Theory limits my tasks to 8+8 even with 'No limit' setting for number of tasks but CMS seems different.

---