LHC@home

LHCb is back too

Great. As far as I know they're not that connected, so let's put it down to coincidence for now.

---

The CMS WMAgent is down

what catches my eye is that this WMAgent has been down quite often in the (recent) past - what is the reason? Misconfiguration?

The CMS WMAgent is down

what catches my eye is that this WMAgent has been down quite often in the (recent) past - what is the reason? Misconfiguration?

I really don't know, it's not my area at all. I do know (I think) that what I call "WMAgent" is a VM running some sort of master server and a number of "components" that each have a job to do -- job creation, job scheduling, accounting, etc. Usually it's one of these components that fail, and sometimes they don't have any direct effect on the job queuing and retrieval (an accounting component failed a while back; jobs still flowed but the graphs disn't update). Often the component can be restarted without affecting the rest. Last night it seemed to be a general failure.
That said, it does seem to happen too often after some part of the system has been upgraded, and that does seem to happen too often on a Friday night, or sometimes at weekends. I'll try to remember to give someone a gentle nudge next week, maybe I can work it into a talk I'm giving to the Collaboration as a whole next Thursday. :-)

---

what catches my eye is that this WMAgent has been down quite often in the (recent) past - what is the reason? Misconfiguration?

In IT you can minimize the errors and interrupts,
but never reach 100%.
This is my experience from more than 30 years.

Cern-IT is doing a great job.

It would be nice if the status of all these servers for VM jobs and the number of available jobs could be visible on the server status page. This of course should be done for all apps, not just CMS.

---

It would be nice if the status of all these servers for VM jobs and the number of available jobs could be visible on the server status page. This of course should be done for all apps, not just CMS.

I fully agree - because what is shown on the server status page now is actually misleading Information :-(

what catches my eye is that this WMAgent has been down quite often in the (recent) past - what is the reason? Misconfiguration?

In IT you can minimize the errors and interrupts,
but never reach 100%.

I fully agree - but yet the WMAgent seems to fail quite often. So I am just curious what's behind this.

Since last night, all tasks failing after 10-12 minutes.
What's the Problem?

Since last night, all tasks failing after 10-12 minutes.
What's the Problem?

No CMS jobs available

No CMS jobs available

any idea when new Jobs will be available again? I guess NOT before next week?

Sorry, just woke up to find this. Can't see what the problem is yet, it doesn't appear to be the WMAgent this time. Could take a while to fix, it was a holiday at CERN on Thursday, so many people turn that into a four-day weekend. (It's a long weekend here in the UK, but the important people are at CERN.)

---

Shouldn't it be time to stop the WU generation to avoid EXIT_INIT_FAILURES?
According to the server status page the #WUs is still high.
According to Laurence's post it should be possible to stop it.

Shouldn't it be time to stop the WU generation to avoid EXIT_INIT_FAILURES?
According to the server status page the #WUs is still high.
According to Laurence's post it should be possible to stop it.

We're running again; we had a backlog of merge jobs that filled the queue.
Message from Laurence, "Next week I will implement the automatic shutdown of tasks as a priority."

---

Two CMS tasks failing after 10 min; each on a different host, once Windows, once Linux:

https://lhcathome.cern.ch/lhcathome/result.php?resultid=144126716
https://lhcathome.cern.ch/lhcathome/result.php?resultid=144113647

Michael.

---

also here: "[ERROR] Condor exited after 609s without running a Job"

complete report can be seen:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=144129

also here: "[ERROR] Condor exited after 609s without running a Job"

complete report can be seen:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=144129

Unfortunately that URL was incomplete; looks like it should have been https://lhcathome.cern.ch/lhcathome/result.php?resultid=144129213 . Things look OK our end, though there is a large oscillation in the number of jobs after an outrage such as this, that we have yet to find an explanation for. It damps down eventually. Other tasks on that machine seem to be getting jobs, so unless it continues to happen I'd suggest regarding it as a one-off glitch.
Looking at your machine details, I'd also suggest that running ten 2-GB VMs on a 32 GB machine might be pushing the limits. Check your memory usage with Task Manager. I'd also strongly suggest a Windows Upgrade if you can; XP is beyond useful life and supported only in exceptional security circumstances (such as the ransomware earlier this month). Unless you've hacked the registry to make Microsoft think it's a PoS terminal...

---

Message from Laurence, "Next week I will implement the automatic shutdown of tasks as a priority."

An automated kill switch has been implemented. The sending of tasks should be stopped when the queue is too low. We will find out if this works the next time there is a problem.

Message from Laurence, "Next week I will implement the automatic shutdown of tasks as a priority."

An automated kill switch has been implemented. The sending of tasks should be stopped when the queue is too low. We will find out if this works the next time there is a problem.

Thanks, Laurence.

---

I've just spotted that our WMAgent has problems again, and the queue is depleting. Best take evasive action if necessary until we see if it gets fixed. There were another two down when I first noticed it, but they are back again so maybe someone is already working on it (my monitors only update every five minutes).

---

I've just spotted that our WMAgent has problems again, and the queue is depleting. Best take evasive action if necessary until we see if it gets fixed. There were another two down when I first noticed it, but they are back again so maybe someone is already working on it (my monitors only update every five minutes).

No change yet:

agent: vocms0159.cern.ch (1.1.2.patch2)
agent last updated: 2017/5/29 (Mon) 19:20:19 UTC : 0 h 0 m
data last updated: N/A
status: Components or Thread down;
team: testbed-vocms0159

---