LHC@home

mea culpa! I realised today that I'd accidentally typed one zero too many in the WMAgent request for the current batch, and launched ten times too many jobs! Alan tells me this could overload the agent, so I've submitted a "normal" batch and have set this one to "force-complete". This will clear out its queue, but I don't know exactly what effect it will have on currently-running jobs.
So, there may be some jobs report as failed, or otherwise faulty, but once the tasks start picking up jobs from the new batch it should all clear up. My apologies, I hope it's not too traumatic.

---

OK, the new batch has started queueing. There was a hiatus of about 35 minutes with no jobs in the queue.

---

... This will clear out its queue, but I don't know exactly what effect it will have on currently-running jobs.
So, there may be some jobs report as failed, or otherwise faulty

what it did was something like this:

2017-07-03 18:44:59 (6304): Guest Log: [INFO] CMS application starting. Check log files.
2017-07-03 18:45:09 (6304): Guest Log: [DEBUG] HTCondor ping
2017-07-03 18:45:09 (6304): Guest Log: [DEBUG] 0
2017-07-03 18:45:29 (6304): Guest Log: [INFO] New Job Starting in slot1
2017-07-03 18:45:29 (6304): Guest Log: [INFO] Condor JobID: 133232.17 in slot1
2017-07-03 18:46:59 (6304): Guest Log: [INFO] WMAgent_JobID = 62678 in slot1
2017-07-03 20:21:16 (6304): Guest Log: [ERROR] Condor exited after 5769s without running a job.
2017-07-03 20:21:16 (6304): Guest Log: [INFO] Shutting Down.

Just curious, what happens if the CMS scientists have no tasks for us? Does the well run dry?
Thanks!

---

Regards,
Bob P.

Just curious, what happens if the CMS scientists have no tasks for us? Does the well run dry?
Thanks!

Yes, basically. That's why I try to keep the pump primed, and also why I warn you when I know a drought is coming up, so you can set No New Tasks, or transfer to another project. It's perhaps a little more onerous than it might look, I have to check every few hours (except when I'm asleep) to make sure the next batch is available before the old one runs out. Current batches run for about two days, depending on how many people are currently running.

---

[quote] ... I have to check every few hours (except when I'm asleep) to make sure the next batch is available before the old one runs out ...

hm, there is no way to get this automated somehow?

[quote] ... I have to check every few hours (except when I'm asleep) to make sure the next batch is available before the old one runs out ...

hm, there is no way to get this automated somehow?

Possibly. I've never bothered because I didn't expect to be on this project for so long... Keeping an eye on the monitors is only slightly more trouble than keeping an eye on my e-mail, anyway.

---

... because I didn't expect to be on this project for so long...

which in a way is good, though; I think you are doing a perfect Job :-)

There will be an upgrade to the HTCondor schedd this afternoon. I'm told it should make no significant disturbance, but be warned...
[Added] Done, with no problems seen. [/Added]

---

Oops, something has gone wrong now. There is a failure in the WMAgent -- it still says there are jobs available, but other monitors and Dashboard say that they have run out.
Suggest you set No New Tasks until I round up the CERN posse.

---

Ivan, any news on this?

Ivan, any news on this?

Yes, Alan fixed the problem and there are jobs in the queue again . Time to restart.
[Added] Hmm, but the LHC@Home CMS queue is showing no tasks and it's not giving me any. -dev has tasks. I'll ping Laurence, he may need to tickle the server queue. [/Added]

---

Thanks Ivan, as always, for your timely responses :-)))

So I'll see tomorrow morning what's happening, falling to bed right now.

Thanks Ivan, as always, for your timely responses :-)))

So I'll see tomorrow morning what's happening, falling to bed right now.

I've actually got some tasks on one of both my machines now, so the problem may just have been that requests are coming in faster than tasks can be created.[/s]

---

My first check of the day is not what I expected.

About 60 ERRORS

VM Heartbeat file specified, but missing.
VM Heartbeat file specified, but missing file system status. (errno = '2')

[quote] ... I have to check every few hours (except when I'm asleep) to make sure the next batch is available before the old one runs out ...

hm, there is no way to get this automated somehow?

Possibly. I've never bothered because I didn't expect to be on this project for so long... Keeping an eye on the monitors is only slightly more trouble than keeping an eye on my e-mail, anyway.

Well, after last night's problems (which essentially happened between my leaving work and arriving at home :-( ), I have implemented a cron job to check the queue every 30 minutes. It's probably got bugs as I'm not a letter-perfect shell programmer... Since the check relies on my having a valid CMS proxy certificate I've put in a check for it expiring -- at least I could test that bit.

---