LHC@home

during last night, Theory ran out of jobs, and after some time - which was good - the download of new tasks was stopped automatically.

Hope that Ivan can do something this morning :-)

Theory?

during last night, Theory ran out of jobs, and after some time - which was good - the download of new tasks was stopped automatically.

Hope that Ivan can do something this morning :-)

Theory?

oh sorry, it should read "CMS" of course. Unfortunately, I cannot edit my original posting any more.

Yeah, sorry. I submitted a workflow at the weekend that hung in submission; I sent another which worked. Then the "failed" job showed up as "new" so I thought it was OK -- turned out it wasn't. I've submitted another batch, and tried setting the "new" batch to "approved". Hopefully things will be up again in 20-30 minutes.

---

Hi Ivan,

Run out again on the back end./

I assume this was the leading edge of the errors that I had.

I have 11 running now so Ivan probably fixed it.

There's another release of WMAgent waiting to be installed. Since the current batch of jobs will end sometime Saturday, I'll pop in a smaller workflow after that, designed to run out early Monday, to give the WMCore team the chance to get the update done. So, be ready for jobs to start becoming unavailable late Sunday night.

---

There's another release of WMAgent waiting to be installed. Since the current batch of jobs will end sometime Saturday, I'll pop in a smaller workflow after that, designed to run out early Monday, to give the WMCore team the chance to get the update done. So, be ready for jobs to start becoming unavailable late Sunday night.

Last batch is running now. I'd estimated it to end around 0800 UTC, but there's been some disruption in the pipeline (network?) today, so not as many jobs are running/completing as yesterday. This may not necessarily be bad as the main person doing the upgrade has just relocated to Notre Dame and is in a rather later time-zone.

---

Job queue is now exhausted, so you will be seeing more EXIT_NO_SUB_TASKS messages now.

---

It seems there is some problem with the WMAgent update, but I've not received any news about it yet.

---

No worries, I set NNT for CMS, so its working though the 300 or so in the work buffer.

OK, the agent is up again, so new tasks will arrive soon. The first two or three batches will be different from usual, so don't freak out! But, do tell me if they create significant problems.

---

OK, the agent is up again, so new tasks will arrive soon. The first two or three batches will be different from usual, so don't freak out! But, do tell me if they create significant problems.

The first test is over, thank you. Now I try to run jobs twice as large as in the past two years.

---

There are no tasks in the BOINC server's queue:
https://lhcathome.cern.ch/lhcathome/server_status.php

There was a glitch restarting the WMAgent, and we were limited to 100 running jobs. It's fixed now and the number of active jobs is increasing.

---

According to Grafana the subtask queue is empty since 5:12 UTC.

No sub-tasks (jobs) available.

2022-02-04 07:53:15 (9436): Guest Log: [INFO] CMS application starting. Check log files.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] glidein exited with return value 0.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] Shutting Down.

No sub-tasks (jobs) available.

2022-02-04 07:53:15 (9436): Guest Log: [INFO] CMS application starting. Check log files.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] glidein exited with return value 0.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] Shutting Down.

Yes, the latest batch (20,000 events) aren't matching the condor criteria (there are 2,000 jobs in the pending queue). It'll take me an hour or two to get into work and find out why.

---

No sub-tasks (jobs) available.

2022-02-04 07:53:15 (9436): Guest Log: [INFO] CMS application starting. Check log files.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] glidein exited with return value 0.
2022-02-04 08:18:55 (9436): Guest Log: [INFO] Shutting Down.

Yes, the latest batch (20,000 events) aren't matching the condor criteria (there are 2,000 jobs in the pending queue). It'll take me an hour or two to get into work and find out why.

I had to resort to asking my Italian colleague to work this out for me. It appears that there was a time-out set at 15.98 hours (haven't worked out why yet) and the jobs were requesting 16 hours! So, the jobs didn't start.
I've reverted to 10,000 events ("two-hour" jobs) while I ponder the implications of this.

---

Hi Ivan,

I though this was set by Lawrence? the runtime should be 12 h, but there is a timeout at 16 h so they don't run forever if there is an issue?