Message boards : CMS Application : no new WUs available
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 20 · Next

AuthorMessage
Erich56

Send message
Joined: 18 Dec 15
Posts: 1687
Credit: 102,658,063
RAC: 121,448
Message 45331 - Posted: 11 Sep 2021, 18:59:21 UTC

no CMS tasks in the download queue :-(
ID: 45331 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1687
Credit: 102,658,063
RAC: 121,448
Message 45352 - Posted: 19 Sep 2021, 13:56:29 UTC

no CMS tasks in the download queue :-(
ID: 45352 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1687
Credit: 102,658,063
RAC: 121,448
Message 45358 - Posted: 20 Sep 2021, 13:51:19 UTC - in response to Message 45352.  

no CMS tasks in the download queue :-(
anyone any idea when CMS tasks will be available ?
ID: 45358 · Report as offensive     Reply Quote
Harri Liljeroos
Avatar

Send message
Joined: 28 Sep 04
Posts: 674
Credit: 43,496,893
RAC: 16,053
Message 45359 - Posted: 20 Sep 2021, 14:17:02 UTC - in response to Message 45358.  

no CMS tasks in the download queue :-(
anyone any idea when CMS tasks will be available ?

This is the best info we have at the moment: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5720&postid=45351#45351
ID: 45359 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45360 - Posted: 20 Sep 2021, 15:31:45 UTC - in response to Message 45359.  

no CMS tasks in the download queue :-(
anyone any idea when CMS tasks will be available ?

This is the best info we have at the moment: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5720&postid=45351#45351

A colleague reports that our WMAgent is up and running, but has no jobs to serve. The status page for the testbed server doesn't show our agent, nor many others that it usually displays. I've been looking at the CERN IT incident reports, and I don't think any of today's problems would have directly affected us.
ID: 45360 · Report as offensive     Reply Quote
Richie_unstable

Send message
Joined: 26 Oct 18
Posts: 90
Credit: 4,188,598
RAC: 0
Message 45372 - Posted: 21 Sep 2021, 9:26:29 UTC

I see tasks are available again. Great
ID: 45372 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45373 - Posted: 21 Sep 2021, 13:31:11 UTC - in response to Message 45372.  

I see tasks are available again. Great

Yes, a problem was eventually found in our WMAgent, but it had been masked by a larger problem in agent status reporting. The cause of that has yet to be ascertained.
ID: 45373 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45421 - Posted: 5 Oct 2021, 8:53:42 UTC

We are running short of jobs at the moment. I've a new batch submitted, but it hasn't been picked up by the queueing system yet.
The number of running jobs has decreased significantly lately, as a bug in the SixTrack submission system was found and fixed, so all our 6T brethren have deserted us for their first love. :-(
ID: 45421 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45422 - Posted: 5 Oct 2021, 9:57:25 UTC - in response to Message 45421.  

We are up again. (Fingers crossed...)
ID: 45422 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45426 - Posted: 12 Oct 2021, 14:24:37 UTC

I'm letting the job queues drain in preparation for a new WMAgent update. Expect jobs to become unavailable some time on Thursday -- hard to estimate exactly when at the moment because we are picking up SixTrack volunteers again at the moment, as they have no new tasks available. Be ready to set No New Tasks when you see the number of running jobs declining.
ID: 45426 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 801
Credit: 649,373,394
RAC: 241,634
Message 45427 - Posted: 12 Oct 2021, 16:53:35 UTC - in response to Message 45426.  

Looks like something else it wrong for me as I only have CMS tasks.

Is there a way in Graphia to see how much work I do as percentage of T3_CH?
ID: 45427 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45430 - Posted: 14 Oct 2021, 20:05:55 UTC - in response to Message 45427.  

Looks like something else it wrong for me as I only have CMS tasks.

Is there a way in Graphia to see how much work I do as percentage of T3_CH?

Not that I'm aware of. Email me for the address of an experimental dashboard that you may be able to bend to your requirements, but at the moment it's focussing on failures rather than overall results.
ID: 45430 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45431 - Posted: 14 Oct 2021, 20:09:20 UTC - in response to Message 45426.  

I'm letting the job queues drain in preparation for a new WMAgent update. Expect jobs to become unavailable some time on Thursday -- hard to estimate exactly when at the moment because we are picking up SixTrack volunteers again at the moment, as they have no new tasks available. Be ready to set No New Tasks when you see the number of running jobs declining.

OK, the job queue is empty, so we are just left with jobs still out in the field. Set NNT to protect your quotas. I'm about to email the responsibles to ask them to do the update as early as possible tomorrow. I'll post here when it's OK to ask for new tasks again.
ID: 45431 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45433 - Posted: 15 Oct 2021, 11:38:10 UTC - in response to Message 45431.  

Right, jobs are queued, our Task Manager has noticed, and jobs are being sent to volunteer machines. Feel free to resume activities.
ID: 45433 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 801
Credit: 649,373,394
RAC: 241,634
Message 45438 - Posted: 15 Oct 2021, 17:19:21 UTC - in response to Message 45430.  

No worries, just wondered if it was possible, bigger fish to fry for you.
ID: 45438 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2090
Credit: 158,490,720
RAC: 128,521
Message 45441 - Posted: 17 Oct 2021, 10:31:27 UTC - in response to Message 45438.  

no CMS tasks in the download queue
ID: 45441 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 801
Credit: 649,373,394
RAC: 241,634
Message 45442 - Posted: 17 Oct 2021, 11:30:10 UTC

I think the WMAgent has run out, so everyone is going though tasks quickly, I see the CPU load dropped off even on running tasks
ID: 45442 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45443 - Posted: 17 Oct 2021, 16:13:37 UTC - in response to Message 45442.  

There have been network problems at CERN over the weekend, and it appears that it has affected cmsweb and cmsweb-testbed (the latter is what we use to manage our agent). WMStats shows that all agents on cmsweb-testbed are down, and I have not been successful at submitting a new workflow. We'll have to ride this out until CERN IT gets things fixed, it seems.
ID: 45443 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1000
Credit: 6,266,171
RAC: 90
Message 45445 - Posted: 17 Oct 2021, 22:32:53 UTC - in response to Message 45443.  

There are some signs that our WMAgent is up again. However, we now need to wait to see if Laurence's BOINC task manager realises that there are jobs available again. If I remember correctly, this could take up to an hour.
ID: 45445 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 801
Credit: 649,373,394
RAC: 241,634
Message 45454 - Posted: 18 Oct 2021, 16:41:58 UTC - in response to Message 45445.  

I burned though the 1000 or so BOINC tasks while it was down, now I only got theroy and ATLAS from the server, I think it was back on the last few.
ID: 45454 · Report as offensive     Reply Quote
Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · 14 . . . 20 · Next

Message boards : CMS Application : no new WUs available


©2024 CERN