Message boards : CMS Application : no new WUs available
Message board moderation

To post messages, you must log in.

Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 20 · Next

AuthorMessage
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 39298 - Posted: 6 Jul 2019, 9:45:46 UTC - in response to Message 39297.  

Same for me. I thought it was my new i7-9700, and glad to see it wasn't.
ID: 39298 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1688
Credit: 103,719,551
RAC: 122,079
Message 39302 - Posted: 6 Jul 2019, 12:10:44 UTC

wasn't there, some time last year, introduced an automated stop for new tasks in the download queue once there are no sub-tasks available?
I'm not sure though whether this was done for Theory only.
ID: 39302 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39305 - Posted: 6 Jul 2019, 16:36:15 UTC - in response to Message 39297.  

Since this morning all CMS tasks fail with EXIT_NO_SUB_TASKS

Sorry, I must have forgotten to check the queues last night. We were running low on jobs (tho' WMStats still reported some pending, but low numbers actually running) when I checked this morning. I sent in a new batch, which has been picked up.
I think the automatic task suspension depends on the condor server having no jobs available; there is always going to be a period before it actually kicks in. Fortunately, I seem to have caught it before too much damage was done.
We have a meeting on Wednesday to discuss central CMS operations taking over job submissions. That may take some weight off my shoulders...
ID: 39305 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39318 - Posted: 9 Jul 2019, 13:47:32 UTC

We need to run down the job queues to do a WMAgent upgrade. Current situation appears to be that we'll start draining tomorrow (Wednesday), late afternoon European time. Unfortunately we don't have any graphs to look at at the moment, due to the old Dashboard being decommissioned. I'm trying to find out how to get equivalent graphs from the new system, but set your machines to No New Tasks in 24 hours or so -- or earlier if you start to see task failures.
ID: 39318 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39319 - Posted: 9 Jul 2019, 15:11:34 UTC - in response to Message 39318.  

We need to run down the job queues to do a WMAgent upgrade. Current situation appears to be that we'll start draining tomorrow (Wednesday), late afternoon European time. Unfortunately we don't have any graphs to look at at the moment, due to the old Dashboard being decommissioned. I'm trying to find out how to get equivalent graphs from the new system, but set your machines to No New Tasks in 24 hours or so -- or earlier if you start to see task failures.

Ah, with a little help from my friends, I have found most of what I was looking for. However, I don't know yet if one needs CMS credentials to access it. Could someone please try to access this page and let me know if it's visible? Thanks.
ID: 39319 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1280
Credit: 8,487,633
RAC: 1,762
Message 39322 - Posted: 9 Jul 2019, 17:10:37 UTC - in response to Message 39319.  
Last modified: 9 Jul 2019, 17:13:11 UTC

Could someone please try to access this page and let me know if it's visible? Thanks.

I've access with my Google account. See 308 Running jobs and 38233 done last week, but can't figure how many are still in the queue to be done, or is that the single 1 Pending?
ID: 39322 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39326 - Posted: 9 Jul 2019, 20:25:19 UTC - in response to Message 39320.  
Last modified: 9 Jul 2019, 20:26:33 UTC

I reach a login page for Grafana email and pass or login with CERN SSO

OK, thanks. I was afraid of that. I never did work out why some pages of the old dashboard required credentials and others didn't. I'll have to ask around some more.
[Edit] Oops, forgot to click send a few hours ago... [/Edit]
ID: 39326 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39327 - Posted: 9 Jul 2019, 20:41:40 UTC - in response to Message 39322.  

Could someone please try to access this page and let me know if it's visible? Thanks.

I've access with my Google account. See 308 Running jobs and 38233 done last week, but can't figure how many are still in the queue to be done, or is that the single 1 Pending?

Ah, OK. There may be ways around it yet.
Yes, WMStats is currently showing that 21,691 jobs have been created (a full workflow is currently around 26,000 jobs), 2,580 jobs are "queued", 1,995 are "pending" and 301 are running. I've never been quite sure of the difference between queued and pending; it might be that the queued jobs have been created but not yet sent to the condor server and the pending are those that are on the condor server but not yet running.
It's probably fair to say that grafana's idea of "pending" is different to WMStats. I'm still trying to work out the commonalities and the differences.
One interesting thing is that many jobs appear to be taking less CPU than I'd expected. I've always aimed for ~1 hour/job, but I'll try double the number of events for the next batch; that should increase the efficiency as proportionately less time is spent on initialisation.
ID: 39327 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39328 - Posted: 9 Jul 2019, 20:47:50 UTC
Last modified: 9 Jul 2019, 22:40:37 UTC

Interesting. I found a little dropdown menu on the graphs with a "share" option. Does this link work without CMS credentials?
LINK
[Edit] Maybe not. When I tried it with FireFox rather than Chrome, it wanted a login. :-( [/Edit]
ID: 39328 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1280
Credit: 8,487,633
RAC: 1,762
Message 39331 - Posted: 10 Jul 2019, 8:54:38 UTC - in response to Message 39328.  
Last modified: 10 Jul 2019, 8:56:34 UTC

Interesting. I found a little dropdown menu on the graphs with a "share" option. Does this link work without CMS credentials?
LINK
[Edit] Maybe not. When I tried it with FireFox rather than Chrome, it wanted a login. :-( [/Edit]
With Internet Explorer neither.

With Chrome it works for me, but with a fixed 7-days period, but can be changed to .... from=now-7d&to=now-12m ...
ID: 39331 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39333 - Posted: 10 Jul 2019, 9:14:19 UTC - in response to Message 39331.  

OK, thanks. I'm still exploring but we'll get the kinks ironed out eventually.
ID: 39333 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,295,440
RAC: 131,866
Message 39336 - Posted: 11 Jul 2019, 13:05:57 UTC - in response to Message 39318.  

We need to run down the job queues to do a WMAgent upgrade. Current situation appears to be that we'll start draining tomorrow (Wednesday), late afternoon European time.

Did I miss it or is it still on the schedule?
ID: 39336 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39337 - Posted: 11 Jul 2019, 13:42:37 UTC - in response to Message 39336.  

We need to run down the job queues to do a WMAgent upgrade. Current situation appears to be that we'll start draining tomorrow (Wednesday), late afternoon European time.

Did I miss it or is it still on the schedule?

Yes, I've just given Alan the go-ahead. WMStats show just two jobs pending and 240 running. As soon as I hear that Alan has finished, I'll submit a new batch. Since the new Dashboard shows that average jobs take less CPU time than I thought, I'll increase the job size from 40k to 100k events. This should increase efficiency without compromising bandwidth considerations.
I'm still trying to work out how to make the new graphs public, but I'm officially taking the day off today (Oz are playing Perfidious Albion in the semi-finals of the World Cup cricket...).
ID: 39337 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 39338 - Posted: 11 Jul 2019, 18:43:23 UTC - in response to Message 39337.  
Last modified: 11 Jul 2019, 20:07:16 UTC

I'm still trying to work out how to make the new graphs public, but I'm officially taking the day off today (Oz are playing Perfidious Albion in the semi-finals of the World Cup cricket...).

And we lost. Schade.
Meanwhile, a new batch of jobs has been submitted. As mentioned, these will be 2-1/2 times longer than before the break, which should increase efficiency. I'll be monitoring progress across the weekend but at the moment I might just cry myself to sleep at losing the cricket to England. :-(

[Edit] Just in case it wasn't obvious, jobs are available now and you can set Allow New Tasks in your BOINC manager. [/Edit]
ID: 39338 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1688
Credit: 103,719,551
RAC: 122,079
Message 43487 - Posted: 12 Oct 2020, 4:58:09 UTC

The tasks queue ran dry last night :-(
ID: 43487 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2098
Credit: 159,796,283
RAC: 143,859
Message 43488 - Posted: 12 Oct 2020, 5:29:01 UTC - in response to Message 43487.  

Ivan wrote last week:
There's a WMAgent upgrade ready for installation, so I'll try to get the queues adjusted so that we drain out around Sunday night. So be prepared for a period of no new jobs Monday-ish (set no new tasks, etc., as you wish).
ID: 43488 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1688
Credit: 103,719,551
RAC: 122,079
Message 43489 - Posted: 12 Oct 2020, 5:37:47 UTC - in response to Message 43488.  

oh, I didn't read that.
Thanks a lot for the information :-)
ID: 43489 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2098
Credit: 159,796,283
RAC: 143,859
Message 43490 - Posted: 12 Oct 2020, 6:18:51 UTC - in response to Message 43489.  

btw it was a point landing from the CMS-Team ;-)
ID: 43490 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1688
Credit: 103,719,551
RAC: 122,079
Message 43491 - Posted: 13 Oct 2020, 4:49:51 UTC - in response to Message 43488.  

Ivan wrote last week:
There's a WMAgent upgrade ready for installation, so I'll try to get the queues adjusted so that we drain out around Sunday night. So be prepared for a period of no new jobs Monday-ish (set no new tasks, etc., as you wish).
Ivan, was there a problem with the Agent upgrade yesterday - still no CMS tasks available for download :-(
ID: 43491 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 1006
Credit: 6,272,128
RAC: 397
Message 43492 - Posted: 13 Oct 2020, 9:26:29 UTC - in response to Message 34172.  

good morning, Ivan

BOINC says "not CMS tasks available" :-(

Hi Erich;
The WMAgent upgrade wasn't completed before I went to bed last night. :-(
I submitted a new batch an hour ago and it appears some jobs have started to run (my internet at home is still so slow that I can't run all monitoring applications).
ID: 43492 · Report as offensive     Reply Quote
Previous · 1 . . . 6 · 7 · 8 · 9 · 10 · 11 · 12 . . . 20 · Next

Message boards : CMS Application : no new WUs available


©2024 CERN