Message boards : News : CMS production pause
Message board moderation

To post messages, you must log in.

AuthorMessage
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 483
Credit: 3,378,120
RAC: 2,943
Message 35079 - Posted: 23 Apr 2018, 15:50:52 UTC

We have run into a problem with the CMS project -- the merged result files processed at CERN are failing to be written to central storage. Consequently I have decided not to submit any more jobs until the experts have clarified what the problem is. The CMS jobs queue is about to start draining and I expect it to be empty of volunteer jobs within a few hours (there may still be post-production jobs, but these run at CERN, not on your machines). I suggest you set No New Tasks or transfer to another project until the situation is resolved.
ID: 35079 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 483
Credit: 3,378,120
RAC: 2,943
Message 35104 - Posted: 28 Apr 2018, 17:23:17 UTC - in response to Message 35079.  
Last modified: 29 Apr 2018, 9:32:31 UTC

There is also a problem affecting the CERN VMs that run the post-production jobs, as T3_CH_CMSAtHome. According to the CERN Service Portal these are still not completely resolved.
I had submitted a couple of small job batches to test the failure mentioned above, but since they do not get beyond the Merge step with T3_CH_CMSAtHome hors de combat, it's not worth doing that until it is working again.
ID: 35104 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 483
Credit: 3,378,120
RAC: 2,943
Message 35110 - Posted: 29 Apr 2018, 9:34:24 UTC - in response to Message 35104.  

T3_CH_CMSAtHome is up again, but it appears we still can't access central storage. I've submitted another small batch to check progress.
ID: 35110 · Report as offensive     Reply Quote

Message boards : News : CMS production pause


©2018 CERN