LHC@home

The upload problems for ATLAS tasks are back.

Yeah, 15 tasks are stuck uploading :-(

The number of running jobs is over 10,000 again. Almost 11,000, maybe therefore the upload problems again?

A couple of my ATLAS WUs are now trying to upload for nearly 30h.
Is anybody working on a (short term) solution?

The upload problems for ATLAS tasks are back.

Yeah, 15 tasks are stuck uploading :-(

Meanwhile 43 stuck :-((

---

Supporting BOINC, a great concept !

I am getting a 100% transfer of file and then fails. It then tries to upload the entire file again and fails at 100%.... frustrating.

Is it the same "HTTP transient error" that has been seen before? I see there is increased load on our file servers but I'm not sure what is causing it. However some results are getting through ok.

File Upload server error..... file_upload_handler PID=-1

Some WU's are uploaded - one 30 min. ago, but a lot are waiting......

1 for me as well:
3570k

6313 LHC@home 3/6/2018 5:53:12 PM Temporarily failed upload of j70KDm0NUCsnDDn7oo6G73TpABFKDmABFKDmSwKKDmABFKDmohi1Vn_2_r1676589227_ATLAS_result: transient upload error

I have 10 waiting, some from as long as 2 days ago.

Here is the BOINC log:

3/6/2018 6:41:17 PM | LHC@home | Started upload of qzJODmxEjCsnSu7Ccp2YYBZmABFKDmABFKDmyQMKDmABFKDmUGlfOn_2_r1113909317_ATLAS_result
3/6/2018 6:41:17 PM | LHC@home | Started upload of vUaKDmlmrCsnSu7Ccp2YYBZmABFKDmABFKDmfhGKDmABFKDmEH2fzm_0_r963091061_ATLAS_result
3/6/2018 6:41:28 PM | LHC@home | [error] Error reported by file upload server: [vUaKDmlmrCsnSu7Ccp2YYBZmABFKDmABFKDmfhGKDmABFKDmEH2fzm_0_r963091061_ATLAS_result] locked by file_upload_handler PID=-1
3/6/2018 6:41:28 PM | LHC@home | Temporarily failed upload of vUaKDmlmrCsnSu7Ccp2YYBZmABFKDmABFKDmfhGKDmABFKDmEH2fzm_0_r963091061_ATLAS_result: transient upload error
3/6/2018 6:41:28 PM | LHC@home | Backing off 03:18:54 on upload of vUaKDmlmrCsnSu7Ccp2YYBZmABFKDmABFKDmfhGKDmABFKDmEH2fzm_0_r963091061_ATLAS_result
3/6/2018 6:41:29 PM | LHC@home | [error] Error reported by file upload server: [qzJODmxEjCsnSu7Ccp2YYBZmABFKDmABFKDmyQMKDmABFKDmUGlfOn_2_r1113909317_ATLAS_result] locked by file_upload_handler PID=-1
3/6/2018 6:41:29 PM | LHC@home | Temporarily failed upload of qzJODmxEjCsnSu7Ccp2YYBZmABFKDmABFKDmyQMKDmABFKDmUGlfOn_2_r1113909317_ATLAS_result: transient upload error
3/6/2018 6:41:29 PM | LHC@home | Backing off 05:35:19 on upload of qzJODmxEjCsnSu7Ccp2YYBZmABFKDmABFKDmyQMKDmABFKDmUGlfOn_2_r1113909317_ATLAS_result

---

Regards,
Bob P.

Are things better now? The file server load has gone back to normal overnight and the half-uploaded files which cause the "locked by file_upload_handler" messages should have been cleaned.

Since Yesterday morning only ONE Atlas-Task was downloading.
Upload is always waiting. After a manual activating, it fall back to waiting.

Edit: -dev Atlas run normally with the new storage backend!

David Cameron wrote:

Are things better now?

Partly.

David Cameron wrote:

... and the half-uploaded files which cause the "locked by file_upload_handler" messages should have been cleaned.

Unfortunately not all of them:

Mi 07 Mär 2018 09:25:46 CET | LHC@home | Started upload of fkHKDmW16CsnDDn7oo6G73TpABFKDmABFKDmXfLKDmABFKDmcutGYo_0_r629916232_ATLAS_result
Mi 07 Mär 2018 09:25:48 CET | LHC@home | [error] Error reported by file upload server: [fkHKDmW16CsnDDn7oo6G73TpABFKDmABFKDmXfLKDmABFKDmcutGYo_0_r629916232_ATLAS_result] locked by file_upload_handler PID=-1
Mi 07 Mär 2018 09:25:48 CET | LHC@home | Temporarily failed upload of fkHKDmW16CsnDDn7oo6G73TpABFKDmABFKDmXfLKDmABFKDmcutGYo_0_r629916232_ATLAS_result: transient upload error
Mi 07 Mär 2018 09:25:48 CET | LHC@home | Backing off 03:22:08 on upload of fkHKDmW16CsnDDn7oo6G73TpABFKDmABFKDmXfLKDmABFKDmcutGYo_0_r629916232_ATLAS_result

Also affected by the same type of error:

GeoNDmDJOCsnSu7Ccp2YYBZmABFKDmABFKDmvALKDmABFKDmuGxLLo_0_r931142306_ATLAS_result
h4eLDmFDLDsnSu7Ccp2YYBZmABFKDmABFKDm0aIKDmABFKDm5bw40n_0_r12622427_ATLAS_result
sGvNDmXMqCsnDDn7oo6G73TpABFKDmABFKDm3eFKDmABFKDmaYr87m_0_r1115052517_ATLAS_result
79MLDm9DJDsnDDn7oo6G73TpABFKDmABFKDm71HKDmABFKDmmbpShm_2_r233222736_ATLAS_result

It may be necessary to run the cleanup script more frequently.
When do you expect the new storage backend will be activated?

Thanks for taking care.

Are things better now?

I had and have only one in Upload pending status. Did not request more ATLAS tasks until it's fixed, but my retry upload does not function:

LHC@home 07 Mar 12:47:56 [error] Error reported by file upload server: [lFvLDmDP1CsnDDn7oo6G73TpABFKDmABFKDmsUJKDmABFKDmwnH64m_0_r894275464_ATLAS_result] locked by file_upload_handler PID=-1
LHC@home 07 Mar 12:47:56 Temporarily failed upload of lFvLDmDP1CsnDDn7oo6G73TpABFKDmABFKDmsUJKDmABFKDmwnH64m_0_r894275464_ATLAS_result: transient upload error

Yes, better now. I had two tasks stuck for over 24 hours, and this morning when I booted the machine, both of them immediately completed their upload and transferred. Thank you!

2018-03-07 5:40:17 PM | LHC@home | Temporarily failed upload of su3MDmEYCCsnSu7Ccp2YYBZmABFKDmABFKDmnbGKDmABFKDm1owStm_0_r1889291022_ATLAS_result: transient upload error

---

Finally all pending uploads could be finished and reported.

+1 :-)

Another locked:
316316 LHC@home 3/12/2018 6:41:14 AM [error] Error reported by file upload server: [E8dMDmPJSFsnDDn7oo6G73TpABFKDmABFKDmquKKDmABFKDmkyA17m_0_r860623117_ATLAS_result] locked by file_upload_handler PID=-1

same here.

Same here.

My uploads are back to normal.
May have been just a glitch.