Message boards : Sixtrack Application : Transfer issues
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · Next

AuthorMessage
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1006
Credit: 47,311,807
RAC: 3,004
Message 33683 - Posted: 6 Jan 2018, 10:16:44 UTC - in response to Message 33681.  

Yes I have 32 cores running these tasks and there is always one of the finished longer tasks stuck on the upload (Transfers) page.
And now the same thing for several of those short running *workplace* tasks.

1/6/2018 2:12:23 AM | | Project communication failed: attempting access to reference site
1/6/2018 2:12:27 AM | | Internet access OK - project servers may be temporarily down.


After 2am so.....goodnight
Volunteer Mad Scientist For Life
ID: 33683 · Report as offensive     Reply Quote
Dr Who Fan
Avatar

Send message
Joined: 26 Nov 16
Posts: 5
Credit: 164,220
RAC: 0
Message 33694 - Posted: 7 Jan 2018, 2:20:35 UTC

Have a SixTrack task that thas been trying to download for about 90 minutes now with failures:

06-Jan-2018 20:07:08 [LHC@home] [http] HTTP_OP::init_get(): http://lhcathome-upload.cern.ch/lhcathome/download/2dd/workspace1_hl13_collision_scan_62.3225_60.3275_chrom_15_oct_-300_B1__35__s__62.31_60.32__10_12__5__30_1_sixvf_boinc3051.zip
06-Jan-2018 20:07:08 [LHC@home] [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\BOINC\ca-bundle.crt'
06-Jan-2018 20:07:08 [LHC@home] [http] HTTP_OP::libcurl_exec(): ca-bundle set
06-Jan-2018 20:07:08 [LHC@home] Started download of workspace1_hl13_collision_scan_62.3225_60.3275_chrom_15_oct_-300_B1__35__s__62.31_60.32__10_12__5__30_1_sixvf_boinc3051.zip
06-Jan-2018 20:07:09 [LHC@home] [http] [ID#10685] Info: Connection 1588 seems to be dead!
06-Jan-2018 20:07:09 [LHC@home] [http] [ID#10685] Info: Closing connection 1588
06-Jan-2018 20:07:09 [LHC@home] [http] [ID#10685] Info: TLSv1.2 (OUT), TLS alert, Client hello (1):
06-Jan-2018 20:07:37 [LHC@home] [http] [ID#10685] Info: Trying 137.138.156.50...
06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Info: Connected to lhcathome-upload.cern.ch (137.138.156.50) port 80 (#1590)
06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: GET /lhcathome/download/2dd/workspace1_hl13_collision_scan_62.3225_60.3275_chrom_15_oct_-300_B1__35__s__62.31_60.32__10_12__5__30_1_sixvf_boinc3051.zip HTTP/1.1

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: Host: lhcathome-upload.cern.ch

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: User-Agent: BOINC client (windows_intelx86 7.8.3)

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: Accept: */*

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: Accept-Encoding: deflate, gzip

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: Content-Type: application/x-www-form-urlencoded

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: Accept-Language: en_US

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server:

06-Jan-2018 20:07:38 [LHC@home] [http] [ID#10685] Sent header to server: án@ºm']÷[-³ZÇQ_«¤2¦a‡¶n†Ò0—ø×iW¡9]jdyÆY¬<1J8|ÒÒK(è _;¢ÌMsÛó®OÇVÕZ§–‰úó«hÔ#†Y'Ï '¼¬nrƒ0rßà¢éÒátu½*ž{T×C9­U(Åâ»ôÀä®8I3Ìv…Ÿ9EÒ¤žòŒQø|ä-õ¬_럱-Ѻ̑BwL%ɐ8oÛðÌûŽ—Y>Õ`Næ(íIyKºHÛ/ùrÓ9ÊþØ4rõ´@Ï1ÃìÞ-]¸PÑ^§iÞ3(ÊP•ù§TËT†PE©ùI
06-Jan-2018 20:08:38 [LHC@home] [http] [ID#10685] Info: Operation too slow. Less than 10 bytes/sec transferred the last 60 seconds
06-Jan-2018 20:08:38 [LHC@home] [http] [ID#10685] Info: Closing connection 1590
06-Jan-2018 20:08:38 [LHC@home] [http] HTTP error: Timeout was reached
06-Jan-2018 20:08:38 [LHC@home] Temporarily failed download of workspace1_hl13_collision_scan_62.3225_60.3275_chrom_15_oct_-300_B1__35__s__62.31_60.32__10_12__5__30_1_sixvf_boinc3051.zip: transient HTTP error
06-Jan-2018 20:08:38 [LHC@home] Backing off 02:13:10 on download of workspace1_hl13_collision_scan_62.3225_60.3275_chrom_15_oct_-300_B1__35__s__62.31_60.32__10_12__5__30_1_sixvf_boinc3051.zip


ID: 33694 · Report as offensive     Reply Quote
AuxRx

Send message
Joined: 16 Sep 17
Posts: 100
Credit: 1,618,469
RAC: 0
Message 33701 - Posted: 7 Jan 2018, 12:42:42 UTC

Can't request work: too many uploads in progress.

And I'm out of tasks.
ID: 33701 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33702 - Posted: 7 Jan 2018, 13:00:54 UTC

Well, in fact, almost all of LHC has been pretty much out of order for about 3 weeks.

It all started out when they swamped the server with an extremely high number of ATLAS tasks - and everything got stuck.
Then they swamped the server with over one million of Sixtrack tasks - again, everything got stuck.
CMS has no tasks since 2 days ago, only Theory and LHCb tasks are available, from which so and so many fail due to connection problems with the "Condor" server :-(

Let's hope that from tomorrow on, when the CERN people are back from vacation, things will improve.
ID: 33702 · Report as offensive     Reply Quote
LHC2018
Avatar

Send message
Joined: 3 Jan 18
Posts: 1
Credit: 53,984
RAC: 0
Message 33704 - Posted: 7 Jan 2018, 14:03:27 UTC

Just got it:
"07.01.2018 14:52:49 | LHC@home | [error] Error reported by file upload server: Server is out of disk space"
ID: 33704 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33705 - Posted: 7 Jan 2018, 14:43:53 UTC - in response to Message 33704.  

Just got it:
"07.01.2018 14:52:49 | LHC@home | [error] Error reported by file upload server: Server is out of disk space"
Well, the indication "Server out of disk space" may or may not be true.
I remember when all these problems at LHC started mit-December, we've seen all kinds of error messages, which were partly wrong. It seems that BOINC at times is messed up by what's going on at LHC :-)
ID: 33705 · Report as offensive     Reply Quote
Dr Who Fan
Avatar

Send message
Joined: 26 Nov 16
Posts: 5
Credit: 164,220
RAC: 0
Message 33706 - Posted: 7 Jan 2018, 14:51:52 UTC

My stuck SixTrack task download finally finished downloading about 10 minutes ago after another forced retry.

ID: 33706 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33707 - Posted: 7 Jan 2018, 15:00:50 UTC - in response to Message 33706.  

My stuck SixTrack task download finally finished downloading about 10 minutes ago after another forced retry.
here, an ATLAS task which got finished 2 days ago is still waiting for upload, despite of numerous forced retries :-(
ID: 33707 · Report as offensive     Reply Quote
AuxRx

Send message
Joined: 16 Sep 17
Posts: 100
Credit: 1,618,469
RAC: 0
Message 33711 - Posted: 8 Jan 2018, 9:06:41 UTC - in response to Message 33701.  
Last modified: 8 Jan 2018, 9:09:20 UTC

I was finally able to get new work by spamming the upload button until enough -- still not all -- tasks had uploaded.

I should add that a lot of tasks were uploaded over night. Also I now find that downloads don't finish either.
ID: 33711 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33712 - Posted: 8 Jan 2018, 9:17:37 UTC - in response to Message 33711.  

Also I now find that downloads don't finish either.
Yesterday, I got 2 ATLAS tasks which took some 15 minutes to download; however, the download was incomplete, and so the tasks failed a few minutes after they started.
Once more I am saying that we can only hope that if the CERN people are back from vacation now, they'll try to straighten out all that mess.
ID: 33712 · Report as offensive     Reply Quote
Alessio Mereghetti
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 29 Feb 16
Posts: 157
Credit: 2,659,975
RAC: 0
Message 33716 - Posted: 8 Jan 2018, 10:29:40 UTC - in response to Message 33712.  
Last modified: 8 Jan 2018, 10:29:50 UTC

Thanks to the IT infrastructure guys, the situation is getting better - please see:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4567#33713
ID: 33716 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33719 - Posted: 8 Jan 2018, 11:01:07 UTC - in response to Message 33716.  

... the situation is getting better - please see:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4567#33713
well, they are saying that they will try to fix some issues. Which may, of course, take time. So far I did not notice any improvement yet.
ID: 33719 · Report as offensive     Reply Quote
T.J.

Send message
Joined: 17 Feb 07
Posts: 86
Credit: 968,855
RAC: 0
Message 33722 - Posted: 8 Jan 2018, 13:01:53 UTC

Indeed my task that is already 15 days trying to upload is still not working and I did many manual tries too.
Another one is for about a week trying to upload, so we still have to wait and see.
Greetings from,
TJ
ID: 33722 · Report as offensive     Reply Quote
pawg

Send message
Joined: 15 Apr 11
Posts: 1
Credit: 110,456
RAC: 24
Message 33723 - Posted: 8 Jan 2018, 15:07:47 UTC

I've uploaded all computed tasks but I can not download any

2018-01-08 15:39:59 | LHC@home | Sending scheduler request: To fetch work.
2018-01-08 15:39:59 | LHC@home | Requesting new tasks for CPU
2018-01-08 15:40:01 | LHC@home | Scheduler request completed: got 0 new tasks
2018-01-08 15:40:01 | LHC@home | No tasks sent
2018-01-08 15:40:01 | LHC@home | No tasks are available for SixTrack
2018-01-08 15:40:01 | LHC@home | Wiadomość z serwera: VirtualBox is not installed
ID: 33723 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 702
Credit: 4,241,529
RAC: 281
Message 33724 - Posted: 8 Jan 2018, 15:15:27 UTC
Last modified: 8 Jan 2018, 15:16:47 UTC

I am trying to upload an Atlas task already past its deadline on a Linux laptop and 8 SixTrack tasks on my main Linux box. Strangely enough, all SixTrack tasks on the laptop uploaded and were reported. Atlas tasks and SixTrack tasks are the only one not failing on my 3 PCs. The third and newest is a Windows 10 PC which has survived, on its A10-6700 CPU and 22 GB RAM the Windows Meltdown/Spectre patch.
Tullio
ID: 33724 · Report as offensive     Reply Quote
Alessio Mereghetti
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 29 Feb 16
Posts: 157
Credit: 2,659,975
RAC: 0
Message 33738 - Posted: 9 Jan 2018, 6:50:12 UTC - in response to Message 33724.  

ID: 33738 · Report as offensive     Reply Quote
PFLIEGER Guy

Send message
Joined: 22 Jun 17
Posts: 12
Credit: 272,799
RAC: 0
Message 33919 - Posted: 19 Jan 2018, 5:27:15 UTC

Impossible to send back sixtrack tasks:
19/01/2018 06:18:53 | LHC@home | Started upload of w-c2_job.B1inj_c2.2158__23__s__64.28_59.31__11.1_12.1__6__24_1_sixvf_boinc13712_0_r471927011_0
19/01/2018 06:18:53 | LHC@home | Started upload of w-c2_job.B1inj_c2.2158__28__s__64.28_59.31__7.1_8.1__6__57_1_sixvf_boinc18808_1_r1137666119_0
19/01/2018 06:19:03 | LHC@home | [error] Error reported by file upload server: [w-c2_job.B1inj_c2.2158__23__s__64.28_59.31__11.1_12.1__6__24_1_sixvf_boinc13712_0_r471927011_0] locked by file_upload_handler PID=-1
19/01/2018 06:19:03 | LHC@home | [error] Error reported by file upload server: [w-c2_job.B1inj_c2.2158__28__s__64.28_59.31__7.1_8.1__6__57_1_sixvf_boinc18808_1_r1137666119_0] locked by file_upload_handler PID=-1
19/01/2018 06:19:03 | LHC@home | Temporarily failed upload of w-c2_job.B1inj_c2.2158__23__s__64.28_59.31__11.1_12.1__6__24_1_sixvf_boinc13712_0_r471927011_0: transient upload error
19/01/2018 06:19:03 | LHC@home | Backing off 04:32:29 on upload of w-c2_job.B1inj_c2.2158__23__s__64.28_59.31__11.1_12.1__6__24_1_sixvf_boinc13712_0_r471927011_0
19/01/2018 06:19:03 | LHC@home | Temporarily failed upload of w-c2_job.B1inj_c2.2158__28__s__64.28_59.31__7.1_8.1__6__57_1_sixvf_boinc18808_1_r1137666119_0: transient upload error
19/01/2018 06:19:03 | LHC@home | Backing off 04:43:40 on upload of w-c2_job.B1inj_c2.2158__28__s__64.28_59.31__7.1_8.1__6__57_1_sixvf_boinc18808_1_r1137666119_0
ID: 33919 · Report as offensive     Reply Quote
Alessio Mereghetti
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 29 Feb 16
Posts: 157
Credit: 2,659,975
RAC: 0
Message 33923 - Posted: 19 Jan 2018, 7:27:52 UTC - in response to Message 33919.  

Is it a task that your BOINC client has been trying to upload in the past days (and most probably got interrupted the first time) or a recent one?
My pc successfully updated a recent result this morning:
Fri 19 Jan 2018 06:29:21 AM CET | LHC@home | Started upload of LHC_2015_LHC_2015_234_BOINC_errors__33__s__62.31_60.32__7.2_7.3__5__64.5_1_sixvf_boinc116391_0_r100608959_0
Fri 19 Jan 2018 06:29:28 AM CET | LHC@home | Finished upload of LHC_2015_LHC_2015_234_BOINC_errors__33__s__62.31_60.32__7.2_7.3__5__64.5_1_sixvf_boinc116391_0_r100608959_0
Fri 19 Jan 2018 06:29:28 AM CET | LHC@home | Sending scheduler request: To report completed tasks.
ID: 33923 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 1982
Credit: 142,811,576
RAC: 100,315
Message 33924 - Posted: 19 Jan 2018, 7:55:33 UTC - in response to Message 33919.  

... locked by file_upload_handler PID=-1

This looks like the (meanwhile) well known file lock issue.
David Cameron wrote about it in the ATLAS MB.

@PFLIEGER Guy
There's nothing you can do to solve the problem.

@Alessio Mereghetti
If not already done the server admins at CERN may check the service that periodically unlocks those files and perhaps run it more often.
ID: 33924 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1511
Credit: 43,739,550
RAC: 42,961
Message 33926 - Posted: 19 Jan 2018, 8:06:01 UTC - in response to Message 33924.  

... and perhaps run it more often.
this may seen to be necessary
ID: 33926 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · Next

Message boards : Sixtrack Application : Transfer issues


©2022 CERN