Message boards : News : Many queued tasks - server status page erratic
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,450,659
RAC: 553
Message 39674 - Posted: 21 Aug 2019, 11:41:17 UTC
Last modified: 30 Aug 2019, 6:45:59 UTC

Due to the very high number of queued Sixtrack tasks, we have enabled 4 load-balanced scheduler/feeder servers to handle the demand. (Our bottleneck is the database, but several schedulers can cache more tasks to be dispatched.)

Our server status page does not currently show in real time the daemon status on remote servers. Hence the server status page may indicate a varying number of processes, depending on which web server is active.

Please also be patient if you are not getting tasks for your preferred application quickly enough. After a few retries, there will be some tasks. Thanks for your understanding and happy crunching!

---the team
ID: 39674 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39682 - Posted: 22 Aug 2019, 5:17:20 UTC

Hi Nils,
there are Atlas Dowonload Error now mostly three at a time for one PC.
Is there a possibility to find a way to reduce this?
https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=122732033
ID: 39682 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,450,659
RAC: 553
Message 39683 - Posted: 22 Aug 2019, 7:40:13 UTC - in response to Message 39682.  

Not sure what causes this. Our 3 file servers are fine and not with much load. It could be a local ISP issue too.
ID: 39683 · Report as offensive     Reply Quote
computezrmle
Avatar

Send message
Joined: 15 Jun 08
Posts: 1140
Credit: 56,122,123
RAC: 96,825
Message 39686 - Posted: 22 Aug 2019, 9:18:09 UTC - in response to Message 39683.  

I doubt that it's an ISP issue.
At least in the past those ATLAS download errors were typically caused by faulty WU batches.
ID: 39686 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39689 - Posted: 22 Aug 2019, 16:06:51 UTC - in response to Message 39683.  

Not sure what causes this. Our 3 file servers are fine and not with much load. It could be a local ISP issue too.

After 5:30 UTC today, no more download Error for Atlas so long.
ID: 39689 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39691 - Posted: 23 Aug 2019, 7:31:02 UTC - in response to Message 39674.  

Nils Høimyr wrote on Aug. 21:
...
Please also be patient if you are not getting tasks for your preferred application quickly enough. After a few retries, there will be some tasks. Thanks for your understanding and happy crunching!
Nils, when will be back to "normal" ?
ID: 39691 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39692 - Posted: 23 Aug 2019, 7:57:11 UTC - in response to Message 39691.  

Erich, be patient and wait after this 3.0 Mio. Tasks are in the past.
You can see this on the server status-page.
ID: 39692 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39710 - Posted: 24 Aug 2019, 5:16:58 UTC - in response to Message 39692.  

Erich, be patient and wait after this 3.0 Mio. Tasks are in the past.
well, we are below 3 Mio tasks now, and the situation has not changed :-(
It is rather laborius to have to manually retry for 20 or 30 minutes until finally new tasks are being downloaded.

I am questioning the rationale behind loading over 3 Mio Sixtrack tasks on the server, if - in turn - this blocks all other sub-projects.
ID: 39710 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39711 - Posted: 24 Aug 2019, 5:27:15 UTC - in response to Message 39710.  
Last modified: 24 Aug 2019, 5:39:15 UTC

We have to wait if ALL tasks (3 Mio.) are finished and
the system went to normally. So be patient....
Scheduler boincai12
feeder boincai12
Download server lhcathome-upload
Upload server lhcathome-upload
This four Server are new to do so many work.
ID: 39711 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39712 - Posted: 24 Aug 2019, 7:23:05 UTC - in response to Message 39711.  

We have to wait if ALL tasks (3 Mio.) are finished
your are not kidding, are you?
ID: 39712 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 330
Credit: 10,777,361
RAC: 16,737
Message 39713 - Posted: 24 Aug 2019, 9:28:38 UTC - in response to Message 39712.  
Last modified: 24 Aug 2019, 10:02:10 UTC

I just attached a machine, and can't get native ATLAS or Theory.
EDIT: But I switched that machine to SixTrack too, and it immediately picked up work. So the moral is simple. If they want SixTrack, do SixTrack.

But another machine is getting a steady supply of SixTrack. So once you are in, you are in.
ID: 39713 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39714 - Posted: 24 Aug 2019, 10:47:05 UTC - in response to Message 39713.  

I am not surprised that it's NOT a problem to download Sixtrack.
But the problem exists with all other sub-projects.
On one of my machines, this morning I first tried about 20 minutes, without success, and then almost 30 minutes, again no success. Just now, I tried some 10 minutes, nothing.
Who has the time to sit there for hours?

A big part of these almost three million Sixtrack tasks should be transferred away from the download server and interimely parked somewhere else.
What sense does it make that non of the other subtasks now cannot be downloaded for quite some time ahead?
ID: 39714 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39715 - Posted: 24 Aug 2019, 12:37:19 UTC - in response to Message 39712.  

We have to wait if ALL tasks (3 Mio.) are finished
your are not kidding, are you?

Erich, you have more than 100k Cobblestones per day!
ID: 39715 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39717 - Posted: 24 Aug 2019, 14:09:44 UTC - in response to Message 39715.  

Erich, you have more than 100k Cobblestones per day!
no, about half is the correct figure.

But what has this to do with this nonsense of swamping the server with 3 Mio Sixtrack tasks, so that, as a consequence, all other subtasks get stuck ?
Can anyone explain that to me?
ID: 39717 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 744
Credit: 27,588,123
RAC: 40,203
Message 39718 - Posted: 24 Aug 2019, 15:58:24 UTC - in response to Message 39717.  

Erich, this stats is your in Boinc-stats for today from LHCatHome:
Cobblestones 143,734 Duration per day 51,976
Let us crunch and do the work they give us.
Cern-IT know how to handle so many tasks.
I have a lot of respect for their work.
ID: 39718 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39720 - Posted: 24 Aug 2019, 16:41:28 UTC - in response to Message 39718.  

Cern-IT know how to handle so many tasks.
in this case: definitely not, sorry to say that :-(
ID: 39720 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 852
Credit: 37,604,900
RAC: 25,040
Message 39721 - Posted: 24 Aug 2019, 21:39:35 UTC

8/24/2019 2:36:39 PM | LHC@home | Not requesting tasks: too many uploads in progress

First time I have ever seen that and I can't send or receive in the last 6 hours
ID: 39721 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 330
Credit: 10,777,361
RAC: 16,737
Message 39722 - Posted: 24 Aug 2019, 21:52:58 UTC - in response to Message 39721.  

8/24/2019 2:36:39 PM | LHC@home | Not requesting tasks: too many uploads in progress

First time I have ever seen that and I can't send or receive in the last 6 hours

If it is the first time that you have seen that, then it is probably the first time in the history of the world.
I am going to give it a break.
ID: 39722 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 852
Credit: 37,604,900
RAC: 25,040
Message 39724 - Posted: 25 Aug 2019, 1:16:24 UTC - in response to Message 39722.  
Last modified: 25 Aug 2019, 1:19:16 UTC

Thanks for that important reply Jim

Btw I finally got all of mine to return to Cern and got all the new ones to d/l
ID: 39724 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1127
Credit: 21,766,747
RAC: 27,729
Message 39814 - Posted: 3 Sep 2019, 18:03:46 UTC - in response to Message 39714.  

I am not surprised that it's NOT a problem to download Sixtrack.
But the problem exists with all other sub-projects.
On one of my machines, this morning I first tried about 20 minutes, without success, and then almost 30 minutes, again no success. Just now, I tried some 10 minutes, nothing.
Who has the time to sit there for hours?

A big part of these almost three million Sixtrack tasks should be transferred away from the download server and interimely parked somewhere else.
What sense does it make that non of the other subtasks now cannot be downloaded for quite some time ahead?
for how long will this bad situation still last?
I have been trying to download ATLAS tasks for more than 1 hour now, sitting here and pushing the "update" button every 5 seconds.
This is more than annoying by now :-(
ID: 39814 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : News : Many queued tasks - server status page erratic


©2019 CERN