Message boards : Sixtrack Application : "no tasks are available for Sixtrack", over 1 million visible
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 34137 - Posted: 29 Jan 2018, 6:19:46 UTC

Even when I make Sixtrack the only WU option, the feeder won't send my machines any Sixtrack.

Is it because there are no Windows apps in the queue or some other reason?
ID: 34137 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 798
Credit: 644,672,204
RAC: 235,923
Message 34138 - Posted: 29 Jan 2018, 7:10:22 UTC

I see the same, athough one of my computers cached 150 tasks even though I have it set for 0.5day
ID: 34138 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1686
Credit: 100,335,921
RAC: 102,416
Message 34139 - Posted: 29 Jan 2018, 7:20:13 UTC

same thing here.
Yesterday afternoon, and also this morning, I was sitting at the PC for quite a while and manually pushed the "Update" button about every 10 seconds. So, in total, probably several hundred times.
Not even a single task was downloaded, even though more than 1.2 million unsent tasks are shown on the Project Status Page.

No idea why they provide that huge a number of tasks, while in reality the crunchers cannot download any.
Something seems to run very wrong there.
ID: 34139 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 798
Credit: 644,672,204
RAC: 235,923
Message 34140 - Posted: 29 Jan 2018, 7:24:22 UTC

I just got some on one computer, the only think I notice is that the 2 that are stuck have AMD GPU, so maybe the fact the the project is missconfigured for AMD GPU causes a problem?
ID: 34140 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,486,046
RAC: 0
Message 34141 - Posted: 29 Jan 2018, 8:03:11 UTC

I have switched four machines here yesterday and sometimes have to try for hours until they get some tasks, but then all four machines get tasks almost simultaneously. So to me it looks like there is a sufficiently large buffer of tasks that can be sent out immediately, but it isn't refilled fast enough.
ID: 34141 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1686
Credit: 100,335,921
RAC: 102,416
Message 34142 - Posted: 29 Jan 2018, 10:02:22 UTC - in response to Message 34141.  

So to me it looks like there is a sufficiently large buffer of tasks
the buffer, at this moment, is 1.103.859 tasks - as can be seen here:
https://lhcathome.cern.ch/lhcathome/server_status.php
ID: 34142 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1114
Credit: 49,501,728
RAC: 4,157
Message 34143 - Posted: 29 Jan 2018, 14:22:50 UTC - in response to Message 34137.  
Last modified: 29 Jan 2018, 14:26:36 UTC

Even when I make Sixtrack the only WU option, the feeder won't send my machines any Sixtrack.

Is it because there are no Windows apps in the queue or some other reason?


I know it isn't a Windows thing because I got them on my Win-10 last night and more this morning. ( I have a couple Win-7's but they are just running LHC-dev's right now)

I did have my pref's set at unlimited but since I also run other Cern tasks at the same time on my 8-core pc's I would have trouble getting them finished in time so I set them to the limit of 8 and still got 8 tasks on each one (instead of 100's) but I left my Boinc settings at *10 days*

I did notice last week I changed my prefs where it says *Use ATI GPU* to NOT use......just because I know we don't have GPU tasks BUT after that I could NOT get tasks so I reset that back to *Use ATI GPU X*
Volunteer Mad Scientist For Life
ID: 34143 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,486,046
RAC: 0
Message 34144 - Posted: 29 Jan 2018, 14:36:39 UTC - in response to Message 34142.  

That's the number of tasks that are in the database but not yet sent out. Before they can actually be send out, however, the server has to load them into memory. This can easily become a bottleneck.

That said, I note that the situation has improved in the last few hours. I have not seen the "no tasks are available for Sixtrack" message since around 11:30 UTC.
ID: 34144 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,486,046
RAC: 0
Message 34149 - Posted: 29 Jan 2018, 19:08:36 UTC - in response to Message 34144.  

... and since 16:00 UTC, it got worse again.
ID: 34149 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1686
Credit: 100,335,921
RAC: 102,416
Message 34153 - Posted: 29 Jan 2018, 20:22:41 UTC

For the past hour, I have pushed the "Update" button manually, in about 10-seconds-intervals.
Yet no chance to get new tasks.
This is more than annoying, nothing is being done to get this problem solved :-(

I now change to a different project.
ID: 34153 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 34159 - Posted: 30 Jan 2018, 4:35:38 UTC - in response to Message 34143.  
Last modified: 30 Jan 2018, 4:40:01 UTC

Even when I make Sixtrack the only WU option, the feeder won't send my machines any Sixtrack.

Is it because there are no Windows apps in the queue or some other reason?


I did have my pref's set at unlimited ... so I set them to the limit of 8 and still got 8 tasks on each one




When I was playing around trying to get a mix of 3 different WU (ATLAS/ Theory, Sixtrack) the unlimited vs 24 changed the behavior of the work distribution.

I've gotten no Sixtrack on the 3 machines that are set for a mix since my initial post.

Trying to set maximum 24 instead of unlimited and will see if there is a change.

EDIT: Looks like the 24 limit is truncated to 8 now.
That's not enough work to fill the cores.
Owell.
Just focus on Theory and ATLAS and let Sixtrack go until the feeder issue is resolved.
ID: 34159 · Report as offensive     Reply Quote
Alessio Mereghetti
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 29 Feb 16
Posts: 157
Credit: 2,659,975
RAC: 0
Message 34163 - Posted: 30 Jan 2018, 7:47:37 UTC - in response to Message 34159.  
Last modified: 30 Jan 2018, 7:50:59 UTC

Dear all,

apologies for the late reply and the project mis-behavior. The issue seems to be related to the combined presence of short SixTrack tasks and a huge backlog of tasks for SixTrack. The former implies that the DB responds slowly, and this heavily slowdown the feeder processes. The latter implies that the scheduler drains the shared memory quite rapidly.

Back last June, we were in a similar situation, and the feeder and scheduler parameters were re-tuned, increasing the scheduler shared memory buffer and feeder query size; the SixTrack priority was increased as well, to avoid that the scheduler buffer runs out of SixTrack tasks. The feeder and scheduler parameters cannot be tune-up further, as they are already at the limit of the server capabilities. On the other hand, we will warn the users to be more cautious in submitting new work.

Hence, this is a transitory situation, so please be patient - the regular distribution of work should re-start soon.

Thanks,
A.
ID: 34163 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,486,046
RAC: 0
Message 34169 - Posted: 30 Jan 2018, 13:27:58 UTC - in response to Message 34163.  

Back last June, we were in a similar situation, and the feeder and scheduler parameters were re-tuned, increasing the scheduler shared memory buffer and feeder query size; the SixTrack priority was increased as well, to avoid that the scheduler buffer runs out of SixTrack tasks. The feeder and scheduler parameters cannot be tune-up further, as they are already at the limit of the server capabilities. On the other hand, we will warn the users to be more cautious in submitting new work.

I'm no expert in dealing with a BOINC server in general and your infrastructure in particular, but depending on which quantities (number of tasks sent out per time interval, fraction of successful work requests by clients, etc.) were optimized by tuning the feeder and scheduler parameters, there might still be room for further improvement. For example, a smaller max_wus_to_send (i.e., the maximum number of tasks sent to a client per scheduler request) could lead to less volunteers receiving the annoying "no tasks are available" message, even if the total number of tasks being sent out per time interval is already as high as possible.
ID: 34169 · Report as offensive     Reply Quote
USTL-FIL (Lille Fr)

Send message
Joined: 11 Dec 09
Posts: 27
Credit: 236,744,737
RAC: 70
Message 34190 - Posted: 31 Jan 2018, 15:21:53 UTC

Same problem here, can't get new tasks...
ID: 34190 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 34192 - Posted: 31 Jan 2018, 16:30:51 UTC - in response to Message 34169.  

Back last June, we were in a similar situation, and the feeder and scheduler parameters were re-tuned, increasing the scheduler shared memory buffer and feeder query size; the SixTrack priority was increased as well, to avoid that the scheduler buffer runs out of SixTrack tasks. The feeder and scheduler parameters cannot be tune-up further, as they are already at the limit of the server capabilities. On the other hand, we will warn the users to be more cautious in submitting new work.

For example, a smaller max_wus_to_send



With that in mind I tried setting up 'school' preferences asking for 1 core, 1 task, Sixtrack only and have tried manually updating for an hour on my old Dell laptop and can't secure even a single WU.

Was hoping that a new Data directory for Sixtrack only requests could solve the issue but this experiment would had to work,
ID: 34192 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,486,046
RAC: 0
Message 34195 - Posted: 31 Jan 2018, 18:33:52 UTC - in response to Message 34192.  

For example, a smaller max_wus_to_send


With that in mind I tried setting up 'school' preferences asking for 1 core, 1 task, Sixtrack only and have tried manually updating for an hour on my old Dell laptop and can't secure even a single WU.

Was hoping that a new Data directory for Sixtrack only requests could solve the issue but this experiment would had to work,

While you might increase everyone else's probability to get tasks if you request only one single task, that's not quite what I was suggesting. However, since work was flowing nicely in the morning but I haven't received a single task after 14:00 UTC, it is unlikely that measures to spread the tasks across more machines would help in this situation at all.

With only two tasks remaining, my machines ran dry a bit earlier than wanted, but I was going to switch to another project tonight anyway.
ID: 34195 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 34204 - Posted: 1 Feb 2018, 8:09:26 UTC - in response to Message 34195.  
Last modified: 1 Feb 2018, 9:06:53 UTC



With only two tasks remaining, my machines ran dry a bit earlier than wanted, but I was going to switch to another project tonight anyway.


My one machine sat with 12 idle cores for 14 hours while the others did Theory.
Trying to shape bandwidth usage with QoS (no success) or hard bandwidth limits on each machines' NIC to allow some free bandwidth for private usage, and give up on Sixtrack.

Almost switched to another project tonight but those projects' science just aren't as important to me.

EDIT: A bunch of Sixtrack came down to all my machines 30-50 minutes ago.
Here's hoping it holds.
ID: 34204 · Report as offensive     Reply Quote
Alessio Mereghetti
Volunteer moderator
Project administrator
Project developer
Project tester
Project scientist

Send message
Joined: 29 Feb 16
Posts: 157
Credit: 2,659,975
RAC: 0
Message 34209 - Posted: 1 Feb 2018, 9:45:34 UTC - in response to Message 34169.  


I'm no expert in dealing with a BOINC server in general and your infrastructure in particular, but depending on which quantities (number of tasks sent out per time interval, fraction of successful work requests by clients, etc.) were optimized by tuning the feeder and scheduler parameters, there might still be room for further improvement. For example, a smaller max_wus_to_send (i.e., the maximum number of tasks sent to a client per scheduler request) could lead to less volunteers receiving the annoying "no tasks are available" message, even if the total number of tasks being sent out per time interval is already as high as possible.


Thanks a lot for the suggestion - we are evaluating it.

At the same time, I suggest also to allow for other projects under the LHC@Home umbrella which do not need VMs - eg ATLAS sends native Linux application, for instance. As a volunteer, this allows me not to waist (much) CPU time.
ID: 34209 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 34212 - Posted: 1 Feb 2018, 13:59:00 UTC - in response to Message 34209.  



At the same time, I suggest also to allow for other projects under the LHC@Home umbrella which do not need VMs - eg ATLAS sends native Linux application, for instance. As a volunteer, this allows me not to waist (much) CPU time.


Sixtrack is the only project that sends native to Windows machines, right?
The only WU I've never tried is Alice.
ID: 34212 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 0
Message 34214 - Posted: 1 Feb 2018, 15:09:19 UTC - in response to Message 34212.  
Last modified: 1 Feb 2018, 15:13:01 UTC

Sixtrack is the only project that sends native to Windows machines, right?

You are right that on Windows, there is no "native" version except SixTrack, to the extent that it is "native".

For anyone interested, there is native ATLAS, but on Linux machines only (it works on Red Hat and Ubuntu and variations thereof; I don't know about the others).
But it requires the installation of "CVMFS" and "singularity". Here are the procedures I use:
http://cvmfs.readthedocs.io/en/2.4/cpt-quickstart.html (using the Ubuntu directions; also you can omit the PROXY part if you don't use one, as on a home network).
http://singularity.lbl.gov/install-linux (there is a slight error in the instructions; it is not "VERSION=2.4.2", but "$VERSION=2.4.2".
ID: 34214 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Sixtrack Application : "no tasks are available for Sixtrack", over 1 million visible


©2024 CERN