Message boards : News : Database problems
Message board moderation

To post messages, you must log in.

AuthorMessage
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,437,252
RAC: 742
Message 38804 - Posted: 13 May 2019, 13:01:57 UTC
Last modified: 13 May 2019, 15:09:20 UTC

We are having database problems and have to schedule an intervention at 3:30pm UTC. The LHC@home servers are back again. We may have some irregular dispatching of some applications over the next hours.
ID: 38804 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 840
Credit: 36,847,527
RAC: 34,245
Message 38828 - Posted: 14 May 2019, 6:38:23 UTC - in response to Message 38804.  
Last modified: 14 May 2019, 6:42:50 UTC

17 hours later and still NOTHING

ID: 38828 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,437,252
RAC: 742
Message 38830 - Posted: 14 May 2019, 6:59:09 UTC - in response to Message 38828.  

Sorry Magic, you are wrong. We have added 2 additional scheduler hosts last night that serve Sixtrack tasks while the main host boincai01 does a slow database query for all applications.

While these other 2 servers reply to the alias lhcathome.cern.ch, the server status page shows the main host boincai01 and daemons as not running, although they are and other daemons are running too, but not shown in the page.

The scheduler will from time to time reply: "No tasks available" and only drip-feed ATLAS, CMS and Theory tasks right now, but Sixtrack tasks should be served every 10 minutes or so. This is a workaround while trying to get back to our regular speed of serving tasks. Please note that a change of database server did not help much.
ID: 38830 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 840
Credit: 36,847,527
RAC: 34,245
Message 38832 - Posted: 14 May 2019, 8:39:14 UTC - in response to Message 38830.  

How am I wrong??

I even post a picture of the server status and state the FACT that I get NOTHING from the server that is a VB task.

AND still am not getting new Theory or CMS VB tasks.....that is a fact AND that has nothing to do with Sixtrack

BUT I do get CMS VB at -dev so I guess I could switch the 50 cores over there.
ID: 38832 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,437,252
RAC: 742
Message 38834 - Posted: 14 May 2019, 9:03:52 UTC - in response to Message 38832.  

Correct, right now there are very few VB tasks being shipped. We have just started a 4'th scheduler server that should soon ship VB tasks.

We hope to get back to our usual operation later, meanwhile this parallelised server setup is a workaround to ensure that enough tasks are flowing for Sixtrack. Tasks for other apps will slowly be coming along later.
ID: 38834 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1119
Credit: 20,832,632
RAC: 37,124
Message 38836 - Posted: 14 May 2019, 13:09:48 UTC

Nils, how can it be that the Server Status Page still shows only "running" for the Scheduler, for the rest of the programs it shows "stopped" ???
ID: 38836 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,437,252
RAC: 742
Message 38837 - Posted: 14 May 2019, 14:07:21 UTC - in response to Message 38836.  

Nils, how can it be that the Server Status Page still shows only "running" for the Scheduler, for the rest of the programs it shows "stopped" ???

Because the cache of the server status page shows the information for the server boincai01 and the server showing you the page is another host boincai02, boincai03 etc. Our server configuration does not show the status of remote daemons on other hosts. (We could display a page for each server, but this has other side effects.)

The daemons and services are running on boincai01, and we have 3 other servers running schedulers and feeders. But you will see this page with "not running" entries intermittently for a few days.

Delivery of other applications than Sixtrack will be slow until the backlog of Sixtrack tasks has been reduced.
ID: 38837 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1119
Credit: 20,832,632
RAC: 37,124
Message 38838 - Posted: 14 May 2019, 14:10:52 UTC - in response to Message 38837.  

Nils, thanks for the information.
ID: 38838 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1119
Credit: 20,832,632
RAC: 37,124
Message 38840 - Posted: 14 May 2019, 16:46:13 UTC - in response to Message 38837.  

... until the backlog of Sixtrack tasks has been reduced.
what I have noticed for the past few hours: also delivery of Sixtrack tasks sucks once in a while.
ID: 38840 · Report as offensive     Reply Quote
Profile Nils Høimyr
Volunteer moderator
Project administrator
Project developer
Project tester

Send message
Joined: 15 Jul 05
Posts: 207
Credit: 4,437,252
RAC: 742
Message 38841 - Posted: 14 May 2019, 18:55:51 UTC - in response to Message 38840.  

Yes, and the DB went down around 8pm. Restarted now.
ID: 38841 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 82
Credit: 7,051,940
RAC: 1,850
Message 38842 - Posted: 14 May 2019, 18:59:40 UTC - in response to Message 38841.  

Thank you very much for your work, even at such a late hour!
I hope things go back to normal soon, i want my Atlas native tasks back :-)
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! www.gridcoin.us
ID: 38842 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1119
Credit: 20,832,632
RAC: 37,124
Message 38843 - Posted: 14 May 2019, 19:29:14 UTC

all my finished Sixtracks were uploaded, but stay in status "ready for reporting".
And even for Sixtrack now (like so far for the VM subprojects which had their problems), when pushing the "update" button, it says "Communication deferred 01:00:00".
Something seems to go very wrong there, I'm afraid.
ID: 38843 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1119
Credit: 20,832,632
RAC: 37,124
Message 38847 - Posted: 15 May 2019, 6:54:58 UTC

good morning, the database problem obviously still exists, no VM tasks can be downloaded.
Also, I notice that access to the Webpage sometimes sucks.

To me, there seems to be a larger problem than originally assumed and/or communicated.
Could it be that LHC was victim of a hacker attack?
ID: 38847 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 715
Credit: 26,438,420
RAC: 27,781
Message 38848 - Posted: 15 May 2019, 7:11:42 UTC

Theory(native) get tasks.
ID: 38848 · Report as offensive     Reply Quote
Profile MAGIC Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 840
Credit: 36,847,527
RAC: 34,245
Message 38849 - Posted: 15 May 2019, 8:05:15 UTC - in response to Message 38847.  

I would like to know why even though all of my settings are for Theory tasks the server keeps sending me Sixtrack tasks????

I have had them set that way for over a year and now when I try to get tasks it sends me only ones I do NOT want.

First it sent me CMS and now Sixtrack......and of course I have been doing this a long time so it isn't at my end.

1am and this now......
ID: 38849 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 401
Credit: 84,525,554
RAC: 78,509
Message 38850 - Posted: 15 May 2019, 10:31:08 UTC - in response to Message 38847.  

Could it be that LHC was victim of a hacker attack?

Nope, it is not a hacker attack, it is Pentathlon-Time and LHC is Project for Sprint


Supporting BOINC, a great concept !
ID: 38850 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 82
Credit: 7,051,940
RAC: 1,850
Message 38854 - Posted: 15 May 2019, 13:36:07 UTC - in response to Message 38850.  

Pentathlon hasn't startet yet...but whatever...

I want my Atlas tasks back, but i can not get any :-(
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! www.gridcoin.us
ID: 38854 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 401
Credit: 84,525,554
RAC: 78,509
Message 38855 - Posted: 15 May 2019, 13:44:06 UTC - in response to Message 38854.  

Pentathlon hasn't startet yet...but whatever...

Nope, it HAS already started. With telling the participiants the name of the Sprint-Project, all try to get WUs immediatly ...
I want my Atlas tasks back, but i can not get any :-(
It is possible, I got some


Supporting BOINC, a great concept !
ID: 38855 · Report as offensive     Reply Quote
djoser
Avatar

Send message
Joined: 30 Aug 14
Posts: 82
Credit: 7,051,940
RAC: 1,850
Message 38856 - Posted: 15 May 2019, 14:00:14 UTC - in response to Message 38855.  
Last modified: 15 May 2019, 14:02:17 UTC

My apologies, you are right of course. People are filling their queues with LHC tasks.

You got Atlas tasks? You lucky dog :-)

Since the problem with the database startet i did not get a single Atlas native task. Not a chance :-(
If i would not allow sixtrack tasks too, my machine would sit idle doing nothing at all.

Cheers!
Why mine when you can research? - GRIDCOIN - Real cryptocurrency without wasting hashes! www.gridcoin.us
ID: 38856 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 715
Credit: 26,438,420
RAC: 27,781
Message 38874 - Posted: 17 May 2019, 6:32:42 UTC
Last modified: 17 May 2019, 6:34:33 UTC

The Server Status page shows for Central-Europe Summer Time -4 from UTC.
Normal is -2 UTC!
ID: 38874 · Report as offensive     Reply Quote

Message boards : News : Database problems


©2019 CERN