Message boards : Number crunching : Because you asked....
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 6 · Next

AuthorMessage
Profile Neasan
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 30 Nov 06
Posts: 234
Credit: 11,078
RAC: 0
Message 16357 - Posted: 15 Feb 2007, 16:03:54 UTC

This is our current To Do List:
* Database backup and restore
* Server backup
* Get test system working

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

* Ensure e-mail is working properly
* Test migrate
* Real migration
* Solve any other problems that come up!

Cheers,
Neasan and Alex.
ID: 16357 · Report as offensive     Reply Quote
Profile Saenger

Send message
Joined: 13 Jul 05
Posts: 64
Credit: 501,223
RAC: 0
Message 16359 - Posted: 15 Feb 2007, 20:50:43 UTC - in response to Message 16357.  

This is our current To Do List:
[snip]

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

[snip]


Find???
Does this mean they have been lost on their way from Geneva to Oxford?
Or were they rejected by the english customs office?

Grüße vom Sänger
ID: 16359 · Report as offensive     Reply Quote
Profile Keck_Komputers

Send message
Joined: 1 Sep 04
Posts: 275
Credit: 2,648,112
RAC: 30
Message 16360 - Posted: 16 Feb 2007, 8:55:27 UTC - in response to Message 16359.  

This is our current To Do List:
[snip]

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

[snip]


Find???
Does this mean they have been lost on their way from Geneva to Oxford?
Or were they rejected by the english customs office?

Most/all of those programs had custom versions at this project, the file deleter is probably the only one even close to the standard version. So the source additions must be found and then figured out. And finaly updated to work with the current BOINC standard code.
BOINC WIKI

BOINCing since 2002/12/8
ID: 16360 · Report as offensive     Reply Quote
Profile Neasan
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 30 Nov 06
Posts: 234
Credit: 11,078
RAC: 0
Message 16361 - Posted: 16 Feb 2007, 9:40:33 UTC - in response to Message 16359.  

Oxford?


Where exactly do you think Queen Mary, University of London is?
ID: 16361 · Report as offensive     Reply Quote
Aaron Finney

Send message
Joined: 14 Jul 05
Posts: 60
Credit: 140,661
RAC: 0
Message 16368 - Posted: 18 Feb 2007, 7:43:42 UTC - in response to Message 16361.  


This is our current To Do List:
* Database backup and restore
* Server backup
* Get test system working

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

* Ensure e-mail is working properly
* Test migrate
* Real migration
* Solve any other problems that come up!

Cheers,
Neasan and Alex.


Hmm ok..

So that's a pretty large 'To-Do' list.

What's been done already? Anything?
ID: 16368 · Report as offensive     Reply Quote
River~~

Send message
Joined: 13 Jul 05
Posts: 456
Credit: 75,142
RAC: 0
Message 16369 - Posted: 18 Feb 2007, 8:33:40 UTC

I would suggest that you need to identify and cure the cause of the proliferation of hostids before you do the real migrate, and in addition to prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.

It would be possible to transfer this bug across and then fix it later, but, in my opinion, that would cause you more difficulties in the long run than solving the issue first.

Having a wildly excessive number of hosts in the db certainly skews any optimisations you may try to test, and may well cause other bugs that would not arise with a plausible number of hosts.

One bug which may (or may not) be a knock on from the mega-ghost-host issue is the stats export issue. Stats export is not going to be high up the list of practical priorities, so I don't anticipate you will spend a lot of time on stats export itself.

On the other hand stats come high on the personal priotirites of many crunchers and if a solution to that came easily out your ghostbusting of the mega-hosts then many crunchers would be delighted, and there would be a corresponding saving in winges on these boards...

Best regards,
River~~
ID: 16369 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 16370 - Posted: 18 Feb 2007, 14:26:12 UTC

Out of curiosity: how big was the database dump that you had to transfer to UK? Considering you have 5739543 entries on host table...
ID: 16370 · Report as offensive     Reply Quote
Profile Neasan
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 30 Nov 06
Posts: 234
Credit: 11,078
RAC: 0
Message 16371 - Posted: 19 Feb 2007, 10:18:16 UTC - in response to Message 16368.  


This is our current To Do List:
* Database backup and restore
* Server backup
* Get test system working

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

* Ensure e-mail is working properly
* Test migrate
* Real migration
* Solve any other problems that come up!

Cheers,
Neasan and Alex.


Hmm ok..

So that's a pretty large 'To-Do' list.

What's been done already? Anything?


I will refer you to this post:
http://lhcathome.cern.ch/forum_thread.php?id=2406&nowrap=true#16298
ID: 16371 · Report as offensive     Reply Quote
Jerzy Ziembinski

Send message
Joined: 23 Jul 05
Posts: 1
Credit: 97,491
RAC: 0
Message 16392 - Posted: 22 Feb 2007, 7:43:10 UTC - in response to Message 16371.  

it will be finished in a month of Sundays ;)
pls, transfer the project to Poland, will be similarly but cheaper :D

This is our current To Do List:
* Database backup and restore
* Server backup
* Get test system working

* Find and compile and test:
- Transitioner
- Validator
- Assimilator
- File deleter

* Ensure e-mail is working properly
* Test migrate
* Real migration
* Solve any other problems that come up!

Cheers,
Neasan and Alex.


Hmm ok..

So that's a pretty large 'To-Do' list.

What's been done already? Anything?


I will refer you to this post:
http://lhcathome.cern.ch/forum_thread.php?id=2406&nowrap=true#16298


ID: 16392 · Report as offensive     Reply Quote
Daxa

Send message
Joined: 29 Dec 06
Posts: 100
Credit: 184,937
RAC: 0
Message 16398 - Posted: 24 Feb 2007, 4:05:18 UTC - in response to Message 16369.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


I like that idea. For those hosts that never submitted work but still want to participate, they can manually add "themselves" back to the project post-migration.


_______

"Three quarks for Muster Mark!"
. . . . . . . - James Joyce, Finnegans Wake . . . .

ID: 16398 · Report as offensive     Reply Quote
Profile Neasan
Volunteer moderator
Volunteer tester
Avatar

Send message
Joined: 30 Nov 06
Posts: 234
Credit: 11,078
RAC: 0
Message 16450 - Posted: 2 Mar 2007, 15:34:24 UTC

TWDYSTI* Friday and we have an update (also on front page):

Yes it is going slower than expected. It is a unique installation of BOINC (very unique) but it is moving ahead and we give you our current To Do List:
* Initial Database Migration
* Get test system working with database
* Ensure e-mail is working properly(trickier than it sounds)
* Test migrate again
* Real migration
* Sort out job submission over AFS by SixTrack User

This is going ahead at a pace to make sure that everything is working at every step of the way as opposed to trying to sort it out post a hasty install. So in a few weeks time the problems of yesteryear will seem like a bad dream that never happened and we'll all be happily crunching away**.
Later days,
Neasan and Alex.



*=Thank Whatever Deity You Subscribe To It's
**=Not a guarantee, the value of investments may rise or fall, terms and conditions apply.
ID: 16450 · Report as offensive     Reply Quote
Ravens

Send message
Joined: 14 Dec 06
Posts: 29
Credit: 128,225
RAC: 0
Message 16451 - Posted: 2 Mar 2007, 18:10:56 UTC - in response to Message 16398.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


I like that idea. For those hosts that never submitted work but still want to participate, they can manually add "themselves" back to the project post-migration.



I joined before Christmas, leave my laptop running and ready for any LHC work but haven't received my first WU yet. System regulary checks now, while I run other BOINC projects.
I'd hate to be deleted after patiently waiting and have to wait for some sort of post-migration period to be over before I can re-apply. Maybe there needs to be a list of ID's that are still active because within the last x months either they submitted WU's or they have been polling for WU's.
ID: 16451 · Report as offensive     Reply Quote
River~~

Send message
Joined: 13 Jul 05
Posts: 456
Credit: 75,142
RAC: 0
Message 16452 - Posted: 2 Mar 2007, 22:46:39 UTC - in response to Message 16451.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


I like that idea. For those hosts that never submitted work but still want to participate, they can manually add "themselves" back to the project post-migration.



I joined before Christmas, leave my laptop running and ready for any LHC work but haven't received my first WU yet. System regulary checks now, while I run other BOINC projects.
I'd hate to be deleted after patiently waiting and have to wait for some sort of post-migration period to be over before I can re-apply. Maybe there needs to be a list of ID's that are still active because within the last x months either they submitted WU's or they have been polling for WU's.


Just to clarify:

There are a huge number of ghost hosts in the system, due to a bug that makes / made some hosts forget their identity. The proposal is aimed at the most straightforward way of pruning them out of the database to make it more manageable.

The deletion of your host, if you have not got any work by then would simply meant that next time your tired for work, the host would be re-initialised. This would happen on the very next connection.

The date shown for that computer joining the project would be the date it was given the new identity, but you would still have your original joining date shown on the forum boards nect to every post.

It is not a perfect solution, but in my opinion offers the most practical balance between taking up the admins time and bringing the database back to something manageable.

R~~
ID: 16452 · Report as offensive     Reply Quote
Ravens

Send message
Joined: 14 Dec 06
Posts: 29
Credit: 128,225
RAC: 0
Message 16453 - Posted: 2 Mar 2007, 23:01:40 UTC - in response to Message 16452.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


I like that idea. For those hosts that never submitted work but still want to participate, they can manually add "themselves" back to the project post-migration.



I joined before Christmas, leave my laptop running and ready for any LHC work but haven't received my first WU yet. System regulary checks now, while I run other BOINC projects.
I'd hate to be deleted after patiently waiting and have to wait for some sort of post-migration period to be over before I can re-apply. Maybe there needs to be a list of ID's that are still active because within the last x months either they submitted WU's or they have been polling for WU's.


Just to clarify:

There are a huge number of ghost hosts in the system, due to a bug that makes / made some hosts forget their identity. The proposal is aimed at the most straightforward way of pruning them out of the database to make it more manageable.

The deletion of your host, if you have not got any work by then would simply meant that next time your tired for work, the host would be re-initialised. This would happen on the very next connection.

The date shown for that computer joining the project would be the date it was given the new identity, but you would still have your original joining date shown on the forum boards nect to every post.

It is not a perfect solution, but in my opinion offers the most practical balance between taking up the admins time and bringing the database back to something manageable.

R~~

Too bad about that ghost hosts programming error - costly if it clogs up the database.
Solution sounds good to me - I don't care that much about joining dates, just as long as I get my first WU and can start crunching it.
Thanks for the update.
ID: 16453 · Report as offensive     Reply Quote
Fritz
Avatar

Send message
Joined: 4 Sep 05
Posts: 13
Credit: 536,862
RAC: 0
Message 16458 - Posted: 4 Mar 2007, 11:17:45 UTC - in response to Message 16453.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


Need to do some searching to find the info again, but this ghosting is showing up in other projects. At least one 'got rid of the ghosts' by running an automated merge based on simple criteria

All hosts with same owner, same machine specs and have no overlapping dates were considered one machine and merged.

This would effectively rid the system of almost all ghosts. Pruning would only eliminate the ghosts that did no work. Merging then pruning systems that had never received work would get few more since there will be cases where the machines reported specs changed.
ID: 16458 · Report as offensive     Reply Quote
J Langley

Send message
Joined: 31 Dec 05
Posts: 68
Credit: 8,691
RAC: 0
Message 16565 - Posted: 16 Mar 2007, 12:15:46 UTC - in response to Message 16450.  

Any chance of another update?
ID: 16565 · Report as offensive     Reply Quote
John McLeod VII
Avatar

Send message
Joined: 2 Sep 04
Posts: 165
Credit: 146,925
RAC: 0
Message 16567 - Posted: 17 Mar 2007, 3:52:59 UTC - in response to Message 16452.  

[...]prune out the millions of unwanted hosts from the db before the real migrate, perhaps only migrating those hostids that have ever submitted work.


I like that idea. For those hosts that never submitted work but still want to participate, they can manually add "themselves" back to the project post-migration.



I joined before Christmas, leave my laptop running and ready for any LHC work but haven't received my first WU yet. System regulary checks now, while I run other BOINC projects.
I'd hate to be deleted after patiently waiting and have to wait for some sort of post-migration period to be over before I can re-apply. Maybe there needs to be a list of ID's that are still active because within the last x months either they submitted WU's or they have been polling for WU's.


Just to clarify:

There are a huge number of ghost hosts in the system, due to a bug that makes / made some hosts forget their identity. The proposal is aimed at the most straightforward way of pruning them out of the database to make it more manageable.

The deletion of your host, if you have not got any work by then would simply meant that next time your tired for work, the host would be re-initialised. This would happen on the very next connection.

The date shown for that computer joining the project would be the date it was given the new identity, but you would still have your original joining date shown on the forum boards nect to every post.

It is not a perfect solution, but in my opinion offers the most practical balance between taking up the admins time and bringing the database back to something manageable.

R~~

I have a slightly different solution that may generate a smaller table. Force a merge between all hosts that are normally allowed to merge that have the same name. This will get rid of more hosts as some of those host will still have WUs attached. Then auto delete any hosts that can be deleted that have not reported in the last 30 days or so that have no WUs attached. This should generate a fairly small table of hosts.


BOINC WIKI
ID: 16567 · Report as offensive     Reply Quote
Profile Bird-Dog

Send message
Joined: 16 Dec 05
Posts: 18
Credit: 1,523,201
RAC: 0
Message 16569 - Posted: 17 Mar 2007, 12:39:37 UTC

What about people who have attached, zero credit, zero rac for the simple fact they couldn`t get any work.
ID: 16569 · Report as offensive     Reply Quote
Profile Ocean Archer
Avatar

Send message
Joined: 13 Jul 05
Posts: 143
Credit: 263,300
RAC: 0
Message 16572 - Posted: 17 Mar 2007, 14:46:53 UTC

I would assume that Bird-Dog's concerns would be taken care of by examing the last time individuals with zero credit and zero RAC polled the site in an effort to obtain work. Anything older than some set point could be discarded


If I've lived this long, I've gotta be that old
ID: 16572 · Report as offensive     Reply Quote
Profile WimTea

Send message
Joined: 27 Mar 06
Posts: 21
Credit: 1,731
RAC: 0
Message 16573 - Posted: 17 Mar 2007, 19:25:29 UTC

This discussion on purging hosts from the database is valuable but a bit premature as long as there is still the old software at work...

Time for an update from the proj admins then? It's been another 2 weeks ...
ID: 16573 · Report as offensive     Reply Quote
1 · 2 · 3 · 4 . . . 6 · Next

Message boards : Number crunching : Because you asked....


©2022 CERN