Message boards : Number crunching : Deletion not happening
Message board moderation

To post messages, you must log in.

AuthorMessage
River~~

Send message
Joined: 13 Jul 05
Posts: 456
Credit: 75,142
RAC: 0
Message 15443 - Posted: 13 Nov 2006, 1:12:31 UTC

If Ben or colleagues are looking in, what is the situation re the Deleter?

I have noticed that no results have been deleted since we changed servers. Is this simply bec you still need the results online for some reason, or is the result deleter broken? I have a fantasy that the server will fill up and then grind to a halt...

R~~
ID: 15443 · Report as offensive     Reply Quote
m.mitch

Send message
Joined: 4 Sep 05
Posts: 112
Credit: 1,864,470
RAC: 0
Message 15490 - Posted: 18 Nov 2006, 2:54:16 UTC

I was hoping that had something to do with all the pending results still showing up. :-( Maybe not. Can't complain though, I only have about 50. This bloke has 216!! That's about 1,080 rows still filling up the database form a fault in one user's part of the project.


Click here to join the #1 Aussie Alliance on LHC.
ID: 15490 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15491 - Posted: 18 Nov 2006, 2:58:08 UTC

file_deleter is not even in the daemon list, so it's not set up to run at all!
ID: 15491 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15492 - Posted: 18 Nov 2006, 2:59:46 UTC - in response to Message 15490.  

That's about 1,080 rows still filling up the database form a fault in one user's part of the project.

Clearing that result list is a separate problem. What the file deleter does is delete *files*. This means the input and output files for those 200,000 results are still eating server diskspace!!! Unless they are removing them manually.
ID: 15492 · Report as offensive     Reply Quote
River~~

Send message
Joined: 13 Jul 05
Posts: 456
Credit: 75,142
RAC: 0
Message 15510 - Posted: 18 Nov 2006, 10:36:45 UTC - in response to Message 15492.  
Last modified: 18 Nov 2006, 10:39:50 UTC

Clearing that result list is a separate problem. What the file deleter does is delete *files*. This means the input and output files for those 200,000 results are still eating server diskspace!!! Unless they are removing them manually.


I had naively assumed that the deleter would *also* update the db to keep the two in sync -- clearly it has to read data from the db in order to know which files are still needed.

If you are telling me I was mistaken, well it's not the first time that has happened ;) But then it seems at first sight like it's a rather odd design to remove the files and not remove the db entries that refer to them.

And if not, then it is not clear to me whether the figure for results waiting for deletion (on the server status page, and in Scarcrows graphs) refers to results listed in the db, or results with files on the file server.

R~~
ID: 15510 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15521 - Posted: 18 Nov 2006, 15:47:55 UTC - in response to Message 15510.  
Last modified: 18 Nov 2006, 15:56:42 UTC

Clearing that result list is a separate problem. What the file deleter does is delete *files*. This means the input and output files for those 200,000 results are still eating server diskspace!!! Unless they are removing them manually.


I had naively assumed that the deleter would *also* update the db to keep the two in sync -- clearly it has to read data from the db in order to know which files are still needed.

file_deleter reads the database first, and deletes output files from completed results, and input files from workunits with no other unsent/in progress results remaining. Workunits and results are kept in database, they can be archived (into compressed XML) and deleted using a different program (db_purge). They are useful to have crash statistics and other stuff (since stderr is kept).

And if not, then it is not clear to me whether the figure for results waiting for deletion (on the server status page, and in Scarcrows graphs) refers to results listed in the db, or results with files on the file server.

Workunits/Results ready to delete = Workunits/Results waiting for file_deleter to do something about them. That is, delete input/output files.

EDIT: I think this would be something more urgent to take care of.
ID: 15521 · Report as offensive     Reply Quote
River~~

Send message
Joined: 13 Jul 05
Posts: 456
Credit: 75,142
RAC: 0
Message 15553 - Posted: 18 Nov 2006, 21:42:04 UTC - in response to Message 15521.  

EDIT: I think this would be something more urgent to take care of.


agree
ID: 15553 · Report as offensive     Reply Quote

Message boards : Number crunching : Deletion not happening


©2024 CERN