Message boards : Number crunching : /stats/ empty
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · Next

AuthorMessage
Profile Keck_Komputers

Send message
Joined: 1 Sep 04
Posts: 275
Credit: 2,056,164
RAC: 1,569
Message 15629 - Posted: 22 Nov 2006, 23:21:41 UTC

Most screen scrapers get hundreds or thousands of pages at a time, this is definately bad. If your program is actually generating less traffic than manually looking at the pages then there should not be a problem.

One reason that the "all scrapers are bad" idea gets into everyones head is that it tends to grow. ie. I can get this page why not this other one too, and more frequently.
BOINC WIKI

BOINCing since 2002/12/8
ID: 15629 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 15634 - Posted: 23 Nov 2006, 9:32:26 UTC
Last modified: 23 Nov 2006, 9:46:57 UTC

I can get this page why not this other one too, and more frequently.

In my experience, dealing with requests from the team and team manager, it is what I do with the data after I have got it that interests them. They want to see how they are doing compared to other team members crunching the project, they want their stats from all projects aggregated onto a single page, they want a BOINC combined rank list, things like that.

I do all of these things simply by manipulating the data I have stored locally. For LHC that means 3 http GET requests against the server 3-4 times a day, (actually, only really needs to be once a day at the moment with the project so quiescent).

Thanks for confirming what I've always believed John.

BOINCer's are competitive, (at least many are), and giving them what they want at low cost to the projects encourages them to crunch harder and faster.

fx Whistles "Everyones a winner babe..." /fx.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 15634 · Report as offensive     Reply Quote
Profile Krunchin-Keith [USA]
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 2 Sep 04
Posts: 209
Credit: 1,482,496
RAC: 0
Message 15681 - Posted: 26 Nov 2006, 15:57:24 UTC

Teams should be retreving thier stats via xml and not screen scrappers.

If you don't know its:

http:// project-url /team_email_list.php?teamid=#&xml=1
Fill in # with your teams ID for that project.

If you are the team founder you can add
&account_key=#
where # is your account 32 digit key and you also get the team members email address included.

This xml returns ALL a teams members with thier name,cpid and current stats.
It is only 1 request to the server for the whole team info and not multiple depending on how many pages of team stats you would have if viewing screen by screen.

And Yes, this currently functions on LHC@home.
ID: 15681 · Report as offensive     Reply Quote
Profile adrianxw

Send message
Joined: 29 Sep 04
Posts: 179
Credit: 673,019
RAC: 0
Message 15695 - Posted: 27 Nov 2006, 15:25:49 UTC
Last modified: 27 Nov 2006, 15:35:20 UTC

There are a few issues here.

First, the concensus is that the screen scraper as described here is not a problem.

Second, when I last looked at the RPC calls, not all projects implemented all RPC's, and some of those that did had non-standard quirks. The concern here is that having many "special versions" of stat readers is a recipe for mistakes.

Third, the RPC calls still hit the project database, whereas the XML files downloaded from the /stats directory do not. You are simply ftp'ing a serial file which has already been generated.

Fourth, I can't say I'd studied that particular RPC. Given it's name, it doesn't sound like the kind of call I'd use to obtain stats. Mind you, many of the RPC's have obscure names!

If this RPC is cross project implemented, and the data is as current as the stats screen, (which the /stats directory XML is not of course), then it may well be worth investigating. Useful suggestion, thanks for that.

*** EDIT ***

Within 15 minutes of typing that, I found that this RPC is not implemented on all projects.

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

ID: 15695 · Report as offensive     Reply Quote
Profile [B^S] ShanerX

Send message
Joined: 14 Jul 05
Posts: 41
Credit: 1,788,341
RAC: 0
Message 15699 - Posted: 27 Nov 2006, 17:15:13 UTC

The home page has been updated!! (actually, on the 22nd or so)

22.11.2006 13:21 UTC
More news on LHC@home server support: it is currently planned to have the migration to Queen Mary College effective in January 2007. Until then, very little admin effort is available and work units will be available from the project only intermittently.

This thread has definitely shown persistence amongst Boinc enthusiasts, and as long as we communicate and educate the flames will die down! l8r ...

ID: 15699 · Report as offensive     Reply Quote
Profile Tigher

Send message
Joined: 13 Jul 05
Posts: 40
Credit: 9,434
RAC: 0
Message 15721 - Posted: 28 Nov 2006, 16:27:40 UTC - in response to Message 15695.  

There are a few issues here.

First, the concensus is that the screen scraper as described here is not a problem.

Second, when I last looked at the RPC calls, not all projects implemented all RPC's, and some of those that did had non-standard quirks. The concern here is that having many "special versions" of stat readers is a recipe for mistakes.

Third, the RPC calls still hit the project database, whereas the XML files downloaded from the /stats directory do not. You are simply ftp'ing a serial file which has already been generated.

Fourth, I can't say I'd studied that particular RPC. Given it's name, it doesn't sound like the kind of call I'd use to obtain stats. Mind you, many of the RPC's have obscure names!

If this RPC is cross project implemented, and the data is as current as the stats screen, (which the /stats directory XML is not of course), then it may well be worth investigating. Useful suggestion, thanks for that.

*** EDIT ***

Within 15 minutes of typing that, I found that this RPC is not implemented on all projects.


I do not think you are screen scraping here. There is nothin wrong with what you do.

As a project admin and owner I do not have a problem with RPCs or HTTP GETs being used by each team for their own team data. If more than a team's own data is needed then the stats xml export should be used.

I do object to hundreds and thousands of GETs for multiple teams being processed by my systems when that is entriely the wrong thing to do. I have already blocked Ip addresses that do that and will block any others I find.

I guess I would also object if every team started collecting data using the GET method though. That would be more work than an ftp of the xml files as it involves dbms activity which is something projects struggle to minimise.

So on balance its OK in small quantities but if everyone starts doing it then I guess projects will find a way to turn it off.

Hope that helps the debate.


ID: 15721 · Report as offensive     Reply Quote
[AF>HFR] ToOm

Send message
Joined: 15 Jul 05
Posts: 3
Credit: 42,688
RAC: 0
Message 15731 - Posted: 29 Nov 2006, 3:03:35 UTC

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php
ID: 15731 · Report as offensive     Reply Quote
Profile sysfried

Send message
Joined: 27 Sep 04
Posts: 282
Credit: 1,415,417
RAC: 0
Message 15732 - Posted: 29 Nov 2006, 7:42:09 UTC - in response to Message 15731.  

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php


have you even tried to read this thread? Or the NEWS on the front page?

Sorry about the lack of kindness, but this is getting anoying...

Sysfried
ID: 15732 · Report as offensive     Reply Quote
watnou

Send message
Joined: 1 Sep 04
Posts: 101
Credit: 1,201,448
RAC: 2,117
Message 15734 - Posted: 29 Nov 2006, 13:40:49 UTC - in response to Message 15732.  

indeed
i wish we had a /slap <user> function on the msgboard.

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php


have you even tried to read this thread? Or the NEWS on the front page?

Sorry about the lack of kindness, but this is getting anoying...

Sysfried


ID: 15734 · Report as offensive     Reply Quote
Profile sysfried

Send message
Joined: 27 Sep 04
Posts: 282
Credit: 1,415,417
RAC: 0
Message 15735 - Posted: 29 Nov 2006, 14:20:10 UTC - in response to Message 15734.  

indeed
i wish we had a /slap <user> function on the msgboard.



how about a: /block_wu-submission <user>
or a: /reset_score <user>

or .... /slap <include block_wu-submission><include reset_score> <user>

*BEG*

sysfried
ID: 15735 · Report as offensive     Reply Quote
watnou

Send message
Joined: 1 Sep 04
Posts: 101
Credit: 1,201,448
RAC: 2,117
Message 15736 - Posted: 29 Nov 2006, 14:26:47 UTC - in response to Message 15735.  

lol

/deattach_project <user>

indeed
i wish we had a /slap <user> function on the msgboard.



how about a: /block_wu-submission <user>
or a: /reset_score <user>

or .... /slap <include block_wu-submission><include reset_score> <user>

*BEG*

sysfried


ID: 15736 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15737 - Posted: 29 Nov 2006, 14:55:58 UTC - in response to Message 15736.  

lol

/deattach_project <user>

!
That would be.. crazy :)
ID: 15737 · Report as offensive     Reply Quote
watnou

Send message
Joined: 1 Sep 04
Posts: 101
Credit: 1,201,448
RAC: 2,117
Message 15738 - Posted: 29 Nov 2006, 15:45:14 UTC - in response to Message 15737.  

hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)


ID: 15738 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15739 - Posted: 29 Nov 2006, 15:51:57 UTC - in response to Message 15738.  

hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)


Oh cmon we're just joking :) By the way, dont_allow_posting_in_forum already exists for moderators. It's called "banish author".
ID: 15739 · Report as offensive     Reply Quote
watnou

Send message
Joined: 1 Sep 04
Posts: 101
Credit: 1,201,448
RAC: 2,117
Message 15740 - Posted: 29 Nov 2006, 15:58:16 UTC - in response to Message 15739.  

hmmm

now we only have to find a moderator

/ask_moderator_2_banish_author <user>


hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)


Oh cmon we're just joking :) By the way, dont_allow_posting_in_forum already exists for moderators. It's called "banish author".


ID: 15740 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15741 - Posted: 29 Nov 2006, 16:19:43 UTC

/msg moderator *poke*
ID: 15741 · Report as offensive     Reply Quote
Profile pschoefer

Send message
Joined: 14 Jul 05
Posts: 21
Credit: 3,193,908
RAC: 0
Message 15816 - Posted: 14 Dec 2006, 15:35:16 UTC

100 days without xml-export then!!!
WHY???
ID: 15816 · Report as offensive     Reply Quote
PovAddict
Avatar

Send message
Joined: 14 Jul 05
Posts: 275
Credit: 49,291
RAC: 0
Message 15817 - Posted: 14 Dec 2006, 15:47:27 UTC - in response to Message 15816.  

100 days without xml-export then!!!
WHY???

You didn't read anything before posting, did you?
ID: 15817 · Report as offensive     Reply Quote
watnou

Send message
Joined: 1 Sep 04
Posts: 101
Credit: 1,201,448
RAC: 2,117
Message 15819 - Posted: 14 Dec 2006, 18:53:56 UTC

at least he didn't create a new topic
so he isn't dad bad
ID: 15819 · Report as offensive     Reply Quote
Profile DoctorNow
Avatar

Send message
Joined: 17 Sep 04
Posts: 19
Credit: 308,023
RAC: 20
Message 15821 - Posted: 14 Dec 2006, 21:34:27 UTC - in response to Message 15817.  

100 days without xml-export then!!!
WHY???

You didn't read anything before posting, did you?

Well, maybe it was only rhetoric meant, Nicolas? ;-)
Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats
ID: 15821 · Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · Next

Message boards : Number crunching : /stats/ empty


©2019 CERN