LHC@home

Most screen scrapers get hundreds or thousands of pages at a time, this is definately bad. If your program is actually generating less traffic than manually looking at the pages then there should not be a problem.

One reason that the "all scrapers are bad" idea gets into everyones head is that it tends to grow. ie. I can get this page why not this other one too, and more frequently.

---

BOINC WIKI

BOINCing since 2002/12/8

I can get this page why not this other one too, and more frequently.

In my experience, dealing with requests from the team and team manager, it is what I do with the data after I have got it that interests them. They want to see how they are doing compared to other team members crunching the project, they want their stats from all projects aggregated onto a single page, they want a BOINC combined rank list, things like that.

I do all of these things simply by manipulating the data I have stored locally. For LHC that means 3 http GET requests against the server 3-4 times a day, (actually, only really needs to be once a day at the moment with the project so quiescent).

Thanks for confirming what I've always believed John.

BOINCer's are competitive, (at least many are), and giving them what they want at low cost to the projects encourages them to crunch harder and faster.

fx Whistles "Everyones a winner babe..." /fx.

---

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

Teams should be retreving thier stats via xml and not screen scrappers.

If you don't know its:

http:// project-url /team_email_list.php?teamid=#&xml=1
Fill in # with your teams ID for that project.

If you are the team founder you can add
&account_key=#
where # is your account 32 digit key and you also get the team members email address included.

This xml returns ALL a teams members with thier name,cpid and current stats.
It is only 1 request to the server for the whole team info and not multiple depending on how many pages of team stats you would have if viewing screen by screen.

And Yes, this currently functions on LHC@home.

There are a few issues here.

First, the concensus is that the screen scraper as described here is not a problem.

Second, when I last looked at the RPC calls, not all projects implemented all RPC's, and some of those that did had non-standard quirks. The concern here is that having many "special versions" of stat readers is a recipe for mistakes.

Third, the RPC calls still hit the project database, whereas the XML files downloaded from the /stats directory do not. You are simply ftp'ing a serial file which has already been generated.

Fourth, I can't say I'd studied that particular RPC. Given it's name, it doesn't sound like the kind of call I'd use to obtain stats. Mind you, many of the RPC's have obscure names!

If this RPC is cross project implemented, and the data is as current as the stats screen, (which the /stats directory XML is not of course), then it may well be worth investigating. Useful suggestion, thanks for that.

*** EDIT ***

Within 15 minutes of typing that, I found that this RPC is not implemented on all projects.

---

Wave upon wave of demented avengers march cheerfully out of obscurity into the dream.

The home page has been updated!! (actually, on the 22nd or so)

22.11.2006 13:21 UTC
More news on LHC@home server support: it is currently planned to have the migration to Queen Mary College effective in January 2007. Until then, very little admin effort is available and work units will be available from the project only intermittently.

This thread has definitely shown persistence amongst Boinc enthusiasts, and as long as we communicate and educate the flames will die down! l8r ...

---

There are a few issues here.

First, the concensus is that the screen scraper as described here is not a problem.

Second, when I last looked at the RPC calls, not all projects implemented all RPC's, and some of those that did had non-standard quirks. The concern here is that having many "special versions" of stat readers is a recipe for mistakes.

Third, the RPC calls still hit the project database, whereas the XML files downloaded from the /stats directory do not. You are simply ftp'ing a serial file which has already been generated.

Fourth, I can't say I'd studied that particular RPC. Given it's name, it doesn't sound like the kind of call I'd use to obtain stats. Mind you, many of the RPC's have obscure names!

If this RPC is cross project implemented, and the data is as current as the stats screen, (which the /stats directory XML is not of course), then it may well be worth investigating. Useful suggestion, thanks for that.

*** EDIT ***

Within 15 minutes of typing that, I found that this RPC is not implemented on all projects.

I do not think you are screen scraping here. There is nothin wrong with what you do.

As a project admin and owner I do not have a problem with RPCs or HTTP GETs being used by each team for their own team data. If more than a team's own data is needed then the stats xml export should be used.

I do object to hundreds and thousands of GETs for multiple teams being processed by my systems when that is entriely the wrong thing to do. I have already blocked Ip addresses that do that and will block any others I find.

I guess I would also object if every team started collecting data using the GET method though. That would be more work than an ftp of the xml files as it involves dbms activity which is something projects struggle to minimise.

So on balance its OK in small quantities but if everyone starts doing it then I guess projects will find a way to turn it off.

Hope that helps the debate.

---

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php

---

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php

have you even tried to read this thread? Or the NEWS on the front page?

Sorry about the lack of kindness, but this is getting anoying...

Sysfried

indeed
i wish we had a /slap <user> function on the msgboard.

Do something please, we want statistics on Boinc Synergy

JMB is the best, no problem for him:
http://jmb.boinc-fr.net/lhc/index.php

have you even tried to read this thread? Or the NEWS on the front page?

Sorry about the lack of kindness, but this is getting anoying...

Sysfried

---

indeed
i wish we had a /slap <user> function on the msgboard.

how about a: /block_wu-submission <user>
or a: /reset_score <user>

or .... /slap <include block_wu-submission><include reset_score> <user>

*BEG*

sysfried

lol

/deattach_project <user>

indeed
i wish we had a /slap <user> function on the msgboard.

how about a: /block_wu-submission <user>
or a: /reset_score <user>

or .... /slap <include block_wu-submission><include reset_score> <user>

*BEG*

sysfried

---

lol

/deattach_project <user>

!
That would be.. crazy :)

---

hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)

---

hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)

Oh cmon we're just joking :) By the way, dont_allow_posting_in_forum already exists for moderators. It's called "banish author".

---

hmmm

now we only have to find a moderator

/ask_moderator_2_banish_author <user>

hmm yes. then we have all those threads about 'i have lost a project. how come?'

maybe then a /dont_allow_posting_in_forum <user>

lol

/deattach_project <user>

!
That would be.. crazy :)

Oh cmon we're just joking :) By the way, dont_allow_posting_in_forum already exists for moderators. It's called "banish author".

---

/msg moderator *poke*

---

100 days without xml-export then!!!
WHY???

100 days without xml-export then!!!
WHY???

You didn't read anything before posting, did you?

---

at least he didn't create a new topic
so he isn't dad bad

---

100 days without xml-export then!!!
WHY???

You didn't read anything before posting, did you?

Well, maybe it was only rhetoric meant, Nicolas? ;-)

---

Life is Science, and Science rules. To the universe and beyond
Proud member of BOINC@Heidelberg
My BOINC-Stats