My rant: Why you should care about the credit-system
6 Sep 2006 9:09:42 UTC
Topic 191798
(moderation:
Hi all, I thought you might be interested in my rant about the BOINC credit-systems in general and why scientists should care about it.
http://itnomad.wordpress.com/2006/09/06/boinc-why-you-should-care-about-the-credit-system/
Feedback is welcome :)
Cheers, Alex.
"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.
My rant: Why you should care about the credit-system
)
Yeah Alexander, that is fairly close to my way of thinking. It was the Science that attracted me but the credit and competition that has kept me. The competition has also made me spend over $12,000 on computers I did not really need and I will probably buy another one soon (what do I really need 5 computers for?).
The recent drop from Rosetta is making me think either quit or get another computer, to keep credit output up (still debating this one, some of my machines are down to 7 credits an hour).
I have accepted the changers but as the credit drops (6 hours for 7 credits in Rosetta and I increased processing time from 4 hours), with you doing more processing time, you start looking around, maybe even to non Boinc projects.
Projects like LHC and Predictor could listen to your words about being on the forums and message boards, Rosetta and Einstein leave them for dead.
the Science is the most important thing but not the only thing, credits are important too or we would not have had such heated arguments over it.
Conan wrote: RE: he
)
Conan wrote:
He, i know that :) Well, i didn't specifically buy my gear to crunch, i had them in the first place; but i let them crunching 24/7 resulting in a 1000 Euro/year electricity-bill. To quote Star Ship Troopers: "I'm doing my part!"
My combined RAC is also down by a couple of hundreds - back when i was only crunching for Einstein it was as high as ~1200 - but i also must admit that i was using Asok's binaries and truXoft's calibrating client, shame on me... ;-)
Why? Both projects are making a huge progress and I'm proud of "doing my part". See, Rosetta got a 10-million-USD grant from the Gates Foundation to do research around the HI-Virus' proteins. Einstein is progressing and their S3-report was great. Lot's of intersting stuff will happen in the future and i don't want to miss it!
Let me be honest, i also care about my RAC - as miw pointed out in the Rosetta-Forum:
miw wrote:
There's nothing i could add. :-)
Conan wrote:
I partly agree. It was bit of "bad style" how we learned about the most recent credit-calibration - but i don't think that the project-members will do the same fauxpas again.
The heated up arguments is really something horrible; the flamewars start to cook over, that was one of the reasons why i wrote the article in the first place.
Well, thanks for your feedback, i appreciate it.
Cheers, Alex.
"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.
I think I would agree with
)
I think I would agree with most of what you have said, Alexander. I would only take issue with how you seem to separate the 'dealing-with-people' aspect from the science of DC. More than a century of social science work (and literally decades of work on issues of volunteerism) would argue that this is part of the science of DC, and that the full potential of DC will never be reached until its social science is given attention on par with the project science and the computer science of DC.
RE: Feedback is welcome
)
Ok, here are my two cents:
The amount of "credit" (which btw. is a term I don't like at all) a participant (well, I don't like "user", too) gets may look fair to him or not, but I think the research and (programming) effort we actually (have) put into the crediting on Einstein@Home should be enough to show that we care a lot about this issue.
Changing the credit system from benchmaked to server-side crediting wasn't silent at all. The server-based credit system was developed (and mostly pushed in the project team by me) because of the trouble we had with the traditional system (wrong benchmarks and cpu-time reports), it was announced for the switch to the new S5 analysis run long before the new run started, and is widely accepted by participants as being much more fair and predictable than the previous system.
The new system required to have a "standard" - how much credit should be granted for a "template", an atomar, equally sized unit our workunits are built from. As a reference we chose the credit the "average" machine got on S4, with some correction factor to make up for the difference to the average credit on other projects, at least the ones with the largest "user-base" (SETI & CPDN). The idea was (and still is) that the "average participant" should get the same credit for contributing the same "CPU cycles" on all projects.
That, however, turned out to be hard to achieve, as the average values change e.g. with new hosts and new applications. We actually had to reduce the amount of credit given for a template to stay on that same level as the other projects, and if we want to continue to do so, we will need to continously watch the credit values and may need to make further adjustments. Note, however, that we don't change the whole crediting system, but a single parameter of it, that affects all "work" (i.e. workunits) of the project proportionally. This should not affect the competition between e.g. participants within a project.
However, "credits" actually don't buy you anything "real". They are just there for the psychological effect. There is no "scientific" meaning of "Milestones", i.e. if a number reaches a value that happens to be a multiple of 10^n. Yet still most of us set such numbers as personal goals, and it gives a good feeling to have reached it. There's still some magic in numbers, even when they're used "scientifically". And that's the reason why I honestly don't feel good when the credit (for future WUs - we're not taking away anything that has been granted) is reduced. The psychological effect of this is pretty bad, even if there is no "objective" meaning to such numbers - credits are all for the psychological effect.
Currently, I'm afraid, we'll stick to that scheme, simply because we don't have anything better to keep fair within the community of BOINC projects. But we (at least I) are still thinking about it and the discussion is going on.
BM
BM
Hello Bernd! Thanks for
)
Hello Bernd!
Thanks for taking your time.
We all appreciate your efforts in improving the application and we can clearly see that you spend a huge amount of time improving the application; hiring Akos was a step which wasn't unnoticed.
Oh this is my fault: When i wrote "silently" i didn't meant the system but the "calibration", however you call it. I'm going to correct this line in my posting, you're totally right.
Yes, this is exactly the point: "within a project". Lot's of people pointed out that, speaking of "credit-whoring", the real problem is not the credit-system or calibration of the individual projects but the competition between the all the projects.
From my point-of-view it looks like you get all the blame for being fair.
It's not your fault - the opposite is true, you spend lots of effort to change the system in a way that it'll be more fair.
Yes, they are, which is the whole point about my posting, a very important effect indeed. If you have a look over at the Rosetta@Home project, there's currently a flamewar going on, a really really bad one - no need to read it in full, ugly length; one participant (to keep consistent with your terminology), a Mac PPC user, found out that the Rosetta-application utilized the CPU completly but used only 20%-30% for real work - he was totally pissed off and angry.
This is one of those communication-GAUs(*) i was hinting too. Your recent credit-calibration fauxpas i was reffering too is nothing compared to that one.
Yes, i didn't expect anything else: Keep up your good work and i bet that everyone who's really interested into your work will accompany you through the transition to something more fair.
However, i think you could place the changes in your system into some more prominent place, like into the news-section; it's just cumbersome to dig up the news which are hidden in some postings.
Thanks a lot for your feedback and time; keep up the good work as we'll keep donating our spare CPU-cycles for Einstein!
Cheers, Alex.
(*) Edit for non-german readers: "GAU" is a german abbreviation for the absoulte worst-case accident which can happen in a nuclear-powerplant - core-meltdown (think Chernobyl).
"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.
Hi Scott!Scott Brown
)
Hi Scott!
Scott Brown wrote:
Hm, I'm not really confident enough to comment on this but what you say sounds interesting. What exactly do you mean with "the social science of DC" - or, let's dub it "social science of volunteerism"?
Are there any practical result one could gobble up and use it directly to improve, err, let's say (om my god, BS-alert), communication with the community?
Would the effort you have to put into an investigation be worth the result?
Thanks for reading,
Alex.
"I am tired of all this sort of thing called science here... We have spent
millions in that sort of thing for the last few years, and it is time it
should be stopped."
-- Simon Cameron, U.S. Senator, on the Smithsonian Institute, 1901.
RE: Hm, I'm not really
)
In DC, there are essentially three components to get it working and get it working effeciently: 1. The scientific application of the actual project (e.g., the theory and method behind gravitational waves, etc.); 2. The computer science of the applications and client (both the actual scientific application doing the analysis--e.g., the versions that Bernd and others work on--and the software that allows it to be distributed--the BOINC client and server code); and 3. the social science of DC (e.g., why people participate (or don't)? Why they stop participating? etc.).
Considerable effort has been put into properly implementing, understanding, and (using multiple meanings of the term) 'optimizing' parts 1 and 2. However, almost no effort has been expended on part 3. Why would part 3 be useful? Well, there are the more obvious things like understanding the who and why of participation that could lead to better methods for attracting participants and keeping them. More subtly, things such as the 'surprise' of the project adminstrators (across many projects) at issues that arise (e.g., issues of fair credit, etc.) could have been avoided with even a cursory reading of the plethora of research on volunteerism across mutliple disciplines in the social sciences, and appropriate steps taken to avoid the 'rough transitions' that have been experienced in some cases. UCB has made a rather pathetic attempt at a survey to get at this, but it was constructed by people with virtually no social science training making it essentially useless (and given the extremely poor construction of the survey instrument, possibly harmful via incorrect resutls).
@Bernd
As you will note from my points above, I would be in fundamental disagreement with you regarding the 'scientific' nature of credits and other 'social' components of DC. I would also point out that you quite incorrectly attribute these types of things to psychology solely. The extensive research on volunteerism and similar behaviors (including issues of rewards, etc.) includes considerable work in sociology and political science in addition to work by psychologists.
RE: @Bernd As you will
)
Sorry for my sloppines. I actually didn't distinguish between psychology and sociology, and didn't mean to attribute any of their work as being non-scientific. I was writing my reponse to Alexanders "rant", picking up his understanding of "scientific" (in terms of "science the project is about"). I should have been more careful, and surely your post would have pointed me in that direction - It was just that I was busy writing my response while you were posting yours.
So thanks for the clarification. I think that apart from my sloppy and incorrect use of terminology we aren't that much apart, and I fully agree with you in that the "social science of DC" hasn't got enough attention yet. SETI (and BOINC) has been started by computer scientists, which are rather focused on technical issues, and which are continously surprised by the social aspects of the projects, the participation, the engagement of the participants in every direction. The currently most burning question for me is what way of granting credit (or reward of any kind) would be the most helpful for DC in general and BOINC projects in particular. However, this again kind of reduces a social question (of fairness) to a technical aspect - I'm a computer scientist, after all.
BM
BM
Personally, I'd settle for
)
Personally, I'd settle for one point per valid WU completed. Long, short or whatever, you finish one WU, it validates, and you get one point. What more do you need? I never, and still don't, understand why anyone felt the need to "level the playing field". You use the 'puters you have and do as many as you can in any given time period.
RE: Personally, I'd settle
)
Probably a lot of people would abort the long ones. So, in this crediting system all WUs should be of equal length. I doubt that is practical with regard to the science. (I guess it would have been done like that if it were feasible in the first place...)
I did not intend to compete with other computers. Still, it felt good to get more credits than others. And it did not feel good when the granted credit dropped (although I totally understand). Dropping is just a negative thing in general. So instead of Einstein dropping credit, all other projects should raise their granted credit to align with a better project. I am curious, what you mean with to "be as fair as possible with the credits", Alexander... As I see it, optimization should only be done to improve the speed of crunching. It should not affect the credit people are given for a given WU. But that would increase the RAC as well, which is not desirable for inter-project competition. I'd say, work on a continuous calibration, to keep average credit level for every day for all projects, instead only after an improvement of the apps for a single project.
I'd like to say that I'm here for the science. But I am NOT. I just gave Einstein permission to use my computer to analyse their data. I would like to get more envolved though. So I agree with Alexander's point on reporting. I certainly hope, that this will be improved in the short future. More than the crediting system!
Keep up the good work,
Bert
Somnio ergo sum