Unlike in many other areas of computer science, theory papers are not reviewed double-blind; reviewers in general know the identity of the authors of a paper. I will say upfront that I don't think there is a real problem with this approach. It's not that I think that we are saintlier than reviewers in other disciplines; it's just that a combination of the nature of the subject and the value system of the area makes objective evaluations a little easier. However,
A Princeton University research team asked people to estimate how susceptible they and "the average person" were to a long list of judgmental biases; the majority of people claimed to be less biased than the majority of people. A 2001 study of medical residents found that 84 percent thought that their colleagues were influenced by gifts from pharmaceutical companies, but only 16 percent thought that they were similarly influenced.We'd like to think that we can "factor out" the influence of author names when reviewing, but
Dozens of studies have shown that when people try to overcome their judgmental biases — for example, when they are given information and told not to let it influence their judgment — they simply can't comply, even when money is at stake.What's also interesting is how we make decisions with limited information,
...researchers asked subjects to evaluate a student's intelligence by examining information about him one piece at a time. The information was quite damning, and subjects were told they could stop examining it as soon as they'd reached a firm conclusion. Results showed that when subjects liked the student they were evaluating, they turned over one card after another, searching for the one piece of information that might allow them to say something nice about him. But when they disliked the student, they turned over a few cards, shrugged and called it a day.Or if you dislike a paper, you look for evidence to reject it, and if you like it, you look for evidence to champion it (rather than looking for evidence first, and making a judgement later).
And yet, all the people who scream 'Bias' when papers submitted to single-blind reviewing are rejected don't necessarily have a point:
And yet, if decision-makers are more biased than they realize, they are less biased than the rest of us suspect. Research shows that while people underestimate the influence of self-interest on their own judgments and decisions, they overestimate its influence on others.What does all of this mean ? I am more biased than I think, but less biased than you think. It's good to keep that in mind (at least the first part), when reviewing papers. It's basic psychology after all.
what does all this mean? maybe that your community shouldn't review papers the way you do, perhaps?
ReplyDeletePosted by instead
How does one implement double-blind review? Forbidding the authors to talk about their research before it is published? Maybe double-blind review works in communities where people compete to be the first to achieve something (and therefore are secretive about their research), but my impression is that in CS theory, we mainly compete for attention and scarce conference talk slots. And therefore we talk about our research as much as possible, to get people interested. By the time one submits a paper to a conference, often many potential reviewers already know who was doing the research. And if not, they may google up the full version from the authors' website.
ReplyDeleteAs for being influenced by the name of the author: as a referee, I am. Not by the fame of the authors, but much so by my previous experiences with them. For conference papers, one does not always get enough details of the proofs or cannot always take the time to judge the correctness of the paper. When I know the authors have a habit of checking their results carefully before submitting, I am more likely to give them the benefit of the doubt than when I know the author has a history of submitting premature papers full of bugs. In particular, I will be quite biased against authors who have a history of conference papers full of bugs, of which a journal version never saw the light of day.
Is it bad to be biased like that? I don't think so.
Posted by Herman
Double blind review is actively used in other fields within computer science, so simply pointing out difficulties as a barrier to its implementation implementation is moot.
ReplyDeletePosted by Anonymous
One can classify the reviewing bias into "positive +bias" (roughly: get your "friends'" papers accepted) and "negative -bias" (roughly: get your "enemies'" papers rejected). Both are an issue, of course.
ReplyDeleteNow, double blind reviewing can work pretty well to reduce -bias, but not +bias. The reason is in incentives: a contributor who wants to avoid -bias can go to great length to disguise the authorship of the paper. On the other hand, a contributor who wants to utilize a +bias has plenty of options to do so, many of them essentially untraceable. The lengthy guidelines on anonymizing submissions can only do so much (and can have negative impact on dissemination of results on the way).
So, no silver bullet. We can as well try to live with the bias out there. Still, it might be worth to switch into double-blind process, just to reduce the -bias. Of course, we would not be able to use authors rep to judge correctness. But it might motivate everyone to include more or less complete proofs in submissions. Which is not a bad thing.
Posted by Piotr
One can classify the reviewing bias into "positive +bias" (roughly: get your "friends'" papers accepted) and "negative -bias" (roughly: get your "enemies'" papers rejected). Both are an issue, of course.
ReplyDeleteIt's also possible to attempt to appear unbiased and by doing so ending up biased against your friends and in favor of your enemies.
There's a Borges short story the name of which I forget (maybe Muthu knows?) about a junior academic who, wishing to get some editorial appointment, attacks a senior colleague under a transparent pseudonym. The senior colleague gives him the position since to do otherwise would appear biased.
Re double blind reviewing: the benefit is less bias, or possibly only the appearance of less bias, in a process that I feel is not very biased already. The cost is, either more tightly controlled distribution of information (blackouts on preprints and talks until the decision has been reached) or abuse of the system as prominent theoreticians give up their anonymity by announcing their results as preprints or blog postings. Is the cost worth the benefit?
I am a student and I am just curious.
ReplyDeleteHow many "enemies" does a fairly senior theoretician (> 5 years after Ph.d. graduation) have? And usually how the enemies became enemies?
Posted by Anonymous
I am more than 5 years out from my Ph.D, and I wouldn't consider myself senior :). having said that, i don't think (at least in theory) it's a matter of personal enemies so much so as a dislike for a particular area of research.
ReplyDeletePosted by Suresh
I remembered a situation in which, as a reviewer, I prefer not to be double-blind:
ReplyDeleteIf a submission is technically good, but the 14th in a sequence of papers on the same subject by the same authors, coming out several per year, with some papers in that sequence being made obsolete by improvements in other papers by the same authors a year later, I'm not going to look favorably on it: at that point in the progression, the authors should wait longer before collecting their results into papers, or write a book, instead of trying to collect as many conference talk slots as possible.
But I'll look more favorably on the same submission by an author who hasn't worked on the same problem before, for two reasons. First, there is no reason to expect that author to have merged the paper with the other ones in the sequence, so I have no reason to be annoyed by the author's behavior. And second, having more people work on a problem helps convince me that it's an interesting area.
Is that kind of reasoning inappropriate?
That's interesting reasoning. i don't think it's inappropriate as long as you make these reasons known during PC discussions or whatever, because I can see that this might not be a universally accepted viewpoint. I'd be somewhat sympathetic to your argument.
ReplyDeletePosted by Suresh
i don't think (at least in theory) it's a matter of personal enemies so much so as a dislike for a particular area of research.
ReplyDeleteI would agree with that. Which I guess makes the whole double blind thing somewhat less attractive. Although it might still be of use to counter the perception of -bias (see the last quote in Suresh's post)
Posted by Piotr
Concerning Eppstein's viewpoint, I think it depends on if the paper is (very) incremental or not. Too many incremental results on the same topic from the same author are annoying.
ReplyDeleteI remember that there was one athlete from Russia (or USSR?), who barely set the new world record in every Olympics game he attended. Everyone knew that he could do it in once.
Posted by Anonymous
Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another.
ReplyDeletePosted by Anonymous
Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another.
ReplyDeletePosted by Anonymous
Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another.
ReplyDelete