tag:blogger.com,1999:blog-6555947.post114525956837646634..comments2024-03-14T01:32:43.610-06:00Comments on The Geomblog: Bias in paper reviewingSuresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-6555947.post-1145314302352396282006-04-17T16:51:00.000-06:002006-04-17T16:51:00.000-06:00Sergei Bubka, ukrainian formerly USSR, made more t...Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145314252422903482006-04-17T16:50:00.001-06:002006-04-17T16:50:00.001-06:00Sergei Bubka, ukrainian formerly USSR, made more t...Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another. <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>AnonymousAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145314211101026882006-04-17T16:50:00.000-06:002006-04-17T16:50:00.000-06:00Sergei Bubka, ukrainian formerly USSR, made more t...Sergei Bubka, ukrainian formerly USSR, made more than 20(was it 30? records) in pole vault. Lot of athletic meets offer prize money for breaking a world record in the meet. Which certainly gives a direct incentive. Like if some people are going to count number of papers or how many papers in last 2 years, it will entice people to publish one improvement after another. <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>AnonymousAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145310837175963362006-04-17T15:53:00.000-06:002006-04-17T15:53:00.000-06:00Concerning Eppstein's viewpoint, I think it depend...Concerning Eppstein's viewpoint, I think it depends on if the paper is (very) incremental or not. Too many incremental results on the same topic from the same author are annoying. <BR/><BR/>I remember that there was one athlete from Russia (or USSR?), who barely set the new world record in every Olympics game he attended. Everyone knew that he could do it in once.<BR/> <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>AnonymousAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145308156406737382006-04-17T15:09:00.000-06:002006-04-17T15:09:00.000-06:00i don't think (at least in theory) it's a matter o...<EM> i don't think (at least in theory) it's a matter of personal enemies so much so as a dislike for a particular area of research. </EM><BR/><BR/>I would agree with that. Which I guess makes the whole double blind thing somewhat less attractive. Although it might still be of use to counter the <EM>perception</EM> of -bias (see the last quote in Suresh's post) <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A><A HREF="http://geomblog.blogspot.com/2006/04/bias-in-paper-reviewing.html#comments" REL="nofollow" TITLE="indyk at mit dot edu">Piotr</A>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145308078499351812006-04-17T15:07:00.000-06:002006-04-17T15:07:00.000-06:00That's interesting reasoning. i don't think it's i...That's interesting reasoning. i don't think it's inappropriate as long as you make these reasons known during PC discussions or whatever, because I can see that this might not be a universally accepted viewpoint. I'd be somewhat sympathetic to your argument.  <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>SureshSuresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145305603242433072006-04-17T14:26:00.000-06:002006-04-17T14:26:00.000-06:00I remembered a situation in which, as a reviewer, ...I remembered a situation in which, as a reviewer, I prefer not to be double-blind:<BR/><BR/>If a submission is technically good, but the 14th in a sequence of papers on the same subject by the same authors, coming out several per year, with some papers in that sequence being made obsolete by improvements in other papers by the same authors a year later, I'm not going to look favorably on it: at that point in the progression, the authors should wait longer before collecting their results into papers, or write a book, instead of trying to collect as many conference talk slots as possible.<BR/><BR/>But I'll look more favorably on the same submission by an author who hasn't worked on the same problem before, for two reasons. First, there is no reason to expect that author to have merged the paper with the other ones in the sequence, so I have no reason to be annoyed by the author's behavior. And second, having more people work on a problem helps convince me that it's an interesting area.<BR/><BR/>Is that kind of reasoning inappropriate?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145303068943374212006-04-17T13:44:00.000-06:002006-04-17T13:44:00.000-06:00I am more than 5 years out from my Ph.D, and I wou...I am more than 5 years out from my Ph.D, and I wouldn't consider myself senior :). having said that, i don't think (at least in theory) it's a matter of personal enemies so much so as a dislike for a particular area of research.  <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>SureshSuresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145300538240060422006-04-17T13:02:00.000-06:002006-04-17T13:02:00.000-06:00I am a student and I am just curious. How many "en...I am a student and I am just curious. <BR/><BR/>How many "enemies" does a fairly senior theoretician (> 5 years after Ph.d. graduation) have? And usually how the enemies became enemies? <BR/><BR/><BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>AnonymousAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145296679912341292006-04-17T11:57:00.000-06:002006-04-17T11:57:00.000-06:00One can classify the reviewing bias into "positive...<I>One can classify the reviewing bias into "positive +bias" (roughly: get your "friends'" papers accepted) and "negative -bias" (roughly: get your "enemies'" papers rejected). Both are an issue, of course.</I><BR/><BR/>It's also possible to attempt to appear unbiased and by doing so ending up biased against your friends and in favor of your enemies.<BR/><BR/>There's a Borges short story the name of which I forget (maybe Muthu knows?) about a junior academic who, wishing to get some editorial appointment, attacks a senior colleague under a transparent pseudonym. The senior colleague gives him the position since to do otherwise would appear biased.<BR/><BR/>Re double blind reviewing: the benefit is less bias, or possibly only the appearance of less bias, in a process that I feel is not very biased already. The cost is, either more tightly controlled distribution of information (blackouts on preprints and talks until the decision has been reached) or abuse of the system as prominent theoreticians give up their anonymity by announcing their results as preprints or blog postings. Is the cost worth the benefit?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145294214078194512006-04-17T11:16:00.000-06:002006-04-17T11:16:00.000-06:00One can classify the reviewing bias into "positive...One can classify the reviewing bias into "positive +bias" (roughly: get your "friends'" papers accepted) and "negative -bias" (roughly: get your "enemies'" papers rejected). Both are an issue, of course.<BR/><BR/>Now, double blind reviewing can work pretty well to reduce -bias, but not +bias. The reason is in incentives: a contributor who wants to avoid -bias can go to great length to disguise the authorship of the paper. On the other hand, a contributor who wants to utilize a +bias has plenty of options to do so, many of them essentially untraceable. The lengthy guidelines on anonymizing submissions can only do so much (and can have negative impact on dissemination of results on the way).<BR/><BR/>So, no silver bullet. We can as well try to live with the bias out there. Still, it might be worth to switch into double-blind process, just to reduce the -bias. Of course, we would not be able to use authors rep to judge correctness. But it might motivate everyone to include more or less complete proofs in submissions. Which is not a bad thing. <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A><A HREF="http://geomblog.blogspot.com/2006/04/bias-in-paper-reviewing.html#comments" REL="nofollow" TITLE="indyk at mit dot edu">Piotr</A>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145284765060209152006-04-17T08:39:00.000-06:002006-04-17T08:39:00.000-06:00Double blind review is actively used in other fiel...Double blind review is actively used in other fields within computer science, so simply pointing out difficulties as a barrier to its implementation implementation is moot. <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>AnonymousAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145278323227505732006-04-17T06:52:00.000-06:002006-04-17T06:52:00.000-06:00How does one implement double-blind review? Forbid...How does one implement double-blind review? Forbidding the authors to talk about their research before it is published? Maybe double-blind review works in communities where people compete to be the first to achieve something (and therefore are secretive about their research), but my impression is that in CS theory, we mainly compete for attention and scarce conference talk slots. And therefore we talk about our research as much as possible, to get people interested. By the time one submits a paper to a conference, often many potential reviewers already know who was doing the research. And if not, they may google up the full version from the authors' website.<BR/><BR/>As for being influenced by the name of the author: as a referee, I am. Not by the fame of the authors, but much so by my previous experiences with them. For conference papers, one does not always get enough details of the proofs or cannot always take the time to judge the correctness of the paper. When I know the authors have a habit of checking their results carefully before submitting, I am more likely to give them the benefit of the doubt than when I know the author has a history of submitting premature papers full of bugs. In particular, I will be quite biased against authors who have a history of conference papers full of bugs, of which a journal version never saw the light of day. <BR/><BR/>Is it bad to be biased like that? I don't think so. <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>HermanAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1145274379789124242006-04-17T05:46:00.000-06:002006-04-17T05:46:00.000-06:00what does all this mean? maybe that your community...what does all this mean? maybe that your community shouldn't review papers the way you do, perhaps? <BR/><BR/><A></A><A></A>Posted by<A><B> </B></A>insteadAnonymousnoreply@blogger.com