tag:blogger.com,1999:blog-6555947.post4805490538859008700..comments2024-03-14T01:32:43.610-06:00Comments on The Geomblog: VLDB dogfight, and peer reviewSuresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-6555947.post-70374033950403065532008-07-29T09:01:00.000-06:002008-07-29T09:01:00.000-06:00Here is another update with additional references ...Here is <A HREF="http://www.cse.ohio-state.edu/~apaydin/docs/VLDB06.pdf" REL="nofollow">another update with additional references to Bloom filters</A>.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-26745041547376352008-07-26T18:17:00.000-06:002008-07-26T18:17:00.000-06:00There is an updated version of the VLDB 2006 paper...There is an updated version of the VLDB 2006 paper in question<BR/>http://www.cse.ohio-state.edu/~hakan/publications/VLDB06.pdfAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-27774653189234871022007-12-14T16:55:00.000-07:002007-12-14T16:55:00.000-07:00The title of this blog entry is a little sensation...The title of this blog entry is a little sensational. Guess that is how one gets readers.<BR/><BR/>Speaking as someone who knows a little about VLDB, it is a pretty good conference in database field. Like <A HREF="http://www.daniel-lemire.com/blog/" REL="nofollow">Daniel</A> said, the peer review process can be improved and it should. SIGMOD, another conference in database field, just instituted <A HREF="http://www.sigmod08.org/sigmod_research.shtml" REL="nofollow">Experimental Repeatability Requirements</A>. It won't have caught the particular problem, but at least it is doing something.John Wuhttps://www.blogger.com/profile/07711588437937278559noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-27789113509767809342007-12-10T18:24:00.000-07:002007-12-10T18:24:00.000-07:00We should first differentiate perception from real...We should first differentiate perception from reality. Is VLDB fair or not? I do not know, and I do not care much for the matter. Can you improve the process? I think we can.Daniel Lemirehttps://www.blogger.com/profile/01566622051558391310noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-90988778968572336662007-12-10T16:54:00.000-07:002007-12-10T16:54:00.000-07:00By the way, I can't see a button to post a comment...By the way, I can't see a button to post a comment for your post a few down on quantum computing, but the answer is no. Measurements are truly random and can't be repeated. The ability to do so would likely have interesting consequences---much faster algorithms for a few problems.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-6850067417041539632007-12-09T15:09:00.000-07:002007-12-09T15:09:00.000-07:00Yes, we form an opinion after the initial pages, b...Yes, we form an opinion after the initial pages, but I (and many others) don't pass Accept/Reject judgements until we read the entire paper. We especially don't make "Accept" judgements until the very end of the paper (fortunately or unfortunately, reviewers are combing the text for reasons to reject a paper).<BR/><BR/>But we digress again :-). I don't believe the system is broken for conferences in my area. Each paper receives 5-6 thorough reviews (based on a reading of the entire paper) and it is extremely rare for a paper with obvious flaws to get accepted. If this is already being done for a conference, it is not clear that the filter is made much stronger by assigning a formal shepherd to every paper (at least not enough to warrant all the work).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-48635097199500469192007-12-09T14:39:00.000-07:002007-12-09T14:39:00.000-07:00Note that the two statements do not contradict: ie...Note that the two statements do not contradict: ie.<BR/><BR/>"Every reviewer must read every paper in its entirety" AND<BR/><BR/>"Reviewers tend to make judgements about papers after reading a few pages"<BR/><BR/>do not contradict each other. The first refers to the letter and spirit of the reviewing process, and the second references the underlying psychological dynamic by which people tend to process arguments. <BR/><BR/>Which is why the main structural features of a reviewing process must address and counteract people's inherent biases; multiple reviewers, a PC discussion, even public shaming if appropriate :)Suresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-62724161225451335092007-12-09T10:11:00.000-07:002007-12-09T10:11:00.000-07:00I am referring to the practice. I don't have much ...I am referring to the practice. I don't have much direct contact with the VLDB community, but if this claim of John Wu (possibly regarding some conferences) is true: "There is a common belief that the conference program committees never read more than a few pages of a conference paper when they make a decision on whether to accept or reject a paper", then such a conference should not be deemed "thoroughly peer-reviewed". A better path for the conference is to insist (and exert peer-pressure) that everyone read every assigned paper. Such a solution is more time-efficient than assigning a shepherd to every paper and less likely to allow a poor paper to slip through the cracks.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-17251125273442787062007-12-09T09:37:00.000-07:002007-12-09T09:37:00.000-07:00well I don't think there's an official mandate to ...well I don't think there's an official mandate to read a paper partially. Reviewers are of course expected to read papers thoroughly: it's the practice that occasionally differs from the principle. <BR/><BR/>I also would hesitate to tar an entire conference with the (alleged) mistakes of one reviewer.Suresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-77152562102385937582007-12-09T09:08:00.000-07:002007-12-09T09:08:00.000-07:00I was a bit surprised at the tone of the initial a...I was a bit surprised at the tone of the initial article (sounds pretty irresponsible in itself... but I digress..)<BR/><BR/>I was surprised that there are fields where the PC only partially reads a paper. I certainly read each of my assigned papers completely and I can safely vouch for most of my fellow PC members (architecture conferences).<BR/><BR/>For papers that require better presentation, the PC typically assigns a shepherd (although, the shepherd is not officially listed on the paper). I see no problem in extending this concept to every paper and giving the shepherd credit. But it looks like this field has a much bigger problem: a conference where PC members only partially read a paper is NOT thoroughly peer-reviewed!! PC members are better off spending a few more hours reading each paper rather than spending many hours shepherding their assigned paper.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-11590329785940849672007-12-08T14:34:00.000-07:002007-12-08T14:34:00.000-07:00Quite a few papers are in a subarea where there mi...Quite a few papers are in a subarea where there might not be any "authoritative" program committee member (off-hand and unverified examples: certain topics in logic, distributed or parallel algorithms in STOC/FOCS, perhaps crypto at SODA etc.).<BR/><BR/>This suggestion might lead to rejecting such papers much more easily, thereby narrowing the conference's scope (by people being selfish). <BR/><BR/>Perhaps at VLDB this is less of a problem because they have very large PCs. On the other hand, a typical PC member at VLDB may review only 10-15 papers (isn't it?) and typically do not use external reviewers, effectively disclosing more information about the actual reviewers of each submission.Anonymousnoreply@blogger.com