tag:blogger.com,1999:blog-6555947.post3721224241252487082..comments2024-03-14T01:32:43.610-06:00Comments on The Geomblog: On PC submissions at SODA 2020Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comBlogger46125tag:blogger.com,1999:blog-6555947.post-20417300094647447852019-04-04T04:21:33.813-06:002019-04-04T04:21:33.813-06:00"(2) being a place to meet peers. "
Act...<i>"(2) being a place to meet peers. "</i><br /><br />Actually this is not the role of conferences anymore in my experience. Dedicated workshops, research visits, special semesters, Dagstuhl, Simons, etc. These are the places where people collaborate. Not some four days of intense talks by hundreds of participants with different research interests. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-68623087822002298522019-04-02T00:57:57.865-06:002019-04-02T00:57:57.865-06:00The single responsibility principle states that ev...The single responsibility principle states that every module or function should have only one reason to change. I think that also applies to CS conferences. The problem of CS conferences is that they have two roles: (1) giving quality stamps (2) being a place to meet peers. Before we bringing any changes to the conferences, shouldn’t we separate these roles at first? In fact, replacing conferences by an online repository with a rapidly rotating “PC” is a good idea. Why not let this online repository play the role of giving quality stamps and let conferences serve solely as a venue to meet peers and present new research?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-15693277530530344392019-03-30T17:23:45.655-06:002019-03-30T17:23:45.655-06:00Actually, I'm focusing only on those students ...Actually, I'm focusing only on those students who submit to FOCS/STOC. The odds that a random student out of those students submitting to FOCS/STOC, has his/her advisor on the PC is very small. <br />For the rest of your comment, I don't see how it relates to my comments. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-1918016098736252812019-03-30T00:36:54.014-06:002019-03-30T00:36:54.014-06:00Since my comment has gotten a reaction, I'll e...Since my comment has gotten a reaction, I'll elaborate:<br /><br />SODA is trying to implement some form of "fairness through awareness" for handling PC submissions. It's far from perfect, but I was hoping that there would be a constructive discussion of its merits and shortcomings. Disappointingly, that has not happened.<br /><br />This blog post seems to dismiss the approach out of hand on the grounds that obviously the solution is fairness through blindness. Well, it's not obvious to me. Double-blinding is also far from perfect.<br /><br />Reviewer bias is a complex and difficult problem. There is no easy solution. And I don't think it is helpful to pretend that there is one. Double-blinding mitigates some problems -- I don't doubt that -- but it also sweeps others under the rug. In particular, I think it does a poor job of addressing PC submissions. Thomas Steinkehttps://www.blogger.com/profile/06443316959910188976noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-65011034439806545662019-03-29T13:59:45.801-06:002019-03-29T13:59:45.801-06:00Sure, if I sample a random PhD student in TCS, the...Sure, if I sample a random PhD student in TCS, the probability I land on one whose advisor is on the FOCS PC is small. This is just a weird way to restate the fact that there are many more TCS PhD students than there are FOCS PC members, and that most people don't have that many PhD students. But a large fraction of these students also would not submit to FOCS, so we are not saying very much with this statement, are we? To get a meaningful statement, you need to condition on students who have results that are publishable in FOCS, and then the chances would look, I conjecture, differently. To CS Prof this is simply evidence of bias, because they apparently live in a world where the quality of a student's work is independent of their advisor being invited to multiple PCs. But even in an ideally unbiased world, you would like the PC to consist of active people doing cutting edge work, and they would do that work with their students. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-35711213285243613232019-03-29T05:53:09.770-06:002019-03-29T05:53:09.770-06:00"what's the justification for spending so..."<i>what's the justification for spending so much money, mostly taxpayers money, on attending?</i>"<br /><br />Yuval, I tend to agree. I don't see a strong justification for this. But this is of course a different discussion. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-70711385686923673252019-03-29T05:45:57.849-06:002019-03-29T05:45:57.849-06:00Thomas, again, the fact that there are some papers...Thomas, again, the fact that there are some papers with many authors who are potentially also PC members does not contradict my claim. The claim is quite simple: the chances that this happens persistently for a given (random) student submitting to a conference are slim.<br /><br />They are slightly less slim when you sample from the pool of those students who are already enjoying a potential implicit or explicit bias by having their advisor well connected (which will tend to correlate well with serving on PC committees). But this is not an argument in favor of changing the rules of the game. On the contrary. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-92142337505362197392019-03-29T04:27:41.463-06:002019-03-29T04:27:41.463-06:00CS Prof: It is not uncommon for a paper to have ha...CS Prof: It is not uncommon for a paper to have half a dozen authors. A total ban on PC submissions could easily see a paper ruled out from two or three relevant conferences, forcing authors to choose between delaying publication by a year or picking a less relevant conference.Thomas Steinkehttps://www.blogger.com/profile/06443316959910188976noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-72474859910122393532019-03-29T03:08:56.310-06:002019-03-29T03:08:56.310-06:00"... let's replace conferences by an onli..."... let's replace conferences by an online repository with a rapidly rotating "PC" and continuous submissions, and we've done away with the issue of PC submissions."<br /><br />I completely agree with Yuval here. This is what we should really be discussing.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-31912320272576019632019-03-29T03:02:35.198-06:002019-03-29T03:02:35.198-06:00Authors abuse CoI declarations. I've encounter...Authors abuse CoI declarations. I've encountered extreme cases bordering on fraud (and perhaps on the wrong side of the border). Also, if the CoI is hidden, it might be hidden from the authors.<br /><br />At FOCS 2019, we'll be implementing the recommendations of the SafeToC ad-hoc committee, and other conferences are likely to do the same.<br /><br />Unfortunately, no formal mechanism will ever solve the problem. It's the responsibility of all PC members to watch for signs of bias, and concealing the names of the authors does not help them do it.Yuval Rabanihttp://www.cs.huji.ac.il/~yrabaninoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-44228496711773124072019-03-29T02:54:20.020-06:002019-03-29T02:54:20.020-06:00Anonymous: All I meant was that PCs should focus o...Anonymous: All I meant was that PCs should focus on the quality of the program and not on the interests of the authors.<br /><br />CS Prof: If indeed conferences merely give papers a stamp of quality and a resting place, what's the justification for spending so much money, mostly taxpayers money, on attending? If that's the case, let's replace conferences by an online repository with a rapidly rotating "PC" and continuous submissions, and we've done away with the issue of PC submissions.Yuval Rabanihttp://www.cs.huji.ac.il/~yrabaninoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-41505544597052324762019-03-29T02:32:32.550-06:002019-03-29T02:32:32.550-06:00I agree with the point CS Prof is making. Notice ...I agree with the point CS Prof is making. Notice that ESA has a guideline that someone can only be on the PC once in 5 years. This would make a PhD student subject to this issue likely only once at most. If a PhD student is frequently blocked from submitting to SODA because their advisor is on the PC, then they have the advantage of a well-connected advisor who is being invited to PCs all the time. <br /><br />One of the reasons given at the SODA 2019 business meeting for allowing PC submissions was that it is difficult to get people to accept PC invitations and this widens the pool. But does our community really do a good job of inviting all eligible people to serve on PCs ? Or do we just ask the same people all the time ? It would be good to have some statistics on this compared to other communities. <br /><br />(For example, it was also mentioned at the SODA 2019 business meeting that for STOC 2019, six people who wanted to submit papers accepted the PC invitation after the rules on PC submission were changed. Is it really the case that the PC chair could not find six other people ?)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-44270077482346220212019-03-28T16:17:19.292-06:002019-03-28T16:17:19.292-06:00This does not seem to contradict my claim. It is s...This does not seem to contradict my claim. It is still the case that the chances are very small for a potential student submitting to a given conference to have his/her advisor on the PC. <br /><br />There is no need to restrict the sampled group of students to only those with advisors that tend to serve on all the usual conferences. As I pointed out, this may even lead to more bias in acceptance: we are changing the rules of the game in favor of those students who are already enjoying some potential implicit of explicit positive bias. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-49916974764734163182019-03-28T14:58:45.439-06:002019-03-28T14:58:45.439-06:00To CS Prof: obviously if the student and the super...To CS Prof: obviously if the student and the supervisor are doing well, scientifically speaking, that increases the likelihood that the supervisor will get asked to be on the next FOCS/STOC/SODA PC. Then the supervisor has to choose between serving on the PC, which can be important for the career of young untenured faculty, or publishing a paper, which is certainly important for the career of a PhD student. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-9339364330549263822019-03-28T14:44:04.702-06:002019-03-28T14:44:04.702-06:00After a brief look, it sounds reasonable.After a brief look, it sounds reasonable.Sashohttps://www.blogger.com/profile/09380390882603977159noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-36546269628393552852019-03-28T13:20:42.455-06:002019-03-28T13:20:42.455-06:00I am not entirely convinced this is a major proble...I am not entirely convinced this is a major problem. The chance that for a given student his/her advisor is on a given PC committee is quite small, when considering the pool of all potential students submitting to the conference. I presume that it becomes non negligible when you count only those students/advisors that have accepted papers or are PC members, but that goes to show the possible (implicit) bias in the process of acceptance. CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-74932888830865845112019-03-28T11:53:17.779-06:002019-03-28T11:53:17.779-06:00One impact that has not been discussed at all is t...One impact that has not been discussed at all is the effect on students. At this point, I'm fine if I skip submitting to a conference or two. But this is not OK for students, for whom, in the short-term, the timing and venue of their papers carries far more weight. I (and other young researchers I talked to) remember being frustrated by not being able to submit when our advisors were on a committee. I know I wouldn't want my service on a committee to be the difference between my student getting a job and not. I've heard that there have been advisors who selflessly removed themselves from papers to not hold back their students from submitting.<br /><br />In short, I feel like students are the ones who suffer the most from not allowing PC submissions. I would like a solution which avoids hurting students in this way, potentially even if it comes at some other cost. I refrain from opining whether the SODA 2020 system is best, or whether double blind is required (as Boaz said, I believe PC submissions and double blind are two separate issues), or something else entirely.Gautam Kamathhttp://www.gautamkamath.comnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-46884928160393323112019-03-28T09:37:07.753-06:002019-03-28T09:37:07.753-06:00Sasho, what do you think of the SIGPLAN guidelines...Sasho, what do you think of the SIGPLAN guidelines that I linked to? They are interesting in that PLDI was historically like theory venues in that they disallowed PC submissions. Now that they do allow them they have a much more detailed process to handle PC submissions. Is that something along the lines of your sense that "more is needed than just DB review"?Suresh Venkatasubramanianhttps://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-14376212492019822952019-03-28T09:10:19.327-06:002019-03-28T09:10:19.327-06:00The SODA rules make me more comfortable than makin...The SODA rules make me more comfortable than making the conference double blind and then declaring that PC submissions are allowed and no special care needs to be taken with them. I’ve recently reviewed for a few ML conferences and I was > 90% sure I could identify some of the authors for a third of the submissions I got. This is without doing anything improper like googling around for the paper title: I just got assigned papers in my narrow subarea, and usually that’s narrow enough to know who does what. Now this may be ok for “PC members” of ML conferences, who are just glorified reviewers and there are so many of them you don’t even know who they all are. But for theory PCs, I would have little confidence in a solution which introduces DB and then handles a PC submission like any other. Sashohttps://www.blogger.com/profile/09380390882603977159noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-10662135430224413712019-03-28T08:44:53.405-06:002019-03-28T08:44:53.405-06:00Actually, Yuval, I tend to disagree even with this...Actually, Yuval, I tend to disagree even with this part:<br /><br /><i>"PCs should be concerned with selecting an attractive program that will draw attendance and will reward attendance. Focusing on fairness towards the authors is the wrong attitude."</i><br /><br />It is not the concern of PCs to select an "attractive" program that will draw attendance nor will reward attendance.<br />The sole concern of the PC is to select the most scientifically important papers within the specified scope of the conference. That is it.<br />"Attractiveness" has no clear meaning here. For instance, one can be attracted to a conference by the hope to meet his/her friends, another by meeting senior academics who he/she would like to make contact with. By your logic, one should always prefer to accept papers by powerful people since this would immediately increase the attractiveness of the conference. <br /><br />Even if what you mean is simply the "attractiveness of the result to the audience, irrespective of the authors", then it does not make sense. By this logic, if you accept many papers in subject X, for example, then the next paper in subject X has an advantage over a paper in a different subject, just because it will be much more attractive to the expected audience (who are already interested in subject X).<br /><br />Moreover, the whole premise of choosing papers towards "having a good conference/program" does not hold in CS. Since in CS conferences are one of the main venues of formal publication, the acceptance of papers is mostly a stamp of quality, a line in the CV, etc. Conferences have already lost its central place as a venue to meet and present new research. It is a place where work is being "accepted" and "archived" for most part. <br />CS Profnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-53063055605527642642019-03-28T06:07:32.691-06:002019-03-28T06:07:32.691-06:00Yuval,
Your point 1 echoes a common misconception...Yuval,<br /><br />Your point 1 echoes a common misconception about fairness. It is not a distraction from the main PC goal, and it is not charity toward the authors at the expense of the audience and the quality of the program. On the contrary, its goal is to ensure that the program is selected based on parameters relevant to its quality, and not irrelevant ones like authors' gender, nationality or institutional affiliation. It may seem intuitive that simply using all available data to make "objective" decisions would lead to fairness as a byproduct, but there is vast and undeniable evidence of the opposite, that it leads to substantial implicit biases. Eliminating those biases should be a concern of PCs precisely because it aligns with their primary goal of selecting the best program.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-2334741625345060262019-03-28T03:25:19.429-06:002019-03-28T03:25:19.429-06:00“The worst conflicts of interest are the hidden on...“The worst conflicts of interest are the hidden ones, such as poor personal relations between author and reviewer”<br /><br />In this case, we need extensive CoI declarations at author submission time.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-80925698115247889442019-03-27T21:42:26.091-06:002019-03-27T21:42:26.091-06:00Since no one mentioned it yet: the proposal for a...Since no one mentioned it yet: the proposal for allowing PC submissions in SODA has been discussed during the business meeting at SODA 19, voted upon, and passed with a substantial support. This is not to say that the discussion should not continue - it should. Although most CS conferences do allow PC submissions, theory conferences are structured differently enough to make any adaptions non-trivial, and it might take a few iterations to converge on a solution.<br /><br />Regarding double-blind submissions: from my point of view, SODA should adopt DB system if the majority of the conference community is supportive and willing to put up with (minor, but non-negligible) inconveniences resulting from the change. The discussion at SODA'18 business meeting indicates that this might indeed be the case, but it is important to discuss specific implementations and their consequences. The experience of ESA'19, which is experimenting with DB submissions this year, should be helpful in this context.<br /><br />Piotr IndykPiotrhttps://www.blogger.com/profile/13283386044289655376noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-60296420368024014862019-03-27T13:36:20.915-06:002019-03-27T13:36:20.915-06:00PART II
3. There are cases where the identity of ...PART II<br /><br />3. There are cases where the identity of the authors is important to the review process. Two obvious examples are: an author with a history of mistakes (the reviewer might want to pay extra attention to verifying the proofs), a paper proposing a new problem or model (such papers fair poorly, for good reasons that judgment of potential significance is difficult, and the past record of the author as a modeler is a good indication). The identity of the authors is often also important in the bidding phase (abstracts don’t always provide good indication), and it helps guarantee a good match between papers and reviewers.<br /><br />4. Double-blind reviewing is largely a hoax. A quick web search often reveals the identify of the authors (arXiv post, title on personal web page, talk announcement, ...). The topic, style of writing, references, often yield an easy guess. The worst conflicts of interest are the hidden ones, such as poor personal relations between author and reviewer, or some vested interest of the reviewer in the paper. PC members are often aware of at least some of these issues regarding their fellow members or external reviewers. When the names are known to the PC, this is common knowledge, so people are careful with bias, and they keep an eye over suspicious situations. When the names are concealed, they might be known or discovered by everyone, but it is not common knowledge that they’re known. So people won’t be as careful with bias and won’t watch over others as carefully. If the identity of the authors is meaningfully beneficial to some, these authors will eventually make sure it’s known, with or without double-blind reviewing, but it would still not be common knowledge that it’s known.<br /><br />Disclaimer: I would refuse to serve on a double-blind PC, and as a general rule I refuse to do double-blind external reviewing (unless I’m really interested in reading the paper, or I’m asked by someone I can’t say no to). If I do review double-blindly, I search the web for the authors. This frequently succeeds.<br /><br />I’m currently serving on the SODA 2020 PC, where for the first time PC submissions are allowed. I don’t have a strong opinion regarding PC submissions at SODA (and wasn’t involved in the decision to do it), but I definitely don’t think that double-blind reviewing has any impact on the negatives. I think the negatives diminish with size of the conference (#submissions and #papers accepted). So the negatives of PC submissions in SODA (600 submissions, 180 accepted) are diminished compared to a similar policy being adopted, for instance, at FOCS (300 submissions, 90 accepted). The numbers are rounded estimates.Yuval Rabanihttp://www.cs.huji.ac.il/~yrabaninoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-41496227628305382562019-03-27T13:34:16.729-06:002019-03-27T13:34:16.729-06:00PART I
The drive for double-blind reviews is moti...PART I<br /><br />The drive for double-blind reviews is motivated, so I think, on the following premises:<br />1. A primary concern guiding PCs should be fairness to the authors.<br />2. The interests of authors are encoded in the one-bit information of accept/reject.<br />3. The identity of the authors is objectively insignificant to the review process.<br />4. Double-blind reviews improve fairness towards authors.<br /><br />I disagree with each and every one of these premises:<br /><br />1. Fairness to the authors should not be a concern of PCs. I don’t mean to say that PCs should be unfair to authors. What I mean is that PCs should not be driven by it, but rather it should be a byproduct of the PCs main driving force, which should be fairness towards the conference potential audience. PCs should be concerned with selecting an attractive program that will draw attendance and will reward attendance. Focusing on fairness towards the authors is the wrong attitude.<br /><br />2. The interests of authors are a bit more complex than this one-bit information. Authors have an interest that the conference where they published in the past and/or will publish in the future will maintain its quality and prestige. Some authors have an interest that some reviewers be impressed with their work (e.g., for reference letters). The most intimate relationship a researcher has with a paper authored by others is often reviewing that paper, and that’s often when the paper and its authors get registered in the reviewer’s awareness. Some people give poor talks but write beautifully. Is it fair to conceal the identity of the authors of a well-written paper that might otherwise be overlooked by the reviewer? Our field is quite accommodating to bright young researchers primarily because of the conference system and the exposure it grants.Yuval Rabanihttp://www.cs.huji.ac.il/~yrabaninoreply@blogger.com