For all but theory researchers, the reaction to the above statement is usually "don't they always?". In theoryCS, we pride ourselves on not having PC members submit papers to conferences. What ends up happening is:
- You can't have too many PC members on a committee because otherwise there won't be enough submissions
- The load on each PC member is much larger than reasonable (I'm managing 41 papers for STOC right now, and it's not uncommon to hit 60+ for SODA)
The main principle at play here is: we don't want potentially messy conflicts or complex conflict management issues if we do have PC members submitting papers. However, it seems to me that the practice of how we review papers is far different from this principle.
Consider: I get an assignment of X papers to review if I'm on a conference PC. I then scramble around finding subreviewers for a good fraction of the papers I'm assigned (I used to do this less, but I eventually realized that a qualified subreviewer is FAR better than me in most subareas outside my own expertise, and is better for the paper).
Note (and this is important) my subreviewers have typically submitted papers to this conference (although I don't check) and I rely on them to declare any conflicts as per conference guidelines.
Subreviewers also get requests from different PC members, and some subreviewers might themselves review 3-4 papers.
Compare this to (say) a data mining conference: there are 30+ "area chairs" or "vice chairs", and over 200 PC members. PC members each review between 5-10 papers, and often don't even know who the other reviewers are (although they can see their reviews once they're done). The area/vice chairs manage 20-30 papers each, and their job is to study the reviews, encourage discussion as needed, and formulate the final consensus decision and 'meta-review'.
If you set "theory subreviewer = PC member" and "theory PC member = vice chair", you get systems that aren't significantly different. The main differences are:
- theory subreviewers don't typically get to see other reviews of the paper. So their score assignment is in a vacuum.
- theory PC members are expected to produce a review for a paper taking the subreviewer comments into account (as opposed to merely scrutinizing the reviews being provided)
- managing reviewer comments for 30 papers is quite different to generating 30 reviews yourself (even with subreviewer help)
- A downside of the two-tier PC system is also that there isn't the same global view of the entire pool that a theory PC gets. But this is more a convention than a rule: there's nothing stopping a PC for opening up discussions to all vice chairs.
- One advantage of area chairs is that at least all papers in a given area get one common (re)viewer. that's not necessarily the case in a theory PC without explicit coordination from the PC chair and the committee itself.
But the main claimed difference (that people submitting papers don't get to review them) is false. Even worse, when submitters do review papers, this is 'under the table' and so there isn't the same strict conflict management that happens with explicit PC membership.
We're dealing with problems of scale in all aspects of the paper review and evaluation process. This particular one though could be fixed quite easily.
The solution is indeed quite easy: move to a journal centered community. Thereby, eliminate the importance of conference publications. This should solve all the ethical problems mentioned.
ReplyDeleteThe other advantages of journal publication are well known, and I will not repeat them here.
"The solution is indeed quite easy"
ReplyDelete[citation needed]
Although I am not familiar with reviewing, I believe that the process of assigning papers to reviewers is arbitrary. I would guess that the most important factor would be the commitee member's area of expertise.
ReplyDeleteWe could require any reviewer to announce any additional restrictions that would make them impartial towards a paper (e.g. I published it myself or the person is related to me). Then, those additional constraints would be considered and the papers would be assigned accordingly.
Also note that it is not necessary for the assignment to be final: When a reviewer spots a paper that he cannot review, he announces it and someone else handles it.
Finally, what is the motivation for researchers to do this? The same as in the current system: Continuing their careers and maintaining a good reputation as a researcher. Those reasons reinforce their motivation in a positive way: i.e. "I must uphold to certain standards of my community" as well as in a negative way: "If I am biased and someone finds out, I will probably be expelled from the community and face severe consequences in my life".
I second "CS prof". Could someone find another field with such a heavy concentration on conferences? With all these problems. Deadlines, half-cocked papers, just paper screening instead of a normal 2-3 months refereeing. And a "half made" science. Where and why we are rushing? That a PC member can reasonable handle 40-60 submissions in a short time, this is a joke. Not mentioning that he/she alone disturbs more that 100 "sub-referees". As once Lance Fortnow said "CS should grow up".
ReplyDeleteThird what CS prof said. I left theory CS because of how awful the quality of conference papers were, and how obviously biased the process was.
ReplyDeleteThe main claimed advantage for not having pc submissions is not that People submitting papers don't get to review them.
ReplyDeleteIt is that the discussion is much better if the pc members have a global view of the submission pool. Having pc submissions creates a conflict of interest between the pc member and his/her submission as well as other submissions in the same subarea. Enabling such discussion is also the main rational behind physical pc meetings. Without them, and/or where there is a huge program committee, it's harder to ensure that members participate in discussions beyond the papers they were assigned.
This issue is rather independent of the conference vs journal question. Whether is a conference or journal, the accept/reject descision needs to be made by someone, and questions of conflict of interest will always arise. The main difference is that journal editorial boards tend to be more stable, as opposed to a pc that changes every year, and so journal editors have more power.
I agree with Anon@1:38pm that the conference/journal issue is a red herring for this specific matter.
ReplyDeleteAnon's point is good in that having a global view is an important consideration. It's true that in two-tier systems, the vice chairs don't necessarily have that global view.
But in practice, it's rather hard for me as a PC member to participate in that global view: it takes a lot of time to issue a competent review in my pile, and at best I can be an interested observer on other papers, listening to what people have to say and maybe weighing in if I have something useful to add. So my participation at the global level is a little limited.
While submitting papers would indeed mess up the mechanics, how much would it really affect my global view? For example SODA takes 500 submissions, and I might submit 3 papers in a good year. The effect of this is minimal in the grand scheme of things.
I should also add that very few conferences have physical PC meetings (I think STOC and FOCS are the only ones at this point). Given the logistics and expense of travel, I don't mind moving towards all-electronic meetings, and with the right software (hotCRP comes to mind) it's not hard to still get a global view of the discussions.
ReplyDelete@Suresh: I still wonder WHY we stick at conferences? (And on all implied problems.) 500 submissions to SODA! 500 quickies? And 500 quick decisions. I think (and hope) science is done other way around.
ReplyDelete@Anon2:30 I don't disagree. But I think that's a different discussion. And I think that any discussion of that nature needs to discuss not the end-goal but how to get there incrementally.
ReplyDeleteThis discussion is about a much more mundane issue, but one that can help improve conference reviewing as it currently stands, by reducing reviewer load.
@Suresh: Unfortunately we cannot come to a "stable state" incrementally. The error is in the ROOT of of this "science making" PRINCIPLE. One cannot correct it by improving "conference reviewing" or the like. We need another principle.
ReplyDeleteIn your overview you say that "there is nothing stopping a PC from opening up the discussions to all vice-chairs". I agree that there is value in the overall discussion but once you open it up like this, how do you handle the COI of vice-chairs who have submitted papers? If vice-chairs are prohibited from submission (which is the case on some non-theory PCs) then it seems like an issue of semantics.
ReplyDeleteTwo-tier vice-chairs = PC members
Two-t4er PC members = subreviewers
@Anon9:26pm: If the meeting is physical, then they leave the room (like NSF panel deliberations). If the meeting is electronic, then the system prevents them from commenting/viewing any papers they have a COI with. They can still submit though. In fact, often conferences have two PC chairs so that even the PC chairs can submit, and the each one handles the other's submissions.
ReplyDeleteHi Suresh,
ReplyDeleteI agree that there is nothing holy about the rule that PC members cannot submit, AS LONG AS other conflict of interests rules are applied. Specifically, the applied CS conferences that I am aware of have pretty strict conflict of interest rules which are being taken rather seriously. This allows them to be more relaxed about PC reviewing, since there are mechanisms to deal with the resulting conflicts. In contrast, until recently, TCS conferences were pretty lax about conflicts. Fortunately this is changing now.
"...just paper screening instead of a normal 2-3 months refereeing."
ReplyDeleteThe standard timeline for journal publications in physics is six weeks, not 2-3 months. Also, reviewers of (even theoretical) physics papers generally do not verify correctness in detail, but only judge whether the paper is "interesting" and look for egregious errors.
Sound familiar?
I've had papers spend years in the journal reviewing process only to have them accepted without significant revision. Conferences have the advantage of time-certain decisions.
ReplyDeleteFor a once-per-year conference from a smaller community I can easily see allowing Pc members to submit. I am not sure that it would make much of a difference for FOCS/STOC and maybe not for SODA.
BTW: In some systems PCs, I've heard that the outcomes of decisions on PC members papers are kept secret from the PC members until the entire program is determined, though one can't completely keep this secret if one is hitting the limit on the number of accepted papers.
Switching to journals won't cause less papers to be written, or more referees to magically appear, and if journals strive for any reasonable turnaround time, then time pressures will also stay the same. (Of course today TCS journals can afford long refereeing processes precisely because their low importance means that not all papers get submitted, and authors don't deeply care about speedy decisions.)
ReplyDeleteSo, the main difference between conferences and journals is the physical meeting and presentations. People can argue if this meeting is a good thing or bad thing, but if someone claims that there is any other essential difference in issues such as quality of refereeing, conflict of interests, etc.. then I'd like to see proof of that.
Regarding PC submissions, I'm not sure that not allowing PC submission is the reason theory PC's are not larger. Nor am I sure that larger PC imply a higher quality reviewing process. (I think there are pros and cons to both large and small PC's, and unfortunately "quality of reviews" is not easily measurable.)
Larger PCs will reduce the reviewing load per reviewer. Independent of everything else, that is a good thing.
ReplyDeleteHaving served on multiple systems committee conferences, I've found PC members submitting papers works well in that context. Conflicted people (for a reasonably broad definition of "conflict", not just authors of a paper) have to leave the room during discussions (or are prevented from viewing the discussion if its an online meeting). And yes, efforts are made to make sure that papers that have PC members on them are not even listed as being in/out until the end of the process, though of course that's not always perfect.
ReplyDeleteOne can argue whether such approaches remove the bias one might expect for (or against) PC-authored papers. In my experience it works well, but that's an opinion; I don't have empirical evidence. I should point out even if it doesn't work perfectly, it may still be better than a system that doesn't allow PC-authored papers, as Suresh is pointing out here.
I've heard people say this won't work for theory conferences, since such COI policies would mean only "non-experts" would read the paper, and decisions would be worse. I find this argument arrogant and self-serving, personally. If you can't make a paper understandable and interesting to people beyond your friends on the committee, maybe it shouldn't be accepted. But I should point out that that was the argument I heard when I tried to put in COI policies.
Suresh, you might want to reference my old, dramatic posts on this issue:
http://mybiasedcoin.blogspot.com/2009/02/stoc-pc-meeting-part-iii-conflicts-of.html
http://mybiasedcoin.blogspot.com/2009/02/stoc-pc-meeting-part-iv-conflicts-of.html
And this less dramatic one.
http://mybiasedcoin.blogspot.com/2010/02/conflicts-of-interest-yet-again.html
@Jeffe: "Also, reviewers of (even theoretical) physics papers generally do not verify correctness in detail, but only judge whether the paper is "interesting" and look for egregious errors."
ReplyDeleteIf so, there is no difference, the same mess, you are right. But I have "old fashioned" views at what does it mean "doing science". Results should be archived, checked, not evaluated. Evaluation is the task of the entire community, and of TIME! Not of PCs or of editors. They should only handle the procedures, check for correctness, and for novelty. But please - no "stamps", no "certificates of truth", no advertising, no "main streams". In this (my) world, TCS conferences look like aggressive markets. Far away form just a silent presentation of scientific findings.
A paper should be *readable* by non experts but it doesn't meant they can *evaluate* it. I don't know how its in systems, but theory has many subareas where evaluating a paper requires specialized expertise that few posses (think lattice based crypto, arithmetic circuits, cell probe data structures, etc etc). A conflict of interest policy that prohibits recent collaborators will lose the most informed opinions, so a disclosure rather than prohibition based model is more appropriate for tcs. What part of this argument is arrogant or self serving?
ReplyDelete@Anon12:36pm: A conflict of interest policy that prohibits recent collaborators will lose the most informed opinions, so a disclosure rather than prohibition based model is more appropriate for tcs.
ReplyDeleteI think this is a bit of a strawman argument. No one has proposed a CoI model that "prohibits recent collaborators". The usual CoI models include same institution, advisor/advisee relationship, and so on. These can always be tweaked for TCS if the community of researchers working in an area is small enough that "collaborator in the last two years" rules out everyone. But there aren't that many cases of this kind of thing, and I'm saying that from personal experience soliciting reviews from people for papers.
Secondly, I'm sorry to say this, but it *is* arrogant to assume that the existence of complicated subareas is something unique to theoryCS. there's plenty of specialization in machine learning and databases (two areas I'm familiar with). And "systems" is an almost meaningless term: do you mean architecture, operating systems, networking, programming languages, verification, concurrency, databases, .....
Anon12:36pm -- I would have responded, but Suresh aptly did so for me.
ReplyDeleteI would add, though, that in my experience, the arrogance that goes with "my area is so difficult only a small number of specialists can properly evaluate it" correlates with the opinion "so many papers in my area are so good, they really should get in" -- a troublesome issue for (what should be) broad conferences like FOCS/STOC/SODA.
Michael's posts mention systems conferences implenting a "coauthor in last x years" (where x is sometimes 2, sometimes 5), and its unclear if he does or does not propose tcs adopt this rule. All tcs conferences I'm aware of have always considered advisor/advisee and family relations as a conflict which people left the room for. So, I'm not sure what exactly michael changed at stoc and whats the big debate about. (Maybe about the "same institution" rule?)
ReplyDeleteSecondly, I didn't claim theory is unique in having specialized subareas with few people inside each one. I just claim that whenever there is a situation like that, adding a "recent coauthor" conflict doesn't make sense.
P.s. The areas I mentioned are not my own: I just threw out lattice based crypto, arithmetic circuits, cell probe data structures as random examples of areas that I'd imagine a typical PC will have 0-2 members knowledgeable about, and you'd want them in the room for discussions.
ReplyDeleteBut the truth is that I'm not sure what's exactly the controversary about, if indeed nobody is proposing adding a "recent collaborator" conflict for theory PC's and subreviewers. (Though I think such collaborations should be disclosed.)
I'll take the liberty of saying that we're converging. If the main issue involved in installing CoI as part of expanding a PC is deciding on a 'recent author' rule, then we're pretty far along towards a workable solution. I agree that there are subareas of TCS that are (1) small and (2) technically dense, and that one has to be careful about recent-author rules so as not to disadvantage such areas.
ReplyDeleteSuresh, you raised several issues:
ReplyDelete1. Reduce the work of being on a large conference PC.
2. The prohibition of PC members submissions stands in the way. A two-tier PC is a solution and we can safely handle PC member submissions at large conferences by a suitable COI rule.
3. An all-electronic PC meeting would help.
Though the size of the PC has grown, the workload for a FOCS/STOC PC member has not changed dramatically since the first electronic-only PCs back in the mid-late 1990's. (Going to on-line reviews and interaction prior to decisions significantly increased the workload versus write-once reviews and in-person only PC meetings.)
The electronic-only FOCS/STOC PCs of the 1990's were pretty awful. It wasn't a software problem; it was a social problem. PC members disappeared from discussions. Accept/reject decisions stretched over days when some PC members could not participate. PC members did not have much basis for making accept/reject decisions for papers that were not in their purview because there was no incentive for overall discussion vs discussion between two or three experts. I therefore strongly object to suggestion 3.
SODA seems to be in a different place from FOCS/STOC. The workload has gotten very high and I have heard that the review process is quite broken. (It seems to suffer from some of the problems of electronic-only FOCS/STOC PCs.)
At a broad conference like FOCS/STOC and probably SODA, the core PC should be involved in an overall discussion of papers as opposed to a narrow area-based discussion. It is not just the "PC members can't submit" rule that pushes against increasing the size of this group. It is also having a group small enough that an overall discussion is not unwieldy.
A two-tier PC might work. I agree that we aren't that far from a second tier with our sub-reviewers. It seems that SODA might be the right place to start because it seems more out of hand. For STOC/FOCS, I think that we are on the edge - if the submission numbers grow we will be in trouble. I would reduce the size of the top tier from the current PC of 22-24 to 15 or so - big enough to have coverage in expertise but not necessarily volume.
With a two-tier PC I would keep the "PC members cannot submit" rule for the top tier. The cost is small and it frees the top-tier PC members a lot. It is common in other communities for the top tier PC to be similarly prohibited from submitting.
4. How would we actually organize the two tiers of the PC? The "area chair" format just doesn't seem to fit our conferences. How would the second tier PC be chosen?
I assume that the top tier PC would be involved along with the second tier PC in some number of reviews, since they are likely to be experts on many papers. I suspect that this would be at most half the current numbers that PC members handle.
I don't think that strict COI rules alone do the job. I prefer the CCC rules on COI: All conflicts must be disclosed clearly, not just what they are but how they are expected to impact a review. The review form suggests a range of potential conflicts that are not necessarily disqualifying. These include items such as "I am a good friend of" or "I have competing/closely related work" not usually covered by strict COI rules. When I want a review on a paper I often want to hear from somebody who has such a conflict, but it can be as much or more of a conflict than a co-author or institutional conflict.
BTW: At theory PC meetings advisor/advisees or people who are at the same institution are always expected to remain out of the discussion on (and certainly not make arguments for) a paper they are conflicted on. The only thing that Michael did differently was suggest that conflicted people not see reviews (not enforced) and require that the conflicted people leave the room during the PC discussion. The only situation where I would enforce a strict "no-communication" rule would be for an author.
Hi Paul
ReplyDeleteThanks for the detailed and thoughtful comment (which should really be a blog post on its own!). Others will have different views on this, but let me weigh in with my take on your questions:
Firstly, regarding electronic PC meetings, I am agnostic on this front: there are pros and cons to this format, and the reduced travel/cost is quite appealing, but I agree that things often reduce to a 'star' network of communication between the PC chairs and the pc members/area chairs/senior PC.
The idea of limiting the top-level PC to no submissions could work as only a small difference from our current model.
In regards to how the PC could be constructed, I've seen different models. one is the area chair model where the area chairs choose their sub-PC. another is the area chair model where the PC members are still chosen by the PC chairs. Finally, there's the 'vice chair' model where the vice chairs are not area-specific. In one conference where I'm a vice-chair, the PC members asked us to nominate 5 people each for potential inclusion on the PC. The vice-chair model might be best for theory conferences, so that there's still a holistic view of the submissions, rather than letting things get balkanized.
How the CoI rules are developed is indeed a matter of discussion, and customized for theory needs. The CCC model sounds reasonable, but I haven't thought deeply about the exact nature of CoI rules, to be honest.
Good discussion.
ReplyDeleteAs far as electronic PC meetings, I am not agnostic on this front - in my experience electronic meetings work better. In part this could be a personal thing - my first PC meeting experience in the early 2000s was electronic, and I think it worked well. My several experiences with SODA was also good.
My main gripe about physical PC meetings is that, invariably, they contain long stretches of time where everyone is pretty wiped out. This includes the time after meals, before the end of the day etc. During those times the quality of the discussion drops, and the decisions can become suboptimal. This is particularly true towards the end of the meetings.
Other negatives include the time and effort spent on physical travel, often cross-country. As the world becomes more and more linked, this seems harder and harder to justify.
For completeness: I do enjoy the personal aspects of physical meetings (meeting people, interactive discussions, etc). But I just think that the cons out-weight the pros.
I'm perfectly comfortable with a CoI model that "prohibits recent collaborators". If I can't convince people I haven't worked with that my work is interesting, novel, and correct, then I haven't done my job.
ReplyDeleteIs that a gun in my hand? Why is it suddenly aimed at my foot?!
But almost every American theorist with a PhD is *already* comfortable with the "no recent collaborators" COI model, because that's the *precise* model that NSF uses when it evaluates grant proposals. And yet NSF's funding decision are mostly sensible [citation needed].
Moreover, the NSF has a 4-year exclusion rule, as opposed to a 2 year rule that many conferences use. I sense in the objections a general chafing at "making us constrained in any way shape or form". And for those folks, it should be noted that all proposals would be adapted to the needs of theoryCS, not applied in a cookie cutter form.
ReplyDeleteI am so happy to read this. I am never a PC member, but soon to be. There is no this rule in my head (maybe in some corner of hard disk, but not in the memory). Maybe people, including myself, are always lazy to build a new possibly better system, because it takes much less work to just revise current system. Also, better always means more complicated. We have to balance fairness and efficiency.
ReplyDelete1. Generally, the ideal system I think.
There is an ideal and quite fair system after published, that is, good papers will be cited and noticed.
The ideal system is that, there is no submitting and reviewing, and every authors and readers can publish papers and reviews somehow freely. (Like blog? :) The authors and the refrees=readers are the same set. At this sense, I think, PC members should be every researcher, which are also latent authors. )
2. Some details of this system.
It needs rules about:
How to defined the upper bound function of frequency of publishing papers by an author.
How to balance the areas researched by a lot of people and a few people (they will not get
as many as readers and reviews. we can not simply say more good reviews means good paper).
How to distribute the probability of each paper that appears before readers.
How to organize the review of review, and the review of review of review, ...
How to discuss the system for building this system, and the system for building for building ...
How to design some index, shown as scores or curves, for a person as author and reader.
(We must be careful about this. If we simply say that a review which is similar to the majority reviews of this paper is a good one, this gives a simple cheating method. We shall also praise the review which is quite opposite to the previous ones but supported by the following ones.)
How to forbid cheating in this system, for example, a clique which always give
high scores only to its numbers. (One method for this, briefly, when a reader want to see
newest papers, he only get a random subset of newest papers. If a member want to
praise another member's paper soon after its appearance, some search operations are needed,
which is easy to detect. We analyze this kind of behavior by algorithms and design it as an index. )