- "I go to meet people, not attend talks"
- "All the interesting conversations happen in the hallways"
- "Talks are too difficult to follow: I'd rather read the paper or just ask the author"
- "I read this paper 6 months ago when it appeared on the arxiv: it's old news now, and there are improvements"
But there's a certain 'band-aid on a bleeding carcass' aspect to this discussion. Indeed, between overloaded reviewers, authors who need the imprimatur of a prestigious conference, and registration fees that skyrocket as meetings get longer, it almost seems like this system is heading for a nervous breakdown.
But there are a number of experiments in play that point the way towards a gentler, kinder conference system (even if we decide not to grow up). In this G+ discussion, Fernando Pereira and Zach Ives describe two models that put together address the main problems with our conference process.
NIPS receives over 1400 submissions, and accepts a small fraction (
- It removes artificial limits on number of papers accepted based on conference duration. Posters are presented in (semi)-parallel.
- It eliminates the "20-minutes of droning with no questions" style of many conference talks. Posters are a much more interactive way of presenting material, and it's easier to skim papers, talk to the authors, and have a good discussion. The papers are still in the proceedings, so you can always "read the paper" if you want. As an aside, it really helps with communication skills if you have to answer questions on the fly.
- There isn't a huge burst of submissions, followed by a draining review process. Reviews are spread out over the year. Moreover, area chairs are used to partition papers further, so any (P)VLDB reviewer only gets a few papers to review each month. This can only improve the quality of reviews.
- The journal-style back-and-forth makes papers better. Authors can make changes as recommended, rather than trying to defend their choices in an often-contentious rebuttal process.
One complaint I have heard from friends in databases is that the VLDB system with decisions at the end of each month is that because the number of submissions is much less each month, it is easier to play politics to get a paper in, or keep a paper out. Submit a paper in a slow month, if you have connections to get it in easily. If you are trying to keep a paper out, since there are fewer papers to compare against, its easier to keep good papers out if desired because there are fewer possible apt comparisons to other weaker papers also accepted.
ReplyDeleteHopefully, this does not happen often. And hopefully the politics are not as bad in algorithms/geometry that this would almost never happen. But thought I would mention the concern. Otherwise, I think it sounds like a great idea.
The link to the G+ discussion gives me a 404.
ReplyDeleteYou ask "You have to wonder why anyone gives talks at a conference any more ! "
ReplyDeleteWhy isn't "meeting people face to face" a good reason ? The current conference format seem to be a proxy for some sort of stamp of quality. It may be or not the case, the argument remains that people need to have "excuses" to get away from the day to day grind of university politics.
Igor.
One problem with poster presentations is the number of people and the size of the hall. Based on my own experience (sample size 1), it is sometimes difficult or impossible to have any meaningful discussions with the authors: i) you have to time yourself so you don't miss the summary that authors often give, ii) you have to position yourself close if you want any of your questions to be heard (not easy if a particular poster is popular) and iii) most annoyingly, if the hall is not big enough, you get a headache by raising your voice sky-high to be heard!
ReplyDeleteThat said, I also agree with the criticism of the 20 min. presentation format!
Would you characterize the recent poster experiment at STOC as successful? I doubt anyone would, really.
ReplyDeleteIt's not a priori clear that the poster system is a good fit for TCS. We could come up with purely theoretical reasons why it should or should not be, but ultimately you need a few successful experiments before anyone would agree to it.
-mihai
Maya: I've been to quite a few poster sessions at data mining conferences (and at some cross-disciplenary events). Most of the time I found it easy to talk to the authors who attended their own posters. Sometimes very junior students will try to dodge questions, but its rarely worse than in the conference talks. Crowds around each poster grow and shrink, so generally if you are willing to wait a few minutes you have no problem talking with the author.
ReplyDeleteAlso, having the poster session concurrent with snacks and beverages leads to a nice crowd and often as good or better than conversations in hallways.
The main disadvantage I have found, is that some authors feel that attending their own poster is optional. And a poorly-made poster (especially unattended) can somehow be even more useless than a poor talk. But at least escaping is not an issue.
A negative consequence of highly selective conferences is reviewer overload with an ensuing drop in review quality.
ReplyDeleteThis is a self-feeding process. The more papers incorrectly rejected, the more resubmissions we have and the more reviews we need.
Think back twenty years ago and let's track a hypothetical SODA/STOC/FOCS worthy result.
In this scenario, the paper is accepted with high probability thus consuming a total of 3 reviews.
On the other hand if the paper was rejected but with high quality negative reviews you use them to re-calibrate your submission, fix any errors and resubmit either to STOC/FOCS/SODA or a more apropriate conference down the ladder. Either way the paper now likely gets in for a total of 6 reviews.
This used to be the dominant case. Contrast this with today, with much higher rejection rates *and* much lower quality reviews.
Nowadays a single negative review is enough for rejection, and since reviews are of much lower quality, the probabilities of an _incorrect_ negative review are substantially increased. At the same time, due to time demands, the other reviewer comments provide no useful feedback.
The paper is then resubmitted essentially in the same form to STOC/FOCS/SODA where it gets rejected--again-- with another incorrect negative review--again--.
Since there is so little actionable feedback, you move epsilon down the scale and resubmit to a conference a notch below which by now has also become ridiculously competitive so chances of rejection of your rather nice result are non zero either. In this case you may have to resubmit to a fourth conference.
In all the number of reviews required has gone up from somewhere between 3 and 6, to somewhere between 6 and 12.
Observe that the same timeline applies to a lesser quality paper, but starting from an initial submission to a more modest conference.
The end result is that we have nearly doubled the reviewer load while simultaneously decreasing the quality of said reviews.
This is a worst of both worlds outcome.
@mihai You are right that it's not a priori clear that the poster format works for TCS. Having said that, I've heard pretty much every single innovation in the conference process (double blind reviews, satellite events, journal-style rebuttals etc) dismissed as "not working in the TCS format", so I'm generally skeptical of this claim without evidence.
ReplyDeleteIn that sense, I'd argue that the poster session at STOC 2011 WAS a success. By all accounts it was well attended, and there were a number of interesting posters (I'm a little biased, since I'm running the STOC 2012 poster session). I'll fully admit to not knowing the right way forward, but I think experimentation along the lines of what other disciplines are doing is healthy and we should do more of it. I'm particularly excited by the new "solicited workshops" call for STOC 2012 in addition to the poster session.
@igor I completely agree that "meeting people face to face" is a good reason to have conferences. But is it a good reason to "have a talk at a conference" ? Any kind of structured interaction (like posters etc) will serve the 'meeting people' purpose and will not eat into time constraints.
ReplyDeleteI'm looking forward to seeing how the STOC 2012 poster session changes from the 2011 version. I found the 2011 poster session to be fun. One comment however: as difficult as it can be to be stuck in a bad talk, I assure you getting caught talking to a bad presenter in a poster session is far worse. When the author of a poster launches into the proof of the first lemma, presuming that you of course want to see all twelve lemmas in full detail that constitute the main theorem, it is very difficult to politely excuse yourself to visit another poster -- especially if you are the only one being held captive at that poster.
ReplyDeleteI'd welcome suggestions on ways to make the 2012 session even more fun. As for the awkwardness issue, I usually stall that by merely reading the poster before even engaging with the speaker. It's a little awkward to be sure, but no more so than the deathly silence in a room after a talk has completed and the session chair asks for questions
ReplyDeleteI hear from people who reviewed for PVLDB last year (the first year when only the journal version ran, instead of the hybrid conference+journal) that there were unintended issues coming up. The root problem was that close to 50% of the papers were submitted in a single month - Febr. 2011. That's b/c March 1st was the last deadline for VLDB 2011: accepted papers submitted prior to March 1st 2011 will be (very shortly) presented at VLDB 2011; those submitted after March 1st will be presented at VLDB 2012. No one wants to wait a year to present their paper in a conference, so I'm guessing that the submission rate was close to zero for the month of March 2011.
ReplyDeleteSo most authors still treated the journal system as a conference system (myself included). The reviewers still got overloaded with papers in Febr. - albeit only about half the usual number. But they also had the additional problem that for papers submitted much earlier in the year, they had no significant selection to compare against. From what I hear, some reviewers were keeping their percentage of accepted papers below the typical VLDB average, in expectation of Febr. 2011 seeing a deluge of good submissions.
Of course, this issue is tightly related to the fact that VLDB has a classical 30-min.(!) talk format, which puts a hard upper bound on the number of accepted papers. So politics aside, reviewers hedge their bets year-round about the quantity and quality of future submissions, knowing they will come up against this upper bound. Quite a guessing game!
Of course, replacing talks by poster sessions would alleviate the upper bound problem. But frankly, few posters convey a clear message, and engaging their presenter can be a two-edged sword, as someone else mentioned.
My favorite solution for now is shorter talks. SIGMOD moved to 20-minute talks this year, and asked us to vote on 15-minute talks. I think this is a good idea: it will force presenters to skip tedious details and focus on one or two ideas they want to convey. And if the talk is still irredeemably bad, at least it's short!
@Magda, this is a very useful update. It's true that a shorter talk duration will alleviate some of the problems associated with the time-constraint-based limitation on the conference acceptance rate.
ReplyDeleteI wonder though if authors would prefer a 15 minute presentation to a (potentially multi-hour) poster session. I understand that for the audience, it's more awkward at a poster, but for an author it might be more satisfying. Clearly there's some sweet spot here. For example, KDD has short talks AND a poster session.
I personally don't find poster sessions very compelling. I don't submit to them myself. I dutifully go to the poster sessions of conferences that have them but I don't get much more from most posters than I would from a booklet of abstracts. And I don't find the opportunity to force the authors to be there and interact of much use. So I don't see this as a helpful solution to our "too much research to take it all in with longer talks" problem.
ReplyDeleteAs for Alex's claim that in the good old days papers used to get three good reviews from their first rejection, enough to calibrate and improve the paper so that it only needed one more submission: maybe he is remembering different good old days than I do. In the ones I remember, you were lucky if you got anything at all from the reviewers beyond the single bit of information that your paper was in or out. You had to personally know and pester the PC chair in order to get a hint of why your paper was even rejected. Our conferences have shifted from a default of having all PC comments "for PC eyes only" to a default of having almost everything go back to the authors, and that's a good thing.
@David, @Alex, I agree with David's assessment of reviews. conference reviews are dramatically better now than they used to be. A review I received from WADS this year was a work of art in itself.
ReplyDeleteHowever, David, I will observe that you are somewhat of an outlier in that you attend lots of talks in conferences, pay close attention to them, and often solve open problems that the speakers pose during the talk !! I think the average conference attendee (myself at the top of the list) is far less diligent than you are.
I agree with David that nowadays we get more comments, particularly as compared to before electronic conference review software was available.
ReplyDeleteHowever, when looking at the least informed of the three reviews my overall impression is that this has gone down in quality, and that this is due mostly to reviewer overload.
These are isolated data points, but at a couple of recent conferences in which I was PC member we ended up with an unusually low load of 10 papers or so per PC member. The reviews and ensuing discussion were unusual by the depth and familiarity with the papers shown by the PC members.
I don't think I'm saying anything particularly shocking here: give people more time to do their job and they'll do a better job.
At WADS we were talking about how come SODA doesn't pass on their reviews to LATIN and STACS which are the usual recipients of SODA resubmissions?
I enjoyed STOC'11 poster session. You can get a lot from staring at the posters, especially if the authors put pictures on them. Though I think I would have enjoyed the session even more if I the posters described the actual accepted papers (i.e., we had a "standard" poster session).
ReplyDeleteHow about mechanisms for increasing the quality of talks? Seriously. A good talk can explain the high-level idea of a paper in 20 minutes, whereas reading the paper could easily take three times as long.
ReplyDeleteAt the last conference I attended, the suggestion was floated that we should move away from the model where, by default, the least senior author gives the talk for a paper. The idea was that more senor authors will, in general, give better talks. I'm not saying this would be a good idea, but seems at least a step toward addressing the issues you raise.
There is something to that. Events where senior folks give talks are often more enjoyable, when the speakers know how to hide details and emphasize the key concepts.
ReplyDeleteMaybe like we review papers, we should review proposals for speakers :)
EuroCG has had SIGGRAPH-style fast-forward sessions the past two years. It's a short session held before a coffee break where each speaker gets 60 seconds and 1-2 slides to promote the talk he/she will be giving in the next session.
ReplyDeleteIt seems to be fairly useful in picking a high-quality subset of talks to attend. Hopefully it also incentivizes the speakers to "step up their game". I personally feel that talk quality has gone up from EuroCG 2010 to EuroCG 2011. Then again, there could be many other reasons for that, not in the least that the time slots for talks have gone down from 20 to 15 minutes.