Monday, June 21, 2010

On acceptance rates and flagship conferences

There's been a lot of back and forth on ways of increasing attendance at STOC, and in our wonderful theory way, all of this has happened in a universe unencumbered by the presentation of actual data.

I thought I'd dig up statistics on what exactly goes on at a number of major conferences in different areas in computer science. My idea was to take some of the major areas in the field, identify their flagship conference (or one of two as the case may be), and compile statistics on acceptance rates, attendance, and general conference activities.

The areas I considered (with main conference in parentheses) were
  • databases (SIGMOD)
  • machine learning (ICML)
  • operating systems (SOSP)
  • networking (SIGCOMM)
  • architecture (ISCA)
  • graphics (SIGGRAPH)
and the results I got were interesting (all data that I compiled can be found in this spreadsheet: feel free to add other areas, or update numbers, as you see fit). Where I could, I tried to get a sense of attendance/acceptance rates from either asking people or looking at published numbers for recent years: the ACM DL has acceptance rates for many of the above. Information on conference activities were taken from the most recent year I could get data for (usually 2010 or 2009). The main points:
  1. All of the listed conferences had attendance in the 500-600 range (except ISCA with average attendance of 400, and SIGGRAPH with 2000+ in the research side). So they are definitely conferences with attendance that STOC would like to mimic. 
  2. Acceptance rates varied, but most were below 20% (ICML being the exception at 25%). STOC is at 28% or so
  3. Number of papers accepted varied widely (23 for SOSP, 150 for ICML). I find this particularly interesting: it would seem that attendance correlates more with the perception of being 'flagship' than the actual number of papers accepted.
  4. Most conferences had long lists of colocated workshops. The smallest number was SIGCOMM last year with 5, and others had many more. STOC had none.
  5. Tutorials were variable: some places had many (SIGGRAPH had 27) and some had none. STOC had 3.
  6. With the exception of ISCA last year, all the conferences had significant poster sessions, either consisting of all papers accepted, or as a separate track with many posters. STOC had none.
  7. The conferences all had other activities: demos, industrial tracks, works in progress or other such things (ISCA being the exception). STOC had none. 
  8. Durations varied between 4 and 6 days (including the initial day). Most had 5. STOC is 4.
To me, there are two things that stand out from this.
  1. The number of papers accepted does not appear to make a difference to the attendance. SOSP happens once every two years, and accepts 23-25 papers, and gets 500 attendees !! ICML gets a similar number of attendees with 150 papers accepted each year. 
  2. There are a TON of activities at these conferences. Indeed, I think ICALP and ESA match them in terms of level of activity, but certainly not STOC. I've been a proponent of satellite events around a conference to increase attendance, and the STOC/EC/CCC colocation does seem to have helped. I'm also intrigued by the idea of colocating SoCG with STOC. 
You may draw your own conclusions...

p.s for the legions of readers who will start protesting that these communities are much larger than the theory community, I will merely point out that almost no one in this discussion thinks that the theory community is 300 strong: the question is more about getting the rather large theory community to show up in force for STOC.

UpdateMichael Mitzenmacher has a post up listing specific logistical issues that come up with expanding the set of activities at a conference. He points out that if we decide to go to multiple satellite events (whether as separate events or whatever), we'll have to adjust to a much greater degree of organizational commitment up front, as well no small amount of 'attitude adjustment'. For anyone who's ever attended a business meeting, this is a scary thought :)

10 comments:

  1. Suresh -- I think building up STOC with more "satellite events' is quite an interesting idea. I just think it's a daunting one. See my related post at:
    http://mybiasedcoin.blogspot.com/2010/06/colocating-conferences-what-will-it.html

    My take is theory, at least historically, has lacked the organization (and people willing to get their hands dirty with, well, the dirty work) for what you suggest. But perhaps that can change....

    ReplyDelete
  2. Re: "SOSP happens once every two years, and accepts 23-25 papers, and gets 500 attendees!!"

    I think most systems folks think of SOSP and OSDI as the dual flagship conferences of the field. They are held in alternating years, are about the same size, and draw pretty much the same crowd. So, "SOSP happens once every two years" is a bit of a red herring, FWIW.

    ReplyDelete
  3. @Chris
    I thought of that when I was writing that line. But then I reasoned that even if OSDI accepts 100 papers each year, the average number of papers across the union of the two is still very small.

    as an aside, are you aware of how many papers OSDI accepts each time ?

    ReplyDelete
  4. Hi Suresh,

    Just a few things I think we should bear in mind in interpreting these attendance data :

    1. Conferences in applied areas typically draw a lot more people from the industry, who are more engineers rather than researchers. There's no such pool of folks for FOCS/STOC to tap into.

    2. Theory folks typically have less travel funding. (As a student, there's been more than one occasion where I struggled to get funding to present papers.)

    3. There's some sense in which the atmosphere at FOCS/STOC is not very friendly towards people who don't have papers at the conference.

    ReplyDelete
  5. hoeteck: on the matter of applied research, I would have agreed with you 10 years ago, but not now. Consider that Google and Yahoo are full of ad auction folk who would easily be induced to attend a workshop at STOC on practical issues in auction design. The same for privacy matters (differential specially), and also for crypto/security (although there's already a thriving community there).

    And we should think beyond applied areas. Why can't we attract more mathematicians (say) to a workshop on 'graphs and geometry', or 'additive combinatorics' and so on.

    ReplyDelete
  6. Shouldn't AAAI, IJCAI and NIPS be included in the "AI" line of the spreadsheet? The acceptance rate for AAAI is apparently around 25%: http://www.aaai.org/Conferences/AAAI/aaai.php .

    ReplyDelete
  7. Suresh, thanks ! Perhaps it would make sense to add FOCS data as well ?

    In fact, FOCS/STOC serve as a de facto single conference occurring twice per year, so we should probably treat them jointly. I am not sure how to estimate the number of distinct elements attending STOC/FOCS though...

    ReplyDelete
  8. Warren, Piotr: AAAI/IJCAI are extremely broad conferences - I'm not sure they're in the same class of conferences as STOC/FOCS, or even ICML. I could definitely have added NIPS in, but I was trying to be representative rather than exhaustive (applies to the FOCS/OSDI issue as well)

    ReplyDelete
  9. From Chris Lesniewski-Laas, whose comment I accidentally deleted:

    OSDI accepts approximately the same number of papers as SOSP. It accepted 26 papers last time, I believe. I don't think this really affects your main point; I'm just proposing a more apples-to-apples comparison.

    ReplyDelete
  10. For what it is worth, here is my experience with the organization of ICALP 2008 in Reykjavik, which I believe is still the most attended and event-packed ICALP conference on record.

    First of all, even without satellite events, ICALP is a three-track conference, though typically only two tracks are running in parallel at each point in time. At ICALP 2008, in addition we had 11 satellite events, including the DYNAMO training school for doctoral students. Overall, nearly 500 people attended the main conference and/or the satellite events.

    Organizing such a conference was not an easy job, but it was not as daunting as it may seem. In hindsight, I think that it was important to organize the event at the university (not a hotel---in passing, I very much prefer attending events held at universities rather than hotels), to have the support of the university support services, of some local students and postdocs, and of experienced conference organizers who took care of the registrations, of the lunches and coffee breaks and of the social programme.

    Overall, I do not think that this ended up being much more work than organizing a single-track conference without satellite events.

    ReplyDelete

Disqus for The Geomblog