Wednesday, August 16, 2006

SIGGRAPH, hiring, and peer review.

There's been an interesting flareup in the graphics community prompted by the posting of a note by Michael Ashikhmin on his web page. Ashikhmin is a known graphics researcher (with SIGGRAPH papers to his credit) who recently (June this year) proclaimed his desire to leave the graphics community because of
my deep disgust for the state of affairs within computer graphics research community and my inability to fit well within existing system
His grievances are many, and appear to be a direct consequence of the hegemonic nature of SIGGRAPH, by far the most prestigious conference in graphics. Specifically, he argues that all the usual problems with conference reviewing (extreme subjectiveness, poor quality of the reviews, clique-formation) are exacerbated by the overwhelming influence SIGGRAPH papers have on one's career (being hired, advancing in one's career, getting tenure, etc).

None of the objections are particularly novel; indeed, I have yet to go to a theory conference where people have NOT complained about the reviewers. However, what takes this beyond just your ordinary embittered-researcher-rant is that many prominent researchers in graphics appear to publicly both agree with him.

Michael says in his letter that senior graphics researchers recommended that he host a forum devoted to discussing this issue, and once he set the forum up, many well known and respected graphics researchers (John Hart, Marc Levoy, and Jonathan Shewchuk among others), commented publicly on the matter. In fact, Marc Levoy (who's the papers chair for SIGGRAPH 2007) went further and organized a town hall meeting at SIGGRAPH 2006 to discuss paper reviewing procedures (I don't know what transpired there).

There are many comments on the forum from anonymous commenters who claim to be published authors at SIGGRAPH. As far as I can tell, not one person disagrees with the primary claims that Michael makes, although Marc does attempt to mount a defense of the paper review process, while still acknowledging the main problems, and outlining strategies that he will employ in 2007 to fix some of them.

Many good suggestions were made. One of the primary ones was to add a new SIGGRAPH-like conference so that one venue didn't have to take all the load (STOC and FOCS were cited favorably here). Prohibiting committee members from submitting was another idea (again, the theory community was cited), although this was frowned upon by Marc Levoy, who complained that he wouldn't be able to find people for a committee (he did aver that he wouldn't be submitting anything).

This is probably the first time I've seen this kind of discussion take place (partly) on the record without dismissing the complaint as sour grapes. The question of course is whether anything will come of it in the long term. It's worth mentioning that even in our conservative (by nature, not by politics) research world, change can happen, and can often happen rapidly. Witness the collective revolt by academicians of all stripes against Elsevier, and closer to home, consider the split in ICRA (one of the main robotics conferences) to form RSS (Robotics: Science and Systems).



  1. Of course, Marc Levoy is the mother of all cliques in the graphics community. Letting him lead the discussion on it is more or less like letting the wolf guard the chickens.

    Poor chickens - not only they are going to be eaten alive, but the wolf is also going to lecture them on his good intentions before that.

    Posted by Anonymous

  2. There is nothing wrong with having a conference such as siggraph that emphasizes novelty over incremental results. Incremental results can be submitted elsewhere. 

    Posted by Anonymous

  3. There is nothing wrong with having a conference such as siggraph that emphasizes novelty over incremental results. 

    Big names in the community bitterly complain about it and yet "there is nothing wrong" with it? I'd posit that the mere fact that so many people are unsatisfied with it proves your statement wrong.  

    Posted by anon

  4. There is nothing wrong with having a conference such as siggraph that emphasizes novelty over incremental results.  

    In the discussion it was pointed out that as a field matures most results (good or bad) are all incremental, and refereeing boils down to judging how big the increment is. Good papers have large increments, mediocre papers have small increments.

    Yet, according to the discussion currently in SIGGRAPH an important increment on a fundamental problem has less chance of getting accepted that a solution to a totally new, made-up problem.

    Fortunately I don't think this is the case in the main theory conferences but we should be vigilant about it, particularly when some start to equate "incremental" with "mediocre". 

    Posted by anon

  5. I don't think these discussions usually take place for a very good reason, and you hint to it: nothing forces you to publish to a particular place. Heck, you can create your very own conference if you want to. It is not that hard.

    The problem, naturally, is that Ph.D. students have to live with whatever gold standards have been set for them... but more mature researchers, that is, employed researchers, should not feel like they have to publish at one particular conference. As I often say, let's encourage people to take risks.

    If people y need specific things on their c.v.s for grant applications, then, we need to go back and fix the grant system. If grant agencies become too conversative, we will soon have other problems in any case.

    I think that what we want to avoid is stressing too much the rigid tier 1 conference system. It is a convenient system because you can judge the value of a paper without understanding it. Or can you, really? If people have worked for 20 years to get at O(n^1.432432) and I get to O(n^1.431432), maybe it is not exciting enough for the big conference, but maybe, somewhere in there, in the way I did it, there is a deep, important insight to be found. On the other, maybe this totally made up problem could open a new field. We have to avoid being rigid in our we attribute value to a paper.

    In fact, with the rising cost of travel, we should be looking at more varied publication types and get back to the roots of the problem. What we want to achieve are "indexed research results". We all want to be read. That is all. Yes, there is a lot of crap and we need filters, but rigid, rule-based, filters are not going to work well, ever.

    I can do a lot of filtering on my own. Thank you. Just let me get at the papers.

    Posted by Daniel Lemire

  6. Fortunately I don't think this is the case in the main theory conferences but we should be vigilant about it, 

    this is partially true: we don't have the exact SIGGRAPH-style biases. But we do have our own blind spots (Lance wrote an article about this a while back, on how to get a paper into STOC/FOCS), and vigilance about one's own blind spots is always a good thing. 

    Posted by Suresh


Disqus for The Geomblog