NIPS Reviewing

This year, NIPS received over 2400 submissions. That's -- well --- a lot!

As a reviewer, I have been assigned 7 papers (note that this number will be utterly incomprehensible to theoryCS PC members who think that 30 papers is a refreshingly low load).

But none of that is as interesting as what NIPS is trying this year. From the PC Chairs:

New this year, we ask you to give multiple scores for technical quality, novelty, impact, clarity, etc. instead of a single global score. In the text boxes, please justify clearly all these scores: your explanations are essential to the ACs to render and substantiate their decision and to the authors to improve their papers.

Specifically, the categories are:

• Technical quality
• Novelty/originality
• Impact/usefulness
• Clarity and presentation

and there are also a few qualitative categories (including the actual report). Each of the numerical categories are on a 1-5 scale, with 3 being "good enough".

I've long felt that asking individual reviewers to make an accept/reject judgement is a little pointless because we lack the perspective to make what is really a zero-sum holistic judgement (at least outside the top few and the long tail). Introducing this multidimensional score might make things a little more interesting.

But I pity the fate of the poor area chairs :).