I thought I would also post a few notes on the SODA business meeting. I am sure I missed some details, but here are the main points.
Everyone thought the organization of the conference was excellent (so far). The part in parenthesis is a joke by Kazuo Iwama towards his students - I guess that is Japanese humor, and encouragement.
Despite being outside of North America for the first time, the attendance was quite high, I think around 350 people. And the splits were almost exactly 1/3 NA, 1/3 Europe, 1/3 Asia.
Yuval Rabani talked about being PC for the conference. He said there were the most submissions ever, most accepted paper ever, and largest PC ever for SODA. Each PC member reviewers about 49 papers, and over 500 were sub-reviewed. We all thank Yuval for all of his hard work.
We voted next on location for 2014 (2013 is in New Orleans). The final votes were down to Honolulu, HI and Washington DC, where Honolulu won about 60 to 49. David Johnson said they would try to book a hotel in Honolulu if he could get hotel prices below 200 USD/night. A quick look on kayak.com made it appear several large hotels could be booked next year around this time for about 160 USD/night or so. Otherwise it will be in DC where the theory group at UMaryland (via David Mount) have stated they would help with local arrangements. They did a great job with SoCG a few years ago, but I heard many suggestions that it be held more downtown than by UofM campus. And also there were requests for good weather. We'll see what happens...
Finally, there was a discussion about how SODA is organized/governed. This discussion got quite lively. Bob Sedgewick led the discussion by providing a short series of slides outlining a rough plan for a "confederated SODA." I have linked to his slides. This could mean several things, for instance:
- Having ALENEX and ANALCO (and SODA) talks spread out over 4 days and intermixed even possibly in the same session (much like ESA).
- The PCs would stay separate most likely (although merging them was discussed, but this had less support).
- For SODA the PC could be made more hierarchical where there are, say, 6 main area chairs. Then each area chair supervises say 12 or so PC members. The general chair would coordinate and normalize all of the reviews, but otherwise it would be more hierarchical and partitioned. Then PC members in each area would have fewer papers to review, and could submit to other subareas even.
- There was also a suggestion that PC chairs / steering committee members have some SODA attendance requirements. (Currently it consists of David Johnson, 2 people appointed by SIAM, and the past two PC chairs - as I think I understand. David Johnson said he would provide a link to the official SODA bylaws somewhere.).
- Should the steering committee consider spreading ALENEX, ANALCO, and SODA talks over 4 days? About 50 to 7 in favor.
- Should the steering committee consider/explore some variant of the Confederated SODA model? About 50 to 2 in favor.
- Should the steering committee consider making the steering committee members elected? About 50 to 1 in favor.
Hawaii is most problematic for the Europeans, I'd think. It would be interesting to see how this affects them. Clearly Kyoto was well attended by SODA standards, so distance alone is not an issue.
ReplyDeleteI'm obviously in favor of the hierarchical model for choosing PCs, although I'm not convinced that a standing steering committee for each topic area is either necessary or even a good idea. Kudos to Bob S for pushing this through, with what looks like massive support.
The precise wording of the third vote (which was a small point of contention just before the vote) was "Should the SODA steering committee contain elected members?"
ReplyDeleteUnlike the other votes, there was no prefix "Should the steering committee consider...?", and despite David Johnson's last-second efforts, there was no suffix "...in addition to members appointed by SIGACT and SIAM."
One can make a case that having SODA in Kyoto, even if expensive, serves the purpose of making the conference international. How does it help to host it in Honolulu?
ReplyDeleteThanks to Jeff for an overview.
ReplyDeleteAs one of people responsible for the " very strong" and (hopefully) "well-argued sides", let me try to quickly summarize some of the opinions about the proposals. In short, there were three proposals:
A. A proposal by Cliff Stein, to have ALENEX, ANALCO and SODA talks scheduled during the same days, possibly during the same sessions. As many/most people in the audience, I believe this is an excellent idea, as this will expose broader SODA community to experimental/analytical approaches to algorithms analysis, with a strong potential of cross-fertilization.
B. A proposal by Bob Sedgewick to create a larger, hierarchical program committee. Several structures were discussed. On this issue,the devil is in the details. Many or most people I talked to during the conference believe that enlarging the committee and introducing a hierarchical management is a good idea (it should increase the quality of referring and enable accepting more papers, which seems appropriate given the ever increasing number of submissions).
At the same time, many people including myself were concerned about having *separate* sub-committees making *separate* acceptance decisions. The concern is that this can easily lead to separate "quotas" for different areas (de facto, if not formally), and uneven acceptance standards. The past experience with short SODA papers is a proof that this is not just a theoretical concern. See
this link (picture at the bottom) for more data driven analysis.
C. A proposal to elect some/all steering committee members (by Bob Sedgewick and other people in the audience).
That is more or less it. I will also post more suggestions (about stimulating and evaluating experimental research at SODA) later.
Piotr
I didn't attend SODA this year (and didn't submit a paper) because of the cost and the fact that Japan is still undergoing a nuclear disaster.
ReplyDeleteCan people who attended say how much going to SODA cost them? From Europe? From Chicago? From the east coast?
People should comment on whether the prohibitive cost kept them from going and thus voting for upcoming SODA location.
Continuing on the issue of of proposed tracks/subcomittees, I would like to focus on one example, namely the proposed experimental papers track (experimental papers is something I have some experience with, but I believe many comments apply to other proposed tracks as well). In short:
ReplyDelete- I believe that the lack of experimental papers at SODA is mostly due to the fact that few good experimental papers are submitted to SODA in the first place
- The best way to change this is to stimulate the supply by creating incentives. Some examples included.
Long version:
I believe most members of SODA community would agree, or at least not disagree, that having more good experimental papers at SODA would be a positive development. However, at present this is not happening: only one purely experimental paper has been accepted this year, despite "experimental" being featured twice in the call for papers.
The proposed experimental track assumes that this situation can be resolved by changing the way the papers are evaluated. There might be some truth to that - expertise in experimental research is a necessary (although not a sufficient) condition to evaluate an experimental paper. However, I would argue that the bigger problem is in the (lack of) *supply*. That is, on average, good experimental papers are not being submitted to SODA. Some reasons:
A. Writing a good experimental paper is hard, in my opinion harder than writing a good theory paper. Such papers need to satisfy many of the criteria of a typical theory paper (address an interesting problem, lead to algorithmic insights or algorithmic improvements) *as well as* more empirical criteria (e.g., meaningful experimental designs, proper data set choice and efficient coding). Writing such papers takes time and effort, in addition to ideas.
B. If a paper contains significant algorithmic improvements of empirical nature, with some extra effort it can often be accepted to a good applied conference. In 2011 alone, members of SODA community had papers in applied conferences such as SIGMOD, VLDB, ICDE, PODS, KDD, ICDM, SDM, WWW, WSDM, NAACL,SIGCOMM, INFOCOM, Allerton, and more. A paper presented at an applied conference can reach hundreds of researchers interested in the topic, as opposed to perhaps a few dozen at SODA.
This said, there are certainly experimental papers for which SODA is a natural choice (e.g., if a problem is general enough so that tuning the algorithm to a particular application would be a distraction, or if an algorithm is a heuristic searching for an explanation or insight, etc). However, the aforementioned factors A and B significantly limit the supply of good experimental submissions to SODA.
To change the status quo we need incentives. We could for example:
- Create the Best Experimental Paper Award. Any SODA paper with an experimental evaluation would be eligible, although the paper would be evaluated based on both experimental and algorithmic qualities.
- In the same vein, dedicate a special issue of a journal to invited experimental papers
I am sure that many other approaches exist.
Piotr, I believe that the JEA does this with ALENEX (that is, invite some papers to a special issue). I could be mistaken though.
ReplyDeleteESA has a separate algorithm engineering track which I've been told is a success. However, I feel that SODA's main problem is not how to attract more papers, but how to deal with the existing and increasing submission numbers. We need to increase the PC size. How do we recruit a larger committee? (Even the present size is hard to recruit.) How does the PC chair manage a larger committee? (Even the current state is challenging.) ESA has significantly smaller submission numbers. Wth SODA's numbers I don't see how setting a separate experimental track would make sense, even if experimental submissions increase ten-fold.
ReplyDeleteIn the business meeting I proposed holding SODA twice a year (with two parallel sessions instead of three each time). ESA may have grown to fulfill the need, but with its split into separate theoretical and applied tracks it seems to be moving away from alleviating the pressure on SODA. If SODA moves in the same direction, it will make our real problems worse, not better,
If one looks at other venues, our PCs seem unusually small. It's possible to make PCs larger but we'd have to move to a more hierarchical model in which PC chairs only really interact with area chairs and so on. We might not want to go that route of course but it is an established model.
DeleteAs far as I see it, the main problem with SODA's PC is not how to attract more submissions, but how to deal with the existing and growing submission numbers. We need a larger committee. How do we recruit it? The current size is already hard to recruit. How does the PC chair manage a larger committee? Current numbers are already challenging. We need perhaps additional talk slots. How do we schedule them? We definitely do not need additional plenary talks. Beyond 2-3 they'll become an annoyance.
ReplyDeleteESA has a separate engineering/experimental track, but it does not have SODA's submission numbers. In SODA I can't find justification for such a track even if experimental submissions increase ten-fold.
In the business meeting I suggested holding SODA twice a year (smaller conference each time). ESA could have served this purpose, but with its split into two tracks, it's moving away from alleviating the pressure on SODA. If SODA moves in the same direction, this will make the problem worse, not better,
Why don't we just put our "submitted" papers on the arxiv and let people vote for the papers that they would want to hear a talk for?
ReplyDeleteI was on the SODA PC a few years ago and with > 50 papers, it was a huge time commitment. Even when I sub-refereed a paper, I didn't feel comfortable just cutting and pasting the review--I wanted to make sure that I agreed with the review and had to read the papers to do that.
Also, we all know the frustration of getting back a review where the reviewer clearly doesn't understand the paper and doesn't care ("This problem is already solved .."). While it is the author's responsibility to convey his ideas clearly, there is still much bad reviewing, maybe because the (overloaded) PC member didn't make a good choice. This could be alleviated by crowdsourcing.
I think the only reason we are not doing something like this is because people who get asked to be on PCs all the time like their power ...
Hierarchical models are good to explore. E.g., the papers could be assigned to the reviewers in a hierarchical, top-down manner; same with leading the discussions.
ReplyDeleteBut we should also ensure good "information flow" across the hierarchy. In particular, we should keep the TCS tradition that all PC members (not only those assigned to papers) can evaluate and comment on the papers, subject to COIs. For example, an experimental graph paper can and should be evaluated by both experimentalists and experts on graph algorithms, wherever in the PC hierarchy they are placed. Otherwise we risk decreasing the quality of the reviewing process, as well as creating scoring inconsistencies between non-communicating clusters.
About the "2 SODAs" approach, the problem is that this would likely split the attendance - many or most people would not be able to attend twice per year (the attendance tends to depend on the number of accepted papers). This would decrease the attractiveness of the conference - there would be fewer people to meet. At the same time, it would indeed provide a simple load reduction mechanism.
Otherwise we risk decreasing the quality of the reviewing process, as well as creating scoring inconsistencies between non-communicating clusters.
ReplyDeleteCouldn't agree more. this is definitely a problem in hierarchical models, and software can allow us to solve this by allowing more access.