Beki Grinter

CSCW Experiments: What is the Role of Computing History in Changing Acceptance Rates?

In academia, computer science, discipline, research on February 16, 2012 at 2:32 pm

I just came back from the CSCW 2012 conference. This year, the conference tried an experiment. The full details are beyond the scope of this post, but in simple form the conference conducted a multiple round reviewing process. Papers were submitted and then reviewed. Some papers were accepted after this round, others rejected, and a third group were selected for further revision (based on the reviews). These papers were then resubmitted and re-reviewed. A good number of that third category were accepted at the end of that second cycle into the final archival proceedings.

This experiment has raised concerns, one focused on the acceptance rate for the conference. Specifically, the concern was that a conference with an acceptance rate as high as that signals a conference that does not set the quality threshold appropriately. In defense of this increased acceptance rate was that it came out of a process that affected the quality of the final submission. Many of those ultimately accepted papers went through a process akin to a journal, and thus the quality was improved (and in the eyes of the reviewers and Associate Chair who looked at the revised version). At the TownHall meeting this led to a question of how we communicate quality if we do not (or perhaps more accurately can not) use the acceptance rate.

In the course of that discussion, one person observed that the field of Computing has spent a lot of time on campuses educating our peers in various other fields of the reason that Computing emphasizes conference publication. One of the historical arguments we’ve used is to show that acceptance rates (particularly lower ones) as that sign that conferences do discriminate. This made me think about another experiment that CSCW has just undertaken, one that I did not hear discussed at the Town Hall.

CSCW has separated itself in terms of how it chooses to assess quality from the rest of the field of Computing. Computing has collectively argued (and it has taken a while to educate people why this makes sense for this field) that low acceptance rates are a hallmark of quality in our conferences. While that measure might not be the right measure (I don’t personally believe it is) but it is the measure that is used, and its been used by an entire field in advocating for that field. Now, one—and only one—conference has changed its approach. What are the implications of that? Particularly for a conference that had, until this year, the sustained marker of “high quality” a relatively low conference acceptance rate.

This seems like a really important open question. One reason I think it matters is the risk that it presents for the very newest people in the discipline, particularly the ones who want to go into jobs where those acceptance rates will be examined (e.g., through the U.S. academic tenure process, at some Industrial Research Labs). What responsibility do we, more senior people in the field, have with regard to the protection and promotion of our newer colleagues, and what are the implications of having a process that is not entirely recognizable to our computing colleagues let alone others outside the discipline?

  1. This is an important issue, and one closely related to the issue of impact that was articulated so well by Judy Olson during her Athena Award Lecture as well as the special session in which you, Leysia Palen and Paul Dourish participated celebrating the Impact of CSCW: Looking back 10 and 20 years.

    I don’t have any useful suggestions to offer, but can offer a little more context about this change: Jonathan Grudin’s February 2011 CACM article on Technology, Conferences and Community: Considering the impact and implications of changes in scholarly communication – which is freely available (unlike Tessa Lau’s earlier blog post on Rethinking the systems review process [grumble] – and my own blog post offering more details, as well as some reflections and projections, on Innovation, research & reviewing: Revise & resubmit vs. rebut for CSCW 2012.

    CSCW 2012 broke several records: highest number of submissions, highest number of accepted papers and highest number of attendees. I think it will be very interesting to see what kind of submission, acceptance and attendance rates we will see next year. Meanwhile, I hope this discussion will continue, as it clearly impacts a significant number of people that I hope will continue participating in our community.

  2. While selectivity can be a virtue for conferences in ensuring work of a certain quality, there are dangers which arise from over-selectivity, as well.

    At WSDM this year, Andrew Tomkins made a great point that when acceptance rates dip below a certain rate, reviewers change their frame of reference: specifically, instead of looking for interesting papers to accept, they look for reasons to reject. Thus, ‘bulletproof’ work which is less groundbreaking will be favored over exciting work with small methodological or other problems.

    While I have read a great deal of work from CSCW, this was my first time attending. I would be curious about the perceptions of attendees of the quality, diversity, and novelty of research as compared to previous years.

    -Sanjay Kairam, Stanford University (@skairam)

  3. I think we must move away from regarding papers in conferences with lower acceptance rates as by default better than papers in conferences with higher acceptance rates. The test is time — what impact does the work have on the field? That cannot be judged by the acceptance rate at the time the work appeared. When I write letters for people up for tenure or promotion, that’s what I look for.

  4. I’m the kind busy explaining the role and importance of conferences in Computing to journal-oriented academics or people from other disciplines.The new CSCW review process somehow resembles those of journals. My experience shows that acceptance rates are not that useful for persuasion. Perhaps the new model can actually help in this regard.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: