Beki Grinter

Snowbird: Peer Reviewing

In academia, computer science, discipline, research on July 20, 2010 at 1:17 pm

The third session I attended at CRA was on peer review, it was a panel organized by Moshe Vardi.

Computer Science is very unique. We rely heavily on conferences as the means of publication. More so than other fields. Additionally we have a model of specialized conferences, unlike other sciences that have an Annual Meeting, the last ACM Annual Meeting was in 1984.

Someone quipped that “a computer science conference is just a journal that meets at a hotel.”

So recently there have been concerned about the number of conferences, the quality of those conferences, and what it means to be driven by conference deadlines. Jeannette Wing also pointed out that this applies to funding deadlines. Another concern raised by her was how this taxes the community of reviewers. She also said something I liked which was a reminder, but well put, that conferences and journals are a means of documenting the discovery of scientific truth by building on past knowledge in order to share it with others. Finally, it was observed that conferences cost time and money.

Perhaps the most troubling concern was the implications of the profusion of conferences for the field of Computer Science. The concerns raised included a tendency towards incrementalism, conservatism (in submission and review I believe), that the field would splinter, and it would miss big ideas. Computer Science would lose it’s vibrancy and excitement.

But why does this happen, why do we continue to submit to conferences? That led to a discussion of how we understand impact. Not surprisingly given that this is largely a crowd of department heads and Deans, it led to a discussion of how impact is measured on the academic vita at those crucial points: admission into graduate programs, faculty hire, tenure and promotion to Full.

So this raises a two questions for me.

First, how do we change this, if we think we should? The scale of the change required seems vast to me, requiring both procedural and cultural changes. It requires changing behaviors of the 1000s of people collectively involved in Computer Science. It also requires convincing those at the earliest steps (the undergraduates who are considering graduate school and working on publications) that they still have a chance of participating in those later steps. Someones just mentioned that it’s going to involve ensuring that every single review letter changes in accordance…

Second, what about considering the production process? We spent our time focused on the outputs, but what about looking at the inputs into the system, i.e. the number of people we’ve trained. Specifically a focus on PhD production. If a faculty member can produce 14 students in 20 years, who are all trained in the process and seek to continue to publish, well that seems like a scaling up.

  1. I think we need a journal review process (for quality), but with a conference presentation (for sharing/publicity). I think entire conferences should move this way (not just a partial attempt like TOCHI/CHI). Reviews need to be more of a multi-pass conversation so that we don’t just publish the easy to understand type of work (often incremental or smaller in scope) and crowd out everything else.

  2. what i don’t understand about the arguments that conferences should not be journals (splintering, incrementalism, etc.) is that the same things seem to happen to journals. over time you get increasingly mediocre and specialized journals because the research metric is papers and they have to go somewhere. we all wish the research we find too tedious to review would go away but it doesn’t. in science blogs you see the same genres of complaints about Science/Nature that you see leveled at “big tent” conferences like CHI.

    the deadline thing i see as an internal management problem in that an experienced researcher can generally manage someone to get incremental results that can be published. for predictable annual deadlines anyway.

    meeting cost is quite serious in this environment. why can we not make the FCRC thing work better for our boutique conferences.

  3. I share some of your thoughts, right now our journals probably look quite attractive, but that’s because we don’t use them in the same way (yet). I completely agree about meeting cost. This year’s CHI and the volcanic eruption really brought that home for me. Extreme, but makes you reflect.

  4. […] This post was mentioned on Twitter by amette, andrea forte. andrea forte said: overheard by @beki70 on peer review: "a computer science conference is just a journal that meets at a hotel” ( ) […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: