The third session I attended at CRA was on peer review, it was a panel organized by Moshe Vardi.
Computer Science is very unique. We rely heavily on conferences as the means of publication. More so than other fields. Additionally we have a model of specialized conferences, unlike other sciences that have an Annual Meeting, the last ACM Annual Meeting was in 1984.
Someone quipped that “a computer science conference is just a journal that meets at a hotel.”
So recently there have been concerned about the number of conferences, the quality of those conferences, and what it means to be driven by conference deadlines. Jeannette Wing also pointed out that this applies to funding deadlines. Another concern raised by her was how this taxes the community of reviewers. She also said something I liked which was a reminder, but well put, that conferences and journals are a means of documenting the discovery of scientific truth by building on past knowledge in order to share it with others. Finally, it was observed that conferences cost time and money.
Perhaps the most troubling concern was the implications of the profusion of conferences for the field of Computer Science. The concerns raised included a tendency towards incrementalism, conservatism (in submission and review I believe), that the field would splinter, and it would miss big ideas. Computer Science would lose it’s vibrancy and excitement.
But why does this happen, why do we continue to submit to conferences? That led to a discussion of how we understand impact. Not surprisingly given that this is largely a crowd of department heads and Deans, it led to a discussion of how impact is measured on the academic vita at those crucial points: admission into graduate programs, faculty hire, tenure and promotion to Full.
So this raises a two questions for me.
First, how do we change this, if we think we should? The scale of the change required seems vast to me, requiring both procedural and cultural changes. It requires changing behaviors of the 1000s of people collectively involved in Computer Science. It also requires convincing those at the earliest steps (the undergraduates who are considering graduate school and working on publications) that they still have a chance of participating in those later steps. Someones just mentioned that it’s going to involve ensuring that every single review letter changes in accordance…
Second, what about considering the production process? We spent our time focused on the outputs, but what about looking at the inputs into the system, i.e. the number of people we’ve trained. Specifically a focus on PhD production. If a faculty member can produce 14 students in 20 years, who are all trained in the process and seek to continue to publish, well that seems like a scaling up.