Beki Grinter

Posts Tagged ‘reviewing’

Snowbird: Peer Reviewing

In academia, computer science, discipline, research on July 20, 2010 at 1:17 pm

The third session I attended at CRA was on peer review, it was a panel organized by Moshe Vardi.

Computer Science is very unique. We rely heavily on conferences as the means of publication. More so than other fields. Additionally we have a model of specialized conferences, unlike other sciences that have an Annual Meeting, the last ACM Annual Meeting was in 1984.

Someone quipped that “a computer science conference is just a journal that meets at a hotel.”

So recently there have been concerned about the number of conferences, the quality of those conferences, and what it means to be driven by conference deadlines. Jeannette Wing also pointed out that this applies to funding deadlines. Another concern raised by her was how this taxes the community of reviewers. She also said something I liked which was a reminder, but well put, that conferences and journals are a means of documenting the discovery of scientific truth by building on past knowledge in order to share it with others. Finally, it was observed that conferences cost time and money.

Perhaps the most troubling concern was the implications of the profusion of conferences for the field of Computer Science. The concerns raised included a tendency towards incrementalism, conservatism (in submission and review I believe), that the field would splinter, and it would miss big ideas. Computer Science would lose it’s vibrancy and excitement.

But why does this happen, why do we continue to submit to conferences? That led to a discussion of how we understand impact. Not surprisingly given that this is largely a crowd of department heads and Deans, it led to a discussion of how impact is measured on the academic vita at those crucial points: admission into graduate programs, faculty hire, tenure and promotion to Full.

So this raises a two questions for me.

First, how do we change this, if we think we should? The scale of the change required seems vast to me, requiring both procedural and cultural changes. It requires changing behaviors of the 1000s of people collectively involved in Computer Science. It also requires convincing those at the earliest steps (the undergraduates who are considering graduate school and working on publications) that they still have a chance of participating in those later steps. Someones just mentioned that it’s going to involve ensuring that every single review letter changes in accordance…

Second, what about considering the production process? We spent our time focused on the outputs, but what about looking at the inputs into the system, i.e. the number of people we’ve trained. Specifically a focus on PhD production. If a faculty member can produce 14 students in 20 years, who are all trained in the process and seek to continue to publish, well that seems like a scaling up.


Scholarly Debates: Revisiting Ethnography Considered Harmful

In academic management, computer science, discipline, empirical, HCI on January 19, 2010 at 3:17 pm

Recently, a colleague of mine Rich DeMillo blogged about Moshe Vardi’s most recent editorial in the Communications of the ACM. For those people who are not familiar with the Communications of the ACM (CACM), it is the monthly flagship magazine of the Association for Computer Machinery (a professional society to which many/most Computer Scientists in the United States, and increasingly across the world, belong).

I agree with Moshe Vardi that we, Computing professionals, could use more debate. I like the point-counter point series that was started in Communications of the ACM. My guess is that for some types of debate the columns are too short, but I’ve enjoyed the ones that have appeared and I feel it helps me understand a bit more about where some of the challenges in Computing reside. I also think written debate as part of the process of developing and refining knowledge leads to a richer more nuanced understanding of the subject matter under discussion.

But, I find myself even more in agreement with Rich’s response. Rich’s response raised a number of points, and reminded me again of how debate requires consideration, not just of the content, but of the mechanisms by which that debate occurs. And that’s what I want to focus on because if we are going to debate we should do it with decency and dignity.

Last year, the HCI community had a similar “learning how to debate” experience that involved a paper called “Ethnography Considered Harmful“, also positioned as a debate about whether non-ethnomethodological forms of ethnography made contributions to the field of Human Computer Interaction. A quote from the paper:

Closer to home, where home means that the studies take place within the researchers’ own country of residence, we find that such settings as the American Mega-church have been accorded a similar treatment, e.g.,

“ … the first thing we noticed when entering sanctuaries were the large screens on both sides of a central stage … During services screens were used extensively to accompany music, illustrate sermons, and share announcements and video. Words to hymns and Bible verse were frequently displayed over a background depicting religiously-inspired imagery. It was also common for the pastor to read selected Bible verses that would then be displayed on the screen. We observed parishioners following along and looking at screens to know what verse to turn to in their Bibles. The appearance of a new verse on the screen was followed by a flurry of paper turning.”

While ethnography has its roots in the study of exotic settings both abroad and at home, and while these sorts of study do illustrate the diversification of digital technologies in everyday life, it is imperative that ethnography provide more than surface descriptions of action and interaction. The danger is that adopting what might be called an ‘exotic tales methodology’ in design will result in descriptions that offer up little more than ‘scenic features’ of action and interaction for consideration, thus sensitizing designers to little more than the grossly observable features of a setting or culture.

The paper that they quote from in the second paragraph is one of mine, and I freely admit that I don’t share their view that it was surface description. The rest of this post, I will try to make a case not about content, but about process. Obviously I disagree with the content, but I will try to refrain from that discussion here.

The paper of mine that they quote from and use to discuss is an interesting genre of submission to the ACM Conference on Human Factors in Computing Systems. It’s not an archival submission, it’s actually something called a “work in progress” which I have always taken to mean exactly that. The resultant “publication” is not archival, copyright is retained by the author. So, I have always interpreted them as an opportunity to get early feedback on something that is, well yes, a work in progress. (Also, if you look at  one, you’ll see that it’s not a lot of words, so I take that as another sign that its not meant to be final, rather an outline and some ideas that will be refined or even rejected through discussion).

I’d like to think that the confusion about the intent of this work in progress could have been avoided if the authors had engaged in a discussion with me about their upcoming paper that was going to debate the role of my work in progress for the field. Debate implies willingness to discuss with people who hold alternate positions. Perhaps if the authors had sent me a copy of their paper as they were writing it I’d have told them that I thought critiquing a work in progress as a final result was weakening their argument. I could have pointed them to final versions of this work. Also, perhaps in talking to me about my surface descriptions, they’d have found some words that would have helped me to understand their points more clearly. (To be honest, I take surface description as an insult, and find it hard to see beyond that slight to what point they are actually trying to make).

The role of the review process also came up in the Vardi-DeMillo context. Ethnography Considered Harmful made it through the review process. And the CHI Conference has a long and careful review process. But reviewing is a human process. There’s been a lot of discussion about the CHI review process, it happens each year. I personally have always blamed the randomness on the size of CHI. CHI is huge and growing at a crazy rate. Submissions climb each year. The number of reviewers required to review also grows, when I was Papers Co-Chair in 2006, it involved 2500 discrete reviewers, and I would expect the number to have passed the 3000 mark by now. I believe that there’s a lot of goodwill in the system but at that scale, things happen.

Again, I think that dependency on the review process might have been mitigated by the authors contacting any of the people whose work they sought to debate. A colleague of mine forwarded me the paper after it was accepted to the conference. Like me, the person who forwarded the paper to me had their own research similarly characterised, and it turned out that there were quite a number of us. What was common among us all was that none of us had been contacted during the writing process. What type of debate begins this way?

Finally, and perhaps this is just me. I think it’s healthier when debates are among peers in the community. Debating requires having a voice, a voice that matches that of those who hold different positions. Why, because the debate will be better, it will be richer, because it will be among peers. Some of the people’s work that the authors of Ethnography Considered Harmful held up turned out to be scholarship by newer members of the community (either through transition into HCI or as students). I am not saying that students and new members should not be engaged by debate but I think that debate requires even more thought and respect when it involves exactly the type of people we would like to recruit and retain because of the new ideas, fresh perspectives and so forth that they bring.

I’d like to engage in debates, I think there’s much to be learnt from discussion, but I’d like them to be civil and conducted around a set of principles that not only respect the products of our discourse but also the scholars that produce them.


In computer science, discipline, HCI on November 8, 2009 at 9:53 am

I think some of the arguments I’ve been making about Computer Science also play out within the disciplines of Computer Science. I am struck by some of the common themes that appear in James’ post about frustrations with CHI and UIST echo thoughts about Systems and CS more generally. Having a discipline in disciplinary flux creates confusion and frustration around the objectives. I think there are upsides too, but I am convinced that they require a type of risk/courage to take. The type of risk that’s not about extending a program to do more, deeply, more of the same, but about taking the risk that what you do might even be perceived as not counting. So, real risk.