Beki Grinter

Posts Tagged ‘peer review’

First time Associate Chair/Program Committee Member

In academic management, discipline, research on July 21, 2011 at 7:18 am

I was recently asked about the responsibilities of an Associate Chair/Program Committee Member. I’ve written about the Program Committee meeting before. Here are my thoughts based on being an Associate Chair and Papers Chair.

There are two types of Program Committee. Ones that review themselves, and ones where the members of the committee are responsible for soliciting reviews from other people not on the program committee, which in HCI is often referred to as being an Associate Chair (AC). If you are in a review the papers within the Program Committee style group then its the same as reviewing papers along with the Program Committee meeting which I’ve already described. So, I’ll focus on the one where you are an AC.

The first responsibility that an AC has is to assign papers. There maybe a period of trading, and while trading can seem attractive, you have to remember that you need to take something out of the pool. There’s a risk to that.

My experience is that the sooner an AC asks reviewers to review a paper, the more likely they are to get the people that they would like. The longer an AC waits the more likely they will spend more time on the activity.

So, how do you find reviewers? I typically read the abstract closely, look at the references, and skim most of the remainder of the paper. These three activities I used to brainstorm a list of people I think could review the paper. When coming up with a list of reviewers I also account for seniority. Specifically, I think that my senior colleagues, and hopefully myself to an extent, can review papers that are further outside of my immediate area. There are limits to this, but I also factor that into my consideration. I also try to identify at least one student per paper to provide opportunities to learn about reviewing to the newest members of the committee. This is a good mentoring opportunity. A final consideration I make is whether to bring in someone who is further outside of the field for a review. This can be very useful when the work is interdisciplinary, or when the domain is very important in the research results. Those reviews help me to assess the interdisciplinary contribution, it’s a review that brings a valuable but different perspective, so it helps me with my task as a meta-reviewer in a different way from other reviews.

In doing the assignments, I frequently contact the person I would like to review outside of any conference management system first to ask. I do this for two reasons. First, I think auto-assigning features in conference systems are rather impolite. They turn on a set of expectations that may or may not be actually true of a person (like that person is available for reviewing, and not chairing a different conference but the system set her up to review — that just happened to me). Also, sometimes contact information changes, and I am sure like others I do not immediately think to update my information in a conference management system! By contacting someone directly, I have secured their commitment, and then it’s just a matter of getting them to update their information if necessary.

Then it’s a matter of waiting for your reviewers to provide their reviews. While doing this I read the papers assigned carefully and formulate my own thoughts about the work.

ACs occupy a rather unique position in the reviewing process. Reviewers contribute their scholarly judgements. AC’s have their own thoughts but are also required to synthesize their reviewers opinions. This is relatively easy when everyone agrees. It’s harder when you and your reviewers disagree, especially when you disagree with all your reviewers. Fortunately that’s rather rare, but it raises the point. An AC represents their reviewers and themselves. Not everyone agrees that an AC should do both, erring to the idea that the AC represents the reviewers. But it’s remarkably hard for a scholar not to have their own assessment. I think recognizing that the two different and potentially competing things are in play is important. I try to write my meta-reviews in a way that distinguishes these two perspectives in such a way that the author can see my thoughts and my analysis of the reviewers thoughts. And I try really hard to weight my own thoughts less in my overall assessment than those of the reviewers I’ve asked to review the paper.

The reviews come back. If your reviewers all agree, and particularly if they all agree either positively or negatively enough that you know that the paper is clearly acceptable or rejectable this is pretty straightforward. In the case of a reject, I think it is particularly important that the metareview do its best to convey some positive aspects of the paper, and to encourage the author to take the work further.

When your reviewers disagree its time to start a discussion among them. This can help. My experience is that rarely does it resolve the differences, but it often softens the differences. Then I try to use the positive reviews in my meta-review as what’s good and important about the work, while using the negative reviews to suggest what needs fixing.

The final decision is one then of summing up the positive and negative concerns and deciding, using a score often as not, whether you believe the balance is favourable or not. There has been some discussion about whether we are too critical. I hear this about sub-areas in HCI, I also hear it about at least one of the academic disciplines of which HCI is a part (Computer Science). So here again I call out Matt Welsh’s post on the case for arguing for a paper to be accepted. Because it’s easier to argue for rejection. But clearly there are some times when a paper needs too much to become acceptable. I try to imagine how much time it would take to rewrite, and in some cases reanalysis, rerun etc. empirical experiments. That’s just not possible in the month from mid-Dec to mid-Jan when the camera ready version comes due. I try to explain what I am accounting for, what I am ignoring (if anything and why) and communicate with the authors. Most of my meta reviews are about as long as this post.

At this point there maybe a rebuttal period which gives the authors a chance to see the reviews and your metareview which is the most significant determinant of the current likely fate of the paper. I say that because the AC synthesizes and interprets the reviews. And what they interpret is the judgement.

Have you, the author, ever wondered why at rebuttal you see some reviews and then when reviews are finally returned at the end of the process there are sometimes more.

For papers that have disagreement among reviewers, or ones that have attracted a lot of neutral responses, or ones that the AC decides they want discussed, these are the ones that go into the program committee meeting. An AC always has the ability to raise a paper for discussion (some conferences discuss all papers, but my experience is that only a subset are discussed). I’ve raised papers for discussion when despite most of the reviewers agreeing one does not, and that one person is senior. I raise it for discussion because in that case I want other colleagues to take a look and see whether the senior person is on to something that I don’t know and I want a sanity check.

The final thing I’ll cover here is conflict of interest. At the program committee meeting you will be asked to step outside the room during a discussion of a paper for which you have a conflict of interest. You are expected to be conflicted with your advisor and your institution. Beyond that, and quite often, its up to you to determine when you have a conflict of interest. I’ve used the National Science Foundation’s conflict of interest criteria as a guideline for myself.


Scholarly Debates: Revisiting Ethnography Considered Harmful

In academic management, computer science, discipline, empirical, HCI on January 19, 2010 at 3:17 pm

Recently, a colleague of mine Rich DeMillo blogged about Moshe Vardi’s most recent editorial in the Communications of the ACM. For those people who are not familiar with the Communications of the ACM (CACM), it is the monthly flagship magazine of the Association for Computer Machinery (a professional society to which many/most Computer Scientists in the United States, and increasingly across the world, belong).

I agree with Moshe Vardi that we, Computing professionals, could use more debate. I like the point-counter point series that was started in Communications of the ACM. My guess is that for some types of debate the columns are too short, but I’ve enjoyed the ones that have appeared and I feel it helps me understand a bit more about where some of the challenges in Computing reside. I also think written debate as part of the process of developing and refining knowledge leads to a richer more nuanced understanding of the subject matter under discussion.

But, I find myself even more in agreement with Rich’s response. Rich’s response raised a number of points, and reminded me again of how debate requires consideration, not just of the content, but of the mechanisms by which that debate occurs. And that’s what I want to focus on because if we are going to debate we should do it with decency and dignity.

Last year, the HCI community had a similar “learning how to debate” experience that involved a paper called “Ethnography Considered Harmful“, also positioned as a debate about whether non-ethnomethodological forms of ethnography made contributions to the field of Human Computer Interaction. A quote from the paper:

Closer to home, where home means that the studies take place within the researchers’ own country of residence, we find that such settings as the American Mega-church have been accorded a similar treatment, e.g.,

“ … the first thing we noticed when entering sanctuaries were the large screens on both sides of a central stage … During services screens were used extensively to accompany music, illustrate sermons, and share announcements and video. Words to hymns and Bible verse were frequently displayed over a background depicting religiously-inspired imagery. It was also common for the pastor to read selected Bible verses that would then be displayed on the screen. We observed parishioners following along and looking at screens to know what verse to turn to in their Bibles. The appearance of a new verse on the screen was followed by a flurry of paper turning.”

While ethnography has its roots in the study of exotic settings both abroad and at home, and while these sorts of study do illustrate the diversification of digital technologies in everyday life, it is imperative that ethnography provide more than surface descriptions of action and interaction. The danger is that adopting what might be called an ‘exotic tales methodology’ in design will result in descriptions that offer up little more than ‘scenic features’ of action and interaction for consideration, thus sensitizing designers to little more than the grossly observable features of a setting or culture.

The paper that they quote from in the second paragraph is one of mine, and I freely admit that I don’t share their view that it was surface description. The rest of this post, I will try to make a case not about content, but about process. Obviously I disagree with the content, but I will try to refrain from that discussion here.

The paper of mine that they quote from and use to discuss is an interesting genre of submission to the ACM Conference on Human Factors in Computing Systems. It’s not an archival submission, it’s actually something called a “work in progress” which I have always taken to mean exactly that. The resultant “publication” is not archival, copyright is retained by the author. So, I have always interpreted them as an opportunity to get early feedback on something that is, well yes, a work in progress. (Also, if you look at  one, you’ll see that it’s not a lot of words, so I take that as another sign that its not meant to be final, rather an outline and some ideas that will be refined or even rejected through discussion).

I’d like to think that the confusion about the intent of this work in progress could have been avoided if the authors had engaged in a discussion with me about their upcoming paper that was going to debate the role of my work in progress for the field. Debate implies willingness to discuss with people who hold alternate positions. Perhaps if the authors had sent me a copy of their paper as they were writing it I’d have told them that I thought critiquing a work in progress as a final result was weakening their argument. I could have pointed them to final versions of this work. Also, perhaps in talking to me about my surface descriptions, they’d have found some words that would have helped me to understand their points more clearly. (To be honest, I take surface description as an insult, and find it hard to see beyond that slight to what point they are actually trying to make).

The role of the review process also came up in the Vardi-DeMillo context. Ethnography Considered Harmful made it through the review process. And the CHI Conference has a long and careful review process. But reviewing is a human process. There’s been a lot of discussion about the CHI review process, it happens each year. I personally have always blamed the randomness on the size of CHI. CHI is huge and growing at a crazy rate. Submissions climb each year. The number of reviewers required to review also grows, when I was Papers Co-Chair in 2006, it involved 2500 discrete reviewers, and I would expect the number to have passed the 3000 mark by now. I believe that there’s a lot of goodwill in the system but at that scale, things happen.

Again, I think that dependency on the review process might have been mitigated by the authors contacting any of the people whose work they sought to debate. A colleague of mine forwarded me the paper after it was accepted to the conference. Like me, the person who forwarded the paper to me had their own research similarly characterised, and it turned out that there were quite a number of us. What was common among us all was that none of us had been contacted during the writing process. What type of debate begins this way?

Finally, and perhaps this is just me. I think it’s healthier when debates are among peers in the community. Debating requires having a voice, a voice that matches that of those who hold different positions. Why, because the debate will be better, it will be richer, because it will be among peers. Some of the people’s work that the authors of Ethnography Considered Harmful held up turned out to be scholarship by newer members of the community (either through transition into HCI or as students). I am not saying that students and new members should not be engaged by debate but I think that debate requires even more thought and respect when it involves exactly the type of people we would like to recruit and retain because of the new ideas, fresh perspectives and so forth that they bring.

I’d like to engage in debates, I think there’s much to be learnt from discussion, but I’d like them to be civil and conducted around a set of principles that not only respect the products of our discourse but also the scholars that produce them.


In computer science, discipline, HCI on November 8, 2009 at 9:53 am

I think some of the arguments I’ve been making about Computer Science also play out within the disciplines of Computer Science. I am struck by some of the common themes that appear in James’ post about frustrations with CHI and UIST echo thoughts about Systems and CS more generally. Having a discipline in disciplinary flux creates confusion and frustration around the objectives. I think there are upsides too, but I am convinced that they require a type of risk/courage to take. The type of risk that’s not about extending a program to do more, deeply, more of the same, but about taking the risk that what you do might even be perceived as not counting. So, real risk.