I was recently asked about the responsibilities of an Associate Chair/Program Committee Member. I’ve written about the Program Committee meeting before. Here are my thoughts based on being an Associate Chair and Papers Chair.
There are two types of Program Committee. Ones that review themselves, and ones where the members of the committee are responsible for soliciting reviews from other people not on the program committee, which in HCI is often referred to as being an Associate Chair (AC). If you are in a review the papers within the Program Committee style group then its the same as reviewing papers along with the Program Committee meeting which I’ve already described. So, I’ll focus on the one where you are an AC.
The first responsibility that an AC has is to assign papers. There maybe a period of trading, and while trading can seem attractive, you have to remember that you need to take something out of the pool. There’s a risk to that.
My experience is that the sooner an AC asks reviewers to review a paper, the more likely they are to get the people that they would like. The longer an AC waits the more likely they will spend more time on the activity.
So, how do you find reviewers? I typically read the abstract closely, look at the references, and skim most of the remainder of the paper. These three activities I used to brainstorm a list of people I think could review the paper. When coming up with a list of reviewers I also account for seniority. Specifically, I think that my senior colleagues, and hopefully myself to an extent, can review papers that are further outside of my immediate area. There are limits to this, but I also factor that into my consideration. I also try to identify at least one student per paper to provide opportunities to learn about reviewing to the newest members of the committee. This is a good mentoring opportunity. A final consideration I make is whether to bring in someone who is further outside of the field for a review. This can be very useful when the work is interdisciplinary, or when the domain is very important in the research results. Those reviews help me to assess the interdisciplinary contribution, it’s a review that brings a valuable but different perspective, so it helps me with my task as a meta-reviewer in a different way from other reviews.
In doing the assignments, I frequently contact the person I would like to review outside of any conference management system first to ask. I do this for two reasons. First, I think auto-assigning features in conference systems are rather impolite. They turn on a set of expectations that may or may not be actually true of a person (like that person is available for reviewing, and not chairing a different conference but the system set her up to review — that just happened to me). Also, sometimes contact information changes, and I am sure like others I do not immediately think to update my information in a conference management system! By contacting someone directly, I have secured their commitment, and then it’s just a matter of getting them to update their information if necessary.
Then it’s a matter of waiting for your reviewers to provide their reviews. While doing this I read the papers assigned carefully and formulate my own thoughts about the work.
ACs occupy a rather unique position in the reviewing process. Reviewers contribute their scholarly judgements. AC’s have their own thoughts but are also required to synthesize their reviewers opinions. This is relatively easy when everyone agrees. It’s harder when you and your reviewers disagree, especially when you disagree with all your reviewers. Fortunately that’s rather rare, but it raises the point. An AC represents their reviewers and themselves. Not everyone agrees that an AC should do both, erring to the idea that the AC represents the reviewers. But it’s remarkably hard for a scholar not to have their own assessment. I think recognizing that the two different and potentially competing things are in play is important. I try to write my meta-reviews in a way that distinguishes these two perspectives in such a way that the author can see my thoughts and my analysis of the reviewers thoughts. And I try really hard to weight my own thoughts less in my overall assessment than those of the reviewers I’ve asked to review the paper.
The reviews come back. If your reviewers all agree, and particularly if they all agree either positively or negatively enough that you know that the paper is clearly acceptable or rejectable this is pretty straightforward. In the case of a reject, I think it is particularly important that the metareview do its best to convey some positive aspects of the paper, and to encourage the author to take the work further.
When your reviewers disagree its time to start a discussion among them. This can help. My experience is that rarely does it resolve the differences, but it often softens the differences. Then I try to use the positive reviews in my meta-review as what’s good and important about the work, while using the negative reviews to suggest what needs fixing.
The final decision is one then of summing up the positive and negative concerns and deciding, using a score often as not, whether you believe the balance is favourable or not. There has been some discussion about whether we are too critical. I hear this about sub-areas in HCI, I also hear it about at least one of the academic disciplines of which HCI is a part (Computer Science). So here again I call out Matt Welsh’s post on the case for arguing for a paper to be accepted. Because it’s easier to argue for rejection. But clearly there are some times when a paper needs too much to become acceptable. I try to imagine how much time it would take to rewrite, and in some cases reanalysis, rerun etc. empirical experiments. That’s just not possible in the month from mid-Dec to mid-Jan when the camera ready version comes due. I try to explain what I am accounting for, what I am ignoring (if anything and why) and communicate with the authors. Most of my meta reviews are about as long as this post.
At this point there maybe a rebuttal period which gives the authors a chance to see the reviews and your metareview which is the most significant determinant of the current likely fate of the paper. I say that because the AC synthesizes and interprets the reviews. And what they interpret is the judgement.
Have you, the author, ever wondered why at rebuttal you see some reviews and then when reviews are finally returned at the end of the process there are sometimes more.
For papers that have disagreement among reviewers, or ones that have attracted a lot of neutral responses, or ones that the AC decides they want discussed, these are the ones that go into the program committee meeting. An AC always has the ability to raise a paper for discussion (some conferences discuss all papers, but my experience is that only a subset are discussed). I’ve raised papers for discussion when despite most of the reviewers agreeing one does not, and that one person is senior. I raise it for discussion because in that case I want other colleagues to take a look and see whether the senior person is on to something that I don’t know and I want a sanity check.
The final thing I’ll cover here is conflict of interest. At the program committee meeting you will be asked to step outside the room during a discussion of a paper for which you have a conflict of interest. You are expected to be conflicted with your advisor and your institution. Beyond that, and quite often, its up to you to determine when you have a conflict of interest. I’ve used the National Science Foundation’s conflict of interest criteria as a guideline for myself.