Program Committee Meetings: Psychology and Structure

Matt Walsh wrote two blog posts, one about what he calls the psychology of Program Committees and one on the structure of Program Committees. When I read these posts I was reminded of all the things I take for granted because I have served on Program Committees, and I thought I would add some of my own reflections on the experiences. I remember that I wished I had known more about program committees before I served on one the first time. To calibrate my advice, the usual caveats apply. I’ve Chaired and been an Associate Chair for a variety of conferences in the HCI area. I’ve also served on some outside, Software Engineering, ICT4D for example.

First, Matt’s psychology post. I’ve those happen, and I’ve also seen people argue against those arguments. One of the things I like about the post is that by making these argumentation types clear it gives anyone who serves on a program committee the opportunity to reflect on whether that’s the argument that’s at work on a particular paper and decide whether that’s a fair or not criticism. One that’s missing, maybe because it’s more unique to HCI is the argument that turns on a mis-understanding that arises from the serious methodological differences that exist within our community. A challenge of HCI’s diversity is understanding how epistemological and ontological differences affect the work produced.

What Matt is very right about is that it’s easier to reject a paper and about the work it takes to champion a paper against a negative review.

The final point is that it is easy to argue to reject a paper; much harder to argue to accept a paper over other reviewers’ objections. If a reviewer is not really sure about the novelty or importance of a paper’s contributions, they often defer to the most negative reviewer, since nobody likes looking like an idiot in a PC meeting. Standing up and championing a paper takes a lot of guts, and by doing so you are taking responsibility for any faults in the paper that might arise later (if it turns out, say, that the exact same idea was published before, or there is a flaw in the experimental design). I think it’s important that every member of a program committee commit themselves to championing one or two papers during the meeting, even if they aren’t so sure about them — otherwise the process can get to be too negative. One way to view your role as a PC member is to identify that beautiful piece of work that would otherwise have been overlooked due to superficial problems that turn off the reviewers.

I’ve done this a few times, I like his idea of doing it more frequently. In a way I think it resonates with the advice that CHI reviewers get which is to not search for the fatal flaws but to weigh them against other concerns like novelty and interest of topic.

Moving on to the structure of the meeting. I’ve been involved in ones that review all the material submitted collectively, and also ones that review by topic. CHI is a review by topic, hence the importance of picking the right subcommittee.

Most of the program committees I’ve been in have started by reviewing and discussing the most highly ranked papers (working downwards by score), perhaps for the first session of the morning, say an hour and a half. The purpose of this is to start off with what the community considers to be the best. Then, say the next hour and a half session, I’ve been on committees that have reviewed the lowest ranked papers (working upwards by score). In some conferences that has been to review the very lowest scores, in others there’s a cut off.

Setting a cut off is quite complicated. A cut off needs to reflect an overall low score, but handle papers of high score variance. Typically I’ve seen scores set somewhere in the 2.X range (on a scoring system where 5 is strong accept and 1 is strong reject) but including lower ranked papers who have significant variance (e.g., scores of say 1 and 5 from different reviewers who were unable to reconcile their differences during pre-meeting online discussion. FYI online discussion among the reviewers is a fairly common part of the review process also). What everyone should know about cut offs is that I’ve never seen one set which didn’t also allow any AC to raise any paper they want discussed irrespective of the cut off. And AC’s do. So cut offs are not definitive, they serve as guidelines.

Terrifyingly even with cut offs being used I’ve been in meetings where there are very few minutes to discuss each paper. Another thing I learnt in meeting is that it’s important to focus on the most important details. For accepted papers (and I would love to hear comments on other things that should be discussed) explaining what it’s contributions are, and why you and the reviewers felt that those mattered seems like the most obvious thing to highlight. I’ll add that sounding enthusiastic and excited about them also helps to make the case, well I think so anyway. In the case of a rejection, I try to highlight what the problem(s) are in the paper and why they are not addressable in a edit cycle.

Back to the meeting. The third session of the day may focus back again on the top, where the previous session left off, and then back to the bottom in the fourth and final session of the day. At the end of the day there are papers in three categories, if not four: accepted, rejected, to rediscuss and shepherd. The rediscuss category means that a decision has not been made. One thing that happens at program committee meetings is that AC’s acquire extra papers to read overnight between the end of the first day and the beginning of the second.. Something I did not know when I attended my first program committee meeting was to budget time in the evening for reading papers and preparing reviews of them, many of which will get entered into the review system (have you noticed that sometimes at rebuttal your paper has 3 reviews and then when it’s returned finally there are 6, yes, that is where those come from).

The second day has to accomplish two major things. Discussing any papers that have not yet been discussed (those in the middle) and resolving all the open papers, i.e. those that were but for which further discussion is required. That I think is the time when the Psychology really comes out. Program Committee meetings are tiring, they are intellectually exhausting. Not only is there a lot going on as papers flurry past and decisions get made, but these are of course also opportunities to catch up with colleagues. A lot is going on.

Conflict management as Matt points out is a huge issue. I’ve been at some meetings where specific guidelines were used about what a conflict of interest constitutes, for example borrowing the same set of rules that the NSF uses to determine the candidate of ineligible reviewers. But I’ve also been in meetings where it’s left to the discretion of each AC to determine their conflicts. But the in and out of the room is a very chaotic experience. The reason that this happens is because its important that someone at Institution Z doesn’t know who the AC is for any paper at Institution Z, that’s a conflict of interest, the most obvious one. So it means each new paper discussed begins with a list of institutions who are authors on the paper and then they are invited to leave the room. I’m all for an iPhone app myself. Finding people who have left the room can be difficult, they get into discussion or decide to take a bio break, these are quite reasonable events, but I’ve certainly been in situations (and likely created ones) where the missing person unable to be found prompts a discussion of lets discuss another paper from the same institution, but in the interim someone has gone looking so you temporarily return only to be evicted from the discussion once again. I don’t think it’s possible to leave people with conflicts in the same room, even if they don’t discuss. It places a burden of knowledge on them that I know I certainly wouldn’t want. The question is whether something more reasonable can be done…. and if so what is it?

Snowbird: Working with your Dean

The final session I attended was a session called working with your Dean. Eight Deans offered their perspectives on this. It was a session that was targetted at Department Chairs, but even though I am not one I found it very useful, and really enjoyed hearing what the Deans had to say. The majority of the session was small group discussions, and I found the dialogs very productive and illuminating.

Some take aways I got from this session.

I continue to develop an understanding of the partnerships that are Deans and Chairs. How they both work together and towards execution of the University’s mission and strategy. That sounds obvious as a single sentence, but what emerged during the discussions were processes for doing this and details about how to go about this. Differences also emerged between Deans with Chairs, those without, Deans who have many departments in their School, and those who have less. And of course, for Computer Science which can be free-standing, within Engineering, within Sciences, etc… each one comes with different sets of constituents and concerns.

I learnt that there is scholarly evidence that shows that women are far more likely to leave if they get an alternate offer. This led to a discussion of how retention policies that force the acquisition of an alternate offer are likely to have a negative affect on diversity. I also learnt that some universities have policies on how many times someone can be a retention case, usually it’s a number of years between each retention bid.

Dual body opportunities also came up. I was delighted to learn that Deans increasingly see dual body opportunities as the normal hiring mode. Some are dual body academics, but there’s a large class of professional partners, so whether the partner happens to be an academic or not, hiring is increasingly having to account for the presence of another person who needs support.

Clear, open, communications came up over and over again. While I think this is true in any working environment, I have the impression that it is uniquely true in academia. Given the role of the faculty in raising funds that support the University operation, tenure and academic freedom, and a variety of other features that are unique to the University setting, communications seems to take on a particularly crucial role.

I heard people talk about the multiple constituencies that a leader interacts with including faculty, staff, students, the educational mission of the University, and the other University leadership.

Snowbird: Local Global Development

I was one of three invited panelists in a session on Global Development.

I presented the case for Computing at the Margins and the results of an NSF sponsored workshop. The key point I made was to argue that Global Development has a domestic component focused on those who have been at the margins of technological innovation. The solutions won’t be identical of course, but there are classes of problems that span underserved parts of Industrialized nations as exist in Emerging nations. And from a scientific perspective, sharing knowledge across these boundaries ensures that we understand what’s generalizable and what is locally specific.

I also used my talk to connect this effort to other problems discussed. For example, DARPA and the NSF both have an interest in socio-computational systems (think wikipedia, North Korea Uncovered, wikipedia). So does Computing at the Margins: what would happen if more people could participate and benefit from these collaborative content creation experiences, what new ones would they create? How would it change or reveal the edges that DARPA seeks to understand? Nomadicity, the reflection that the global population is increasingly migratory, may also contribute to understanding why the edges no longer conform to national borders.

My two co-presenters, Lakshminarayanan Subramanian and Tapan Parikh talked about a variety of the technical challenges in Global Development and about the role it can play in education, as well as for the future of Computing. I was delighted to be in such good company. In this post I want to make notes about the things that got me thinking some more.

Tapan mentioned an article by Ammon Eden that articulated three paradigms of Computer Science research. I saw lots of people writing down the title of the paper, and I have previously blogged about it myself (it includes a link to the paper). Global Development he argued fits into the Science definition. I’m inclined to agree. But, as I’ve written about before I think global development exposes an interesting set of assumptions in Computing, so even if it fits paradigmatically, it’s not without challenges (or more optimistically, game changers that will productively extend the field of Computer Science). One other that now springs to mind, is the co-evolution of Computer Science the discipline with the National Science Foundation. Given how central the NSF is to Computing, and how long the two have co-evolved, it makes it even clearer to me that the NSF’s support is crucial in advancing this field. Now I understand this, the question I have is what to do about it. I’ll take answers from readers please.

Solving the right problem is important. It’s always important. But, in much of Computer Science the problem discovery phase takes far less time than the problem solution phase. Global Development, rather like HCI and Software Engineering I think, requires attention on problem discovery.

We talked about whether industry could make progress on these problems. We varied somewhat in how much we thought that industry was engaged in this space, but it is clear that corporate America is paying attention to emerging nations, as emerging markets. We also discussed the role that basic research, unfettered by the need to begin with existing platforms and solutions, could contribute.

The phrase end-to-end came up in two distinct way. First, there was agreement that this area of research requires solutions that span the distinct sub-fields of computer science. Simply put you need people who understanding networking, operating systems, and hci (and much more) in order to create a workable solution. That was dubbed end-to-end systems. Then there was also an end-to-end methodological discussion, about how both problem discovery requires and evaluation requires empirical research that wraps around the system development.

I’ve wondered this before, but I’ll wonder out loud on my blog, is HCI style research rather uniquely positioned to take advantage of these end-to-end requirements. Methodologically, HCI already has practices in place (I think some of these practices will not work in these settings and that innovation is required there, but that’s a different problem from not having any practices in place). And while this may be controversial, perhaps it’s a good time to let people know that HCI doesn’t just concern the interface and nor do HCI researchers limit themselves to toolkits for that. HCI researchers partner with or engage directly in a variety of technical concerns that go down the stack. When I’m being uncharitable I tend to think that sometimes CS think that HCI is rather superficial and afraid of the machine, I disagree intensely. (Which also reminds me that I heard someone talk about the field of database usability for the first time. Is there any part of CS to which HCI doesn’t have something to offer. No of course not!)

Colorado has just started a Masters Program in ICT4D.

We heard from a number of people about how working in this space had been a personally life changing event. The rewards of this research space are very significant. While agreeing I also observed that the broader impacts of this work are so blinding that they overwhelm the question of what the science is in this space. I feel strongly that the Computing Community is going to have to work together to make the scientific case for this space, to ensure funding and also tenure and promotion reward for people who engage in this space.

We were invited to create a layered diagram that illustrates the types of challenges for Computer Science across the spectrum when pursuing global development. And another person asked us whether you could create an introductory course on Computer Science using problems from global development as the examples. That’s a fascinating question.

And then the name discussion came up. The name for this field is extremely complicated and loaded. I suggested Computing for Normal People, a riff on Gary Marsden’s observation that computing has largely served the hyper-developed world, and that the next 5 billion constitute what is normal.

I heard of “the last electrical engineer” phenomenon. The idea is that once everything is known you only need one person who knows it to ensure that the knowledge is not lost.

Oh one other thing. There was a session on HealthIT. Health and wellness is a significant target for investment, including technological investment. People who seek to stay well and who have health issues come from all walks of life. Health is also a global issue, what starts in one place and easily spread to many. And what it means to be well, what it means to treat someone, also varies culturally. Development confronts and deals with issues of cultural variance and its implications for technological relevance all the time. There are also more technological-centric challenges such as getting care to everyone that needs it, where ever they are and whatever access to bandwidth and health care they may have. Access and empowerment through technologies seems like a crucial part, a domain, for Development. Conversely, dealing with underserved groups is a target domain for HealthIT.

Snowbird: Faculty Hiring Gridlock

Day 2 at Snowbird included attenting a panel on faculty hiring processes. The concern that triggered this panel is that there’s a gridlock associated with faculty hiring and that this is not good for departments or for the candidates themselves. The problem as explained was that faculty slots actually go unfilled, even in a tight market, because late offers mean that candidates accumulate offers (while waiting for that last late offer hopefully). When the candidate decides and other offers are turned down it’s too late in the hiring season for the Universities with unfilled slots to recruit in that year.

One presenter showed evidence that last year of the 114 slots that departments had (113 departments interviewed), only 71 were filled. There was a widespread belief, one that I concur with, that this was detrimental to the Ph.D.’s searching for appointments.

There’s a solution on the table, it turns on several parts. First, move all the deadlines earlier, submission of applications and the time of first offers to April 1. There was some discussion of which was more important, and the backend date seems to be the more important. Second, inform candidates who will not be interviewed early so that they can make alternate plans rather than waiting for things that never come. Third, to have deadlines for telling candidates who were interviewed that they will not receive an offer and also to have deadlines for how long offers are open. I should add that the solution was not proposed as “law” but perhaps more of as a set of guiding principles…

But there are logistical constraints. One challenge is that earlier application deadlines can be difficult because sometimes Deans/Departments don’t know whether they have positions. This is especially true in difficult budget times. But, during the discussion several other fascinating deadlines and complications emerged. Semesters versus quarters seem to have a significant effect of the hiring schedule. For example, May 1 first offer deadline is better for people in semesters since the faculty are around until the middle of June than it is for those on semesters whose faculty disappear by the end of May. That a difference exists makes it hard to lock down certain dates. And of course, it’s interesting how the summer arrangement (i.e., where faculty are not paid by the institution but through their own grants) also complicates the hiring process (by reducing the amount of the year in which it can be conducted).

Another piece of the solution proposed was to tell candidates earlier that they are not going to be interviewed, or for those who do interview, that they will not receive an offer. This runs up against legal concerns in some Universities, who do not permit rejection letters to go out until the slot is filled. I did not know, but I learnt that Universities in the AAU are required to make offers to tenured faculty by April 1, and some pointed out that perhaps we should take that deadline and make it a goal for all offers.

One final observation that fits into the “you can tell you’re working with Computer Scientists” category was the number of people who described this as a game theory problem, and applied such approaches to the understanding and resolution of this problem.

I attended since I was curious about what the problem was, and how one might propose a solution that needs to be coordinated across institutions, and this panel was valuable for understanding that process.

Snowbird: Thinking Big in Computer Science

I’m in a session at the CRA Snowbird conference focused on thinking big in Computer Science as a means to pursue large grants. The session is organized by Debbie Crawford at the NSF. There were a range of speakers who each took a turn to provide their thoughts on pursuing large projects.

The first project is about robotic bees, the research to create them (it’s an NSF Expeditions). The problem set up is lovely. 30% of the worlds food requires pollination by bees. But bee colonies are dying. Can robotic bees help? It is simple to explain (and not to answer) and very compelling. Then there’s the team structure. They have 10 or so faculty in different research areas/disciplines, but all with core interests focused on robotic bees and other insects (I think that’s what I took away). Of course this suggests lots of related and prior work by the team members. Additionally they are all collocated, in Boston, with one person in Washington DC. Finally, collaborations existed among various team members also existed, so although the whole team had not worked together they all had some experience of working with other members of the team.

The process the Robobees team used to create the grant was a brainstorm meeting, collocated, purpose of which was to generate the outline for the Expeditions grant. They used the outline to divide the work, with each PI contributing text and figures where appropriate. Then a smaller number (guessing the lead PIs) integrated the text and circulated the document for feedback.

The next person to speak was from the DoE. I didnt personally get quite as much out of this talk as the others, I am sure that was due to my interests. What I did take away was that the DoE has lots of opportunities for computing, ranging from architectures, systems software, operating systems, programming languages, as well as the fields that make up computational science and engineering. If I was surprised, and perhaps I shouldn’t have been, the DoE is also focused on networks and remote collaboration tools to support distributed science.

The next person to speak was from DARPA. He talked about how to win (a DARPA contract).

In order of priority, he began with ideas matter. There’s a paper called the Army Capstone Concept that potential investigators should read. He asked the community to aim higher and bolder. He didn’t speak to this point, but I thought I saw on that slide it also said that the idea must be doable. The previous DARPA plenary said that it was alright to aim high and fail (at least initially) so I’d have liked to know more about doable. Second, it must fit the DARPA mission. Third and fourth were cost realism and the proposers’ capabilities and related experience. With respect to related experience he emphasized more than once that it should not just be your stature, but actual experience. He also said that it sometimes helped to write your proposal in parts, with budgets for the various parts because that can help in contracting (they may ask for some but not all of the parts I inferred from this, so modularity is advisable). Finally he emphasized engagement with the Program Managers, before the BAA and after the grant is awarded, he also reminded the audience that they read a lot of proposals, which I took as a reminder to make it engaging and interesting to read.

The next speaker came from the University of Michigan. He provided the experience of someone who has run large centers, and therefore has been successful in raising money for them. He did a great job of providing the faculty/lead PI perspective as well as suggesting what department chairs should do to help faculty who want to write large grants. He began by saying that not all faculty are interested in writing large grants and that in his opinion it’s pointless trying to encourage everyone to do so. Instead, find those who are willing and support them in doing it.

He argued that the reason to write large grants is the visibility and impact for both the individual and institution. Another value he highlighted was that a large effort can create a locus for other activity. So, a large center can spawn and facilitate related research efforts. But there are challenges. One is interdisciplinary. Not just external to CS but also among the specialities of CS. So all the challenges of doing interdisciplinary work apply. Another challenge is that you have to have complete coverage in the space you are proposing around, you have to plan for it from the beginning and the PI has to ensure that any gaps are filled, even if he gaps that need filling are not attractive research to the person that takes the ultimate responsibility. I had the impression that what he was saying was that one of the responsibilities of the PI was to fill those gaps.

So what can department chairs do to help? Reductions in teaching load, staff support, and institutional support. And recognize that the time spent can diminish ongoing research activities. Also since it requires resources, pick the best opportunities. And if you are successful recognize that you get a part of the action, because the large projects will span units and institutions.

Finally a CISE NSF person spoke. She mentioned two large center programs the ERC and the STC, as well as a center-like entity, university-industry centers for partnership. CISE has a center-like program called Expeditions, the current round of which will be announced next month. There’s no restriction on topic, but it should have impact on CISE, society, and possibly the economy. They look for something where the whole is greater than the sum of the parts, if they could have funded ten small projects instead of one large one it is not compelling. Expeditions was partially designed to fill a gap created by DARPA (but there was an observation that it might now be a gap that the new DARPA is filling).

Expeditions is also a mechanism to engage the Computer Science community to engage more in center like activities. The NSF representative observed that Computer Scientists participate less in (I presume this means lead) less ERC and STC centers than other disciplines. Expeditions is a launch pad for potentially taking things to a center activity when the Expedition is done. An interesting note, the number of submissions has dropped massively for Expeditions, 68, 48, 23 in the three years that it’s been active. Finally she noted that some Expeditions had lead PIs that were not Full Professors, noting that Assistants and Associates did succeed with these efforts.

Snowbird: Democratizing Innovation

Just finished listening to a talk by the deputy director of DARPA. It was one in a series of talks about the “new” DARPA, which in this case was positioned as one that’s going to align more effectively with the culture of Universities.

Much could be said, but I want to focus on one aspect of the talk. One of the thrusts within DARPA is focused on understanding what social networks make possible. He talked about the Iranian election and how technologies were used to mobilize people in protest. This was part of a discussion about democratizing innovation.

What is that? My understanding is that it’s a focus on how social networking technologies make it possible for large groups to mobilize around shared interests (ideological, political, religious, entertainment) that are not related to geographic borders. Technologies are creating new borders, new edges, that DARPA needs to understand.

And this reminds me of Computing at the Margins and Global Development. We need to understand these edges too… It’s not just building technologies for those who have none, but leveraging what’s already in use to develop it further. But, I think that we’re also very actively attempting to change the boundaries, by bringing more people into the digital society. I’m still pondering the implications of this, while listening to the DARPA director discussion human motivations and the need for sociologists and so forth to understand how social networks work.