Scholarly Debates: Revisiting Ethnography Considered Harmful

Recently, a colleague of mine Rich DeMillo blogged about Moshe Vardi’s most recent editorial in the Communications of the ACM. For those people who are not familiar with the Communications of the ACM (CACM), it is the monthly flagship magazine of the Association for Computer Machinery (a professional society to which many/most Computer Scientists in the United States, and increasingly across the world, belong).

I agree with Moshe Vardi that we, Computing professionals, could use more debate. I like the point-counter point series that was started in Communications of the ACM. My guess is that for some types of debate the columns are too short, but I’ve enjoyed the ones that have appeared and I feel it helps me understand a bit more about where some of the challenges in Computing reside. I also think written debate as part of the process of developing and refining knowledge leads to a richer more nuanced understanding of the subject matter under discussion.

But, I find myself even more in agreement with Rich’s response. Rich’s response raised a number of points, and reminded me again of how debate requires consideration, not just of the content, but of the mechanisms by which that debate occurs. And that’s what I want to focus on because if we are going to debate we should do it with decency and dignity.

Last year, the HCI community had a similar “learning how to debate” experience that involved a paper called “Ethnography Considered Harmful“, also positioned as a debate about whether non-ethnomethodological forms of ethnography made contributions to the field of Human Computer Interaction. A quote from the paper:

Closer to home, where home means that the studies take place within the researchers’ own country of residence, we find that such settings as the American Mega-church have been accorded a similar treatment, e.g.,

“ … the first thing we noticed when entering sanctuaries were the large screens on both sides of a central stage … During services screens were used extensively to accompany music, illustrate sermons, and share announcements and video. Words to hymns and Bible verse were frequently displayed over a background depicting religiously-inspired imagery. It was also common for the pastor to read selected Bible verses that would then be displayed on the screen. We observed parishioners following along and looking at screens to know what verse to turn to in their Bibles. The appearance of a new verse on the screen was followed by a flurry of paper turning.”

While ethnography has its roots in the study of exotic settings both abroad and at home, and while these sorts of study do illustrate the diversification of digital technologies in everyday life, it is imperative that ethnography provide more than surface descriptions of action and interaction. The danger is that adopting what might be called an ‘exotic tales methodology’ in design will result in descriptions that offer up little more than ‘scenic features’ of action and interaction for consideration, thus sensitizing designers to little more than the grossly observable features of a setting or culture.

The paper that they quote from in the second paragraph is one of mine, and I freely admit that I don’t share their view that it was surface description. The rest of this post, I will try to make a case not about content, but about process. Obviously I disagree with the content, but I will try to refrain from that discussion here.

The paper of mine that they quote from and use to discuss is an interesting genre of submission to the ACM Conference on Human Factors in Computing Systems. It’s not an archival submission, it’s actually something called a “work in progress” which I have always taken to mean exactly that. The resultant “publication” is not archival, copyright is retained by the author. So, I have always interpreted them as an opportunity to get early feedback on something that is, well yes, a work in progress. (Also, if you look at  one, you’ll see that it’s not a lot of words, so I take that as another sign that its not meant to be final, rather an outline and some ideas that will be refined or even rejected through discussion).

I’d like to think that the confusion about the intent of this work in progress could have been avoided if the authors had engaged in a discussion with me about their upcoming paper that was going to debate the role of my work in progress for the field. Debate implies willingness to discuss with people who hold alternate positions. Perhaps if the authors had sent me a copy of their paper as they were writing it I’d have told them that I thought critiquing a work in progress as a final result was weakening their argument. I could have pointed them to final versions of this work. Also, perhaps in talking to me about my surface descriptions, they’d have found some words that would have helped me to understand their points more clearly. (To be honest, I take surface description as an insult, and find it hard to see beyond that slight to what point they are actually trying to make).

The role of the review process also came up in the Vardi-DeMillo context. Ethnography Considered Harmful made it through the review process. And the CHI Conference has a long and careful review process. But reviewing is a human process. There’s been a lot of discussion about the CHI review process, it happens each year. I personally have always blamed the randomness on the size of CHI. CHI is huge and growing at a crazy rate. Submissions climb each year. The number of reviewers required to review also grows, when I was Papers Co-Chair in 2006, it involved 2500 discrete reviewers, and I would expect the number to have passed the 3000 mark by now. I believe that there’s a lot of goodwill in the system but at that scale, things happen.

Again, I think that dependency on the review process might have been mitigated by the authors contacting any of the people whose work they sought to debate. A colleague of mine forwarded me the paper after it was accepted to the conference. Like me, the person who forwarded the paper to me had their own research similarly characterised, and it turned out that there were quite a number of us. What was common among us all was that none of us had been contacted during the writing process. What type of debate begins this way?

Finally, and perhaps this is just me. I think it’s healthier when debates are among peers in the community. Debating requires having a voice, a voice that matches that of those who hold different positions. Why, because the debate will be better, it will be richer, because it will be among peers. Some of the people’s work that the authors of Ethnography Considered Harmful held up turned out to be scholarship by newer members of the community (either through transition into HCI or as students). I am not saying that students and new members should not be engaged by debate but I think that debate requires even more thought and respect when it involves exactly the type of people we would like to recruit and retain because of the new ideas, fresh perspectives and so forth that they bring.

I’d like to engage in debates, I think there’s much to be learnt from discussion, but I’d like them to be civil and conducted around a set of principles that not only respect the products of our discourse but also the scholars that produce them.