MOOC Participation: Diversity and Assumptions of Development

Continuing my series of posts about MOOCs. Today’s is about a type of open/development rhetoric I keep hearing associated with MOOCs. It’s well meant I am quite sure, but I’ve heard the following sentiment: MOOCs will allow anyone from any continent to access content. And that in turn leads to increased education, skills for all.

I have a number of problems with this argument.

Starting with the obvious, this sentiment makes important assumptions about access. That access to the Internet and its content is uniform across the world. But it’s not. The Internet is a very different experience if you have a smartphone as your only means of access, versus if you have a laptop. Behind the hardware, there are questions of corporate policies and pricing mechanisms that influence access. Bandwidth caps, bandwidth pricing can influence how people use their phones, and in many parts of the world also how they use the wired network.

Behind these crucial practical questions of access lurk other assumptions, which warrant questioning. Is the content we create relevant or useful for everyone? What assumptions do the producers of content make about, say, what has been previously taught? What assumptions are made about the types of hardware and software the students have access too? And most critically, what assumptions get made about why the person is taking the course and whether that content will ultimately be most useful?

Although its not used too much, I have heard the word “Africa” used to describe diversity. I do think its well meant but it has the danger to collapse all of these questions into a stereotype of a person. Africa is not a person, nor is it a country, it’s a continent of great diversity in all senses. A person from Africa may well contribute to diversity in a MOOC setting, but so might a person from America.

Like others, I see this as being part of understanding the participation divide that shapes the Internet today. Some of that divide is the question of access, its costs, modalities, and so forth. But that’s not all that shapes the participation divide. When we overly simplify an entire continent we close down the question of what shapes participation in very problematic ways. If we are really committed to understanding how online education might help more people learn, the participation divide is precisely the question we ought to open up, to really take account of the highly diverse population of people that have some reach to the Internet. Because it’s only when we actually take diversity seriously that we have any shot at getting to something better than more education for the already well educated.

Knitting Needles dont Knit, People Do

I keep hearing this line about guns. Guns don’t kill people, people do. So I thought it would be interesting to explore the argument via knitting needles.

I knit, I create knitted artifacts. But, the knitting needles I use are pretty crucial to the experience. It’s not impossible to knit without knitting needles, I’ve tried with chopsticks, it’s possible but not as satisfying. You can also use the knitting needles for other, non-knitting things, I’ve used mine to tie my hair up. But they are better for knitting than as hair decorations.

Knitting needles shape the experience by being very intentionally designed for that experience (e.g., the different thicknesses suitable for different thicknesses of yarn, circular for working knitted objects in a round, double-pointed for socks, as well as the traditional straight needles). Knitting needles are designed to help people who knit knit. Without them people could knit, but the experience of knitting with knitting needles is the most common one and it’s not surprising, they were designed for it.

Beyond the design/function argument there is something else about knitting needles and knitting. When I have knitting needles in my hands, I am visibly a knitter. I’ve written before about the types of conversation that that starts up, about how to knit, what I am knitting, recollections of family members who knitted. It makes me a part of a world in which I am seen as a knitter, and in which others are a canvas of potential knitters or people who are curious. Just the other day I was knitting at my Godson’s school play, and so was the person sat next to me. Not only did we have conversations about our favourite local yarn stores, but we also received joking commentary from others about “keeping the knitters together.” I still don’t know her name, although I do know the name of her granddaughter who was also in the play (and about the same age as the children in the shooting that has triggered this reflections on knitting). Sometimes the associations are less amusing, I fly with knitting needles, its allowed, but it doesn’t mean that others on the plane don’t look at me, and the needles as if they are weapons and I am potentially a risk. Context matters, its uncomfortable for me to be seen as a terrorist risk when I knit on a plane, but it’s a space where contexts transform the meaning of the technology.

When I knit the technology that helps me do that is knitting needles. It changes what I can do, as well as supporting me in that, but it also changes my relationship to the world itself. I become associated with my needles. So, I don’t think you can separate guns from people, because you can’t separate the needles from the knitting.

The Autobiographical Turn

There has been a turn towards the autobiographical in some ethnographic research. The idea is that by sharing one’s autobiography—the relevant parts—that it will make it easier for the reader to understand the analysis process. Understand where, you, the analytic instrument starts.

When I first learnt about this I thought that that was very valuable. I thought that it would be useful to understand something about where the author stands with respect to the material. I also knew from experience in studying religion that people made various assumptions about my beliefs (ranging from atheism to fundamentalism). Realizing how any position along that continuum could be applied to my motives for the research made me think that putting something clearly out there was potentially very useful.

But lately I’ve been thinking that there is a problem. Putting something about yourself into a research publication puts it into the professional public forum. Most of the time we spend in professional forums is very carefully managed to create a good impression. But is that what we bring to analysis? If we bring parts of ourselves to analysis, is it more/as likely to be the far more complex experiences of our lives? Do the experiences that shape us mostly deeply come from the types of things that are easy to share or are far more complex and not something that we would choose to put into the professional domain? More  cynically, I began to wonder whether sometimes the autobiographic turn was being used in pursuit of professionalism (but that was me at my most cynical).

Back to the question of my religion. I have wrestled with writing about my religious position, and in the end I find that I am relatively uncomfortable in writing about it autobiographically because to make it useful in any meaningful way I would have to reveal far more and discuss a whole set of choices and experiences that I have little desire to share with the HCI research community. Here’s my religious position, I have no strong position on the question of religion. Of course that’s very convenient because it fits nicely into a professional position—the type of distance is in line with ideas about how empirical science is conducted). Also in the absence of knowing far more, doesn’t seem to be helpful. It’s a nice way (of course) of saying “its complicated.”

And now when I read these autobiographical turns I find myself asking two questions: how is what you are telling me tied to the professional image that you are trying to project, and what is being omitted as a consequence.

How to Outwit Your Thermostat and Other Tales from the Future

Having lived with an Internet scale that will tweet my weight if I want (what planet did the designers inhabit before they moved to Earth) and mastered the art of not allowing it to do so, we are now experimenting with “smart thermostats”, i.e. the products from Nest. Let me say from the outset that I’m very excited about these. They look beautiful. Gone is the old white thermostat with its “I’m an 80′s calculator, or perhaps even a wrist watch” interface. This has a turning dial, it’s smooth, it glows blue when cooling and red when heating. It also tracks our energy usage, so I can see how many hours a day we are cooling our house. Also because its connected to the Internet it knows how hot it is in Atlanta and uses that to infer whether we’ve possibly used more energy to cool today because the weather has been warmer.

So that’s all working very well.

What has been more interesting is that the Nest is trying to generate an automatic schedule for us. What I mean is that based on the way we set it when we am here, the Nest has been building up a data corpus that it’s now using to control the settings in the house, including settings for when it thinks we are away. You can turn this feature off, but we thought it would be interesting to experiment with it.

This works well when we are on a regular schedule, but the summer for academics is not always routine. So a new feature for me is when I am home trying to pretend, at least to my thermostats, that I am still away. One way they detect that I am home is through movement, so I have found myself in the bizarre situation of attempting to sneak past my thermostat in order to get somewhere without it knowing. I’ve been mostly successful.

I also find myself thinking “what are my thermostats doing?” I hope that this will wear off with time, but while they are still learning I wonder what they are getting up to at home while I am away. Fortunately Nest has an account, you can log on, download the iPhone/iPad apps and while away those boring moments in meetings by checking in on your thermostats. I’ve had to turn mine up and down several times, especially in the early phases when they didn’t understand my schedule. I have a thought for a really good rouse which is to set the temperature on someone while they are at home. Do I think that the Home Office is set too low by its current occupant, now I am empowered to change it on them. Bhwaa haa haa…

I’m pretty happy with the Nests, I’m enjoying learning more about my energy usage (although since I can’t compare it with, say my neighbors, I’m not sure whether I’m doing better or worse than others). But, I am reminded about how with each innovation in home automation, so I’m adding another little to-do into my life. So, I’m balancing its sensible aspects and adapting some of my behaviours (like sneaking about my house) in order for it to make sense of the routines I want it to know about, not the ones I don’t.

Doing Ethnography without the Ethnographer

While catching up on some reading, I came across some references to consumer research firms’ efforts to do what I can only describe as ethnography without the ethnographer. There are a variety of ways in which this is done. Blogging is one, have people write their own stories about experience in order to understand it better. On the back end use some tools to distill it. I’ve been wondering what the implications of this are ever since.

First, it seems to me that this is part of a trend to make a retail version of a professional knowledge. The migration of ethnography into corporations as their “secret sauce” initially followed the “hire anthropologists or sociologists” model. But it didn’t have too, and apparently it isn’t always following that model any more.

Second, it looks to me like a type of deskilling. If the methods of ethnography (and I don’t just mean the data collection but also the data analysis) have been broken down into their component parts and the ethnographer replaced by technology that seems like a classic case of deskilling. I’m reminded of an article I read in Scientific American a long time ago now by Joan Wallach Scott who wrote about the processes of deskilling as breaking up work into component parts so that it could be migrated from men’s work (in its richest forms) to women’s work to the work of machines. Machines were the sign that the work was at it most routinized.

I’m troubled by this of course, its the type of work that I don’t think can be dealt with in this way, but it is and I am paying attention to the future of it.

CHI 2012: Reviewing

I attended a few sessions devoted to discussing reviewing for CHI.

In the end I feel that there are two “camps” of ideas for improving the reviewing process and I do not think that they are reconcilable.

One set of suggestions I heard was to conduct experiments with papers and reviews. Several were mentioned. For example, take papers and their reviews and then have other people review them and see whether you can come up with same set of reviews. Another set of thoughts are around the generation of reviewing metrics. Metrics about how long a reviewers review is, how timely they are, and so on and so forth with the goal of creating a record of their behavior that can be used in the future to assess their reviewing ability. Behind these, and other experiments, seems to me at least to be a firm belief that reviewing should be treated as a quantifiable science.

But, then there are counter arguments.

For example, Danyel Fisher made the very astute observation that averages are a relatively meaningless concept in reviewing, even though we make use of them. As he put it, a score of 3 is not the same as a score of 5 and another score of 1. But when we average that’s what we turn those scores into. And he made me reflect on how we can and do talk about the scores in this way…

Jeffrey Bardzell makes an equally compelling case that reviewing is not a science with a comprehensive and  fantastic series of articles (1,2, and 3) in which he argues that it is a process of providing expert judgement. Danyel and Jeff are both, in my mind, getting at the same thing, which is that reviewing is a subjective act, based on expertise and such both its processes and its outputs should be understood and treated in such terms.

And it doesn’t stop with reviewers and ACs. Being a Program Chair is also a matter of expert judgement—one of assigning papers to AC’s and reviewers. Making decisions about how to compose the program committee are all not matters of science but of judgement.

I think the reviewing as science model is doomed to failure, and along the way it will create more work for everyone involved as we try to pursue a set of metrics that do not accurately characterize the work that we do, but become a substitute for it, with all the problems that that can bring. I think we need to take up more seriously the question about how we come to think of ourselves and practice a critical review practice based on a belief that we are experts not participating in a scientific process and what it means to handle not just the process but its products in those ways.

Dual Body: Question about Women in Academia

I saw this post on Mark Guzdial’s blog. The last point intrigued me

About 82% of technical women have a partner who works fulltime, compared with 37% of technical men. (Anita Borg Institute)

I wonder what the ratios are for women in academia? I wonder how many have a partner that also works in academia as well as having a partner who works, and how that compares with men in academia. Also I wonder whether it varies by discipline at all. Does anyone know?

H-Index: Some concerns

This is one of those posts that’s been coming on for a while. I’ve written in the past about my concerns about metrics including in academia. I’m beginning to have those concerns about the h-index. I’ve written before about whether I would pick the same papers to be at the top of my h-index but I want to write about two other concerns.

First, I am concerned about what gets lost when we talk about an h-index. The h-index is a measure of a paper’s impact grounded in citation counts. That certainly is one way to measure impact. But one thing that is not counted by this metric is the presence of papers in a class. What is the impact of research publications and results on classroom teaching and activity? That’s just one example of a type of impact that a paper might have that is not captured by the h-index. The h-index is a partial measure of impact, it only measures, impact that can be seen in citation practices.

But that partial nature of the measure is lost when we talk about the h-index as a number. The number becomes significant. And the number is comparable: my h-index is higher than some and lower than others. Reducing impact to a number risks reducing conversations about what impact might be to those that can be conveyed in a number. We lose context when we substitute the numeric h-index for impact.

There’s a list of h-indexes in Computer Science. The very presence of this list is exactly the type of reification of the number as being what matters and using it to rank order people. This is a list of people as measured by that partial measure of their impact, that which can be seen in citation indexes, and we can all ask whether that’s a useful, appropriate ranking, and if so of what and why?

But, back to the list, and my second concern. To get on the list you have to have an h-index of 40. Is that good? I suspect that it was once and soon it will shift. As the discipline matures, more people will probably have h-indexes of 40 making the list not very helpful for narrowing down the field. And herein lies my second concern, I wonder whether the h-index is subject to grade inflation. Do we know what good even is, how long will it take to calibrate not just for the field of Computer Science but also to a field that is changing in composition as it matures. And, what does it mean to routinely use a metric that may be in transition?

And none of this would trouble me if the h-index wasn’t so seductive. Things like the h-index share a lot in common with the dominant beliefs about the nature of truth, particularly for scientists. It’s an externally derivable “fact”. That gives the h-index real power. But asking what kind of fact it is, what truth it represents and what gets lost and how all seem equally as important when dealing with the h-index.

The Role of Free Will in HCI

I have been wondering about the role of free will in HCI research lately. It’s a statement of the obvious to say that there are many different theories that inform HCI research, and those theories make different assumptions about knowledge and truth. And sometimes when I read or listen to conversations about those theories, and the methods associated with them, I hear talk about choice. Most specifically that we can choose the most appropriate theory for the research that we want to conduct.

But can we? Can we really choose among them, is it that simple? I am not so sure. Perhaps it’s just me but I find myself drawn to theories and methods that are commensurate with values I hold. I tend to choose things that produce results (even surprising ones) that I find compelling.

I should say that I am not opposed to others using methods that do not align with my values. In fact, I find the resulting scholarship quite interesting. But I also think I tend to be drawn to those papers in ways that take the results and use them to ask questions that are answerable using methods and theories that align with my values.

As HCI reflects on its methodological and theoretical plurality, I would like the field to reflect on how it talks about those methods and theories and whether we are in fact free to choose, and how free we are?

It’s all in the way you say it…

I was reading a paper when I came across the following sentence…

We intentionally biased our data sample in terms of type and size

There’s so much going on here but lets start with the high order bit that saying this in a paper might as well be accompanied with the following sentence

Please reject this now.

Lets start out with the statement, what sounds good about intentionally biasing our data sample in the following ways? Well I did have one thought, it’s better than unintentionally biasing it (which just seems careless). No the authors knew what they were doing. And, also a plus, they admitted it in case the reviewers didn’t know what they were doing. Whoo hoo, I would have given up as a reviewer and just written up “The authors admit that they’ve conducted a flawed experiment.”

Moving past the idea that the authors are flaunting the rules of experimental design this phrase raised other questions.

The paper in which I found this sentence was a qualitative piece of work. For example, one question, what is bias in qualitative sampling? In fact, some forms of sampling are the pursuit of quite intentional people. People with a particular expertise for example. (Imagine your three cycles into your Grounded Theory and you have some very particular questions that only a few people in the corporation you’ve been studying can answer because it falls within their job responsibilities, well then you’re either going to select these people to interview or you are going to waste a lot of time trying not to be selective in who you select to talk too).

Questions about size can be complicated as well. Size often suggests a numerical size but as I’ve said before, 12 does not equal theoretical saturation (tip: having a fully worked out theory does).

Behind these questions lie a type of care with terminology. The authors talk about data samples, bias, type and size, often words applied to experimental design. These are not the right ways to talk about qualitative research. Sure you want to talk about who you interviewed or observed, your participants, but they are rarely a data sample, they are the people that led you to the collection of a particular set of data… the logic of who they are is not about sampling from a population to ensure coverage, but about selecting people who can help develop the theory or analysis, and the size is dependent on different ways of determining completion.

I know that this sentence was thoughtlessly written. It was honest, but it sets up the reviewers in a variety of ways as I hope I’ve pointed out. And if I have to put it crassly, don’t use experimental terms to write about non-experimental ways of conducting empirical research. It’s just ugly.