Beki Grinter

It’s all in the way you say it…

In academia, discipline, empirical, HCI, ICT4D, research on April 9, 2012 at 7:02 pm

I was reading a paper when I came across the following sentence…

We intentionally biased our data sample in terms of type and size

There’s so much going on here but lets start with the high order bit that saying this in a paper might as well be accompanied with the following sentence

Please reject this now.

Lets start out with the statement, what sounds good about intentionally biasing our data sample in the following ways? Well I did have one thought, it’s better than unintentionally biasing it (which just seems careless). No the authors knew what they were doing. And, also a plus, they admitted it in case the reviewers didn’t know what they were doing. Whoo hoo, I would have given up as a reviewer and just written up “The authors admit that they’ve conducted a flawed experiment.”

Moving past the idea that the authors are flaunting the rules of experimental design this phrase raised other questions.

The paper in which I found this sentence was a qualitative piece of work. For example, one question, what is bias in qualitative sampling? In fact, some forms of sampling are the pursuit of quite intentional people. People with a particular expertise for example. (Imagine your three cycles into your Grounded Theory and you have some very particular questions that only a few people in the corporation you’ve been studying can answer because it falls within their job responsibilities, well then you’re either going to select these people to interview or you are going to waste a lot of time trying not to be selective in who you select to talk too).

Questions about size can be complicated as well. Size often suggests a numerical size but as I’ve said before, 12 does not equal theoretical saturation (tip: having a fully worked out theory does).

Behind these questions lie a type of care with terminology. The authors talk about data samples, bias, type and size, often words applied to experimental design. These are not the right ways to talk about qualitative research. Sure you want to talk about who you interviewed or observed, your participants, but they are rarely a data sample, they are the people that led you to the collection of a particular set of data… the logic of who they are is not about sampling from a population to ensure coverage, but about selecting people who can help develop the theory or analysis, and the size is dependent on different ways of determining completion.

I know that this sentence was thoughtlessly written. It was honest, but it sets up the reviewers in a variety of ways as I hope I’ve pointed out. And if I have to put it crassly, don’t use experimental terms to write about non-experimental ways of conducting empirical research. It’s just ugly.

  1. This seems a little harsh.. the terms you identify as being exclusive to experimental design are found elsewhere – most obviously in social survey work – where what you complain about is called a ‘purposive sample’. One famous(ish) example of this is the ‘affluent worker’ studies of Goldthorpe, Lockwood, Bechoffer and Platt where the authors decide to investigate the ’embourgeoisement thesis’ (that working class people are becoming middle class) by looking at Luton, reasoning that if embourgeoisement was happening anywhere it would be happening here.. and, of course, all predictive voting studies tend to be purposive, stratified samples. As for sample size, the numbers argument, we could have another argument about that, – whilst I understand the vague unease people have with small samples (however they might be defined), once you have gone past your sample of one there is generally no logical stopping point until you have reached 100% of your population, since the argument then becomes to what extent do these form part of a class? (which is a different argument than mere numbers) and your twelve, rather than constituting a class may merely be twelve individuals, that is, twelve one’s.. and each ‘one’ is just ‘another one time through’ for the analysis and then what matters is how thorough the first time through is..

    • Perhaps you’re right about the tone. You don’t see so much survey work in HCI of course… but I still think that its important to pay attention to terms.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: