Beki Grinter

The Politics of Metrics

In computer science, discipline, empirical, research on May 27, 2009 at 11:50 pm

Today we were discussing some ideas around a new research space.  I suppose we’ve always been engaged in two types of science, the science of intervention (the design and deployment processes) as well as the science of assessment (how do we know whether we’ve succeeded).  But, it was the latter that made me think about assessment… it brought back some memories (including the discovery that a building I worked in is now the Headquarters for a large Office Supplies chain)… wish we’d had their access to stationery when I was there.

Anyway.

Many years ago, a colleague and I conducted a study of a corporate metrics program. Metrics were used by the corporation (as many/most/all do) to understand the overall product portfolio. These metrics gave senior management an overview of where each product stood in terms of its release time, allowing them to make decisions about revenues, and so forth.

From a research perspective what makes metrics so interesting is their role in accounts of the company. Metrics became the distillation of all the nuanced work it takes to make software work, the work of 100’s of people, distilled down to some graphs. Metrics made it possible to compare, to rank, to judge, … and so metrics far from being straightforward numeric assessments of a “ground truth” became far more complex.

We saw how the purpose of metrics became attenuated when crossing organizational boundaries. What seemed so crystal clear to the people who sought the metrics, became increasingly less clear as you moved away from the source of the request, and closer to the people who would fulfil the request, i.e. those who would produce the metrics themselves. Even simple metrics had this feature, they lost their rationale in translation. The loss of the original intention was replaced. It’s very human to infer meaning to action. So metrics acquired other explanations, based on the local culture and also the experiences that those cultures had had with previous metrics initiatives. (Obviously, this is especially bad if metrics have been used to isolate “poor performing” projects and then cancel them).

Our original study took a few months and was an interesting journey into the company. What we then were tasked to do about it was something that took several years. Just proving that if you point out a problem you should always be aware that you might get asked to fix it.

So, why was I reflecting on this. Well today in a conversation about the science of assessment I started to think about these experiences. I started to wonder about the politics in the assessment methods that we use… wondering whether there’s more to be done in examining the politics of these methods. Just a thought…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: