Metrics: Numbers and Processes

Surprise, surprise, another post about metrics. Its not just the numbers themselves that can be problematic, but here’s a recent encounter with the processes used to compute the numbers.

Some time ago, Paul Krugman wrote this:

As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. (Full article).

I love this phrase, and I tend to concur with him. Perhaps its because he has a Nobel Prize in Economics (2008).

But, I don’t think you even need to have very impressive-looking mathematics. Perhaps in Economics, perhaps to justify research outcomes the nature of the mathematics matters. Not, however, for a variety of metrics.

I was recently asked how I computed my h-index. While many people use tools, I compute mine manually. The mathematics of this is not complicated, although the truth is.

R. Grinter and R. Grinter are two different people. Some of my citations omit the E. that I use (Rebecca E. Grinter). This can, when not caught, lead to grade inflation. One way to catch it is by looking carefully for any focused on spectrosopy. But, even taking those rogues out you can see the consequences of interdisciplinarity on the h-index. Many of the tools let you refine by discipline, but while that removes most of the Chemistry, it doesn’t catch the interdisciplinary pieces that the other R. Grinter worked on. More over, if you get too restrictive, R. E. Grinter and of course my nemesis mis-cite, R. Grinter get lost categorised into other disciplines. Finally, there’s my nick-name. Beki. Happy to be Beki, but people will pick me up as B. Grinter. So, I usually need to pay attention to that. In retrospect I wish I’d just started with B actually, because as far as I can tell there are no other B Grinter’s currently research active.

So that’s why I compute my h-index manually, I do to provide a truth, one that reflects my awareness of the flaws associated with the tools. But, by computing manually I’ve separated my h-index from the one computed by tools, and thus to compare mine to others requires computing all the h-indexs manually. And comparing is frequently at the heart of metrics. And this to me seems like another fine example of the problem of numbers as truth. In the process of trying to supply truth (an accuracy through manual computation) I’ve simultaneously taken away another type of truth, one that is comparative based on using the same process to compute that “truth.”

New Forms of Impact

This article came to my attention via several different sources.

I want to say from the outset I agree with a number of the author’s points, and clearly so do others given that its a report from a conference and also has been reposted in several places.

But here’s the open question: how should we evaluate non-traditional forms of impact? The easier task is to argue that a problem exists, the harder one is proposing alternatives. Take the case of blogging. Reader counts are attractively numeric (always good for a metric) but it doesn’t answer the question of who the people are? Nor does it address what forms of impact that reading the blog might be having? Is it enough that it’s a completely unknown but large group. Should that group have to *do* something based on the post?  One thing that is rather nice about the traditional peer-review citation practices is that it’s concrete (we can compute the h-index if we chose), but we can also see the impact of our own work in that of others.

Perhaps we don’t want to reduce it to metrics. But I think the question that’s still on the table is what are these new forms of impact, where do we find evidence that something has happened? (And another set would be can everyone agree that this new form of impact is an appropriate form of impact — another open question, but without agreement I don’t think the form of impact will “stick”).

Why I Wish to Keep my Teaching Comments Out of My Evaluation

I’ve written a lot about metrics in the past, today my focus is on how qualitative data is generated and the implications for evaluation. I am aware that my management (and I use that term deliberately since this is an evaluation situation) want to see the comments that students write. They currently only see the numeric scores. Their argument is that the comments would enrich their ability to evaluate my teaching.

But, I find myself very resistant to the idea.

First, how do comments shed light on teaching? How do the comments, often typed out hastily in the throws of week 15 of a 16 week semester explain the ebb and flow of the class, the work I did to bring the class together, to draw the timid into discussion, to manage the differences in perspectives among class participants, to listen and counsel the students who brought them problems not related to the class but to their lives and their struggles and joys? These are subtleties of the experience that I’ve never seen in students’ comments. Not surprising, they’re not teachers! Teaching is an intimate and deep experience, one that can only be truly understood through experiencing the classroom. I realize the desire to measure it, but teaching evaluations are only partial instruments hence the ability to improve the scores without improving the actual teaching. Adding comments won’t change that.

Second, I have a particular concern as a woman. I am sure I am not alone in having comments about my body as part of the feedback. It’s tough enough knowing that as a woman my body and its “problems” is a part of the students’ discourse. But I accept that to be young is not always to be thoughtful or kind, and I teach despite that, knowing that I get to keep those indiscretions out of the professional discourse about me. While I respect my all male management, I find the idea that they can read remarks about my body embarrassing. It transforms an annoying inequity confronted by female scientists into a public humiliation.

And that’s why I don’t want my teaching comments made public.

Dress to Improve Teaching Scores

Another metric that has recently gained popularity at Tech is the teaching score. Teaching scores always mattered, but lately I’ve heard more about them. Teaching scores are of course a measurement of our teaching success.

So, I was delighted to learn that without doing anything to improve my teaching I can do something to improve my teaching score. Dress better. I thought it was a joke, so I looked further, and found an article on a variety of ways to improve teaching scores without doing anything about teaching. And here the dress advice is to dress accessibly. It took a while, but I think this is reconcilable with the first advice. I think it’s too dress in such a way that the students respect you, think you know dress codes, but I wouldn’t try to over-do the dress like them (despite having a real temptation to wear pyjamas like some of my students do to class).

At Georgia Tech the t-shirt I get the most compliments about is the space invaders one I own. It’s always a hit in the classroom and on the Tech Trolley were it’s a conversation starter. And so I’ve solved the problem of what I shall be wearing on the first day.

Once again I am reminded that just because you can measure it, it doesn’t mean you should. Or perhaps in this case, you should measure it but then recognize its limitations as a metric explicitly so that everyone knows how much attention is being paid to this metric. And of course issue dress citations to low performers.

People are not Numbers

I think by now you all know my concerns about metrics programs. Metrics work by reducing people to numbers. Sometimes this is through products (e.g., software development) but since those are human-built this is also a metric. We can collect it and measure it, and so we do (and I think perhaps engineering oriented cultures are particularly prone to this), but far less often do we ask does that quantity represent quality.

Another way I see this made manifest is in the seduction of studying online social networks. We leave our digital footprint when we use tools like Facebook. The danger is that we confuse the online with the real. Is what we can digitally trace and subject to computation and visualization a good representation of our lives, experiences, desires and so much more? How well, for example, do Facebook check-ins deal with people who go to places that they don’t want you to know about and what does that tell you about location? How many Facebook statuses are lies, and what does that tell you about the truth versus identity maintenance?

And then of course there are the critiques of the economic collapse. Did the increased reliance on mathematical modeling cause our economic downturn? No I don’t think so. Did our over-reliance on mathematical models of people? Well this article suggests yes and it’s not alone in doing so.

Metrics: Just Because You Can Doesn’t Mean You Should

The Chronicle of Higher Education recently posted an announcement that a journal ranking system in Australia has been cancelled. It had caused a lot of controversy. Explaining why, Sen. Kim Carr said:

Sen. Kim Carr, Australia’s minister for innovation, industry, science, and research, announced on Monday that the rankings would be jettisoned. “There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings,” Mr. Carr said in a written statement. Instead of rankings, he said, the Australian system will incorporate “journal quality profiles.” Mr. Carr added that “the removal of the ranks and the provision of the publication profile will ensure they will be used descriptively rather than prescriptively.”

This was also a problem in the metrics effort we studied. But what was also a problem subsequently is that once any metrics program had been used “inappropriately” (in this case to conduct layoffs) every initiative that followed was greeted with healthy suspicion. And why not. Once you use a metrics initiative like that, it’s pretty easy to see why people would be skeptical about anything that followed it. Of course organizations can continue to “enforce” metrics initiatives, and we learnt that when they did that, people learnt how to creatively report and count.

H-Index versus Your Index

The h-index is a metric for assessing the impact of scholarly contributions using the number of times each paper has been cited (until that number is smaller than the number of the paper on a list that starts 1,2,3,… ).

My question, if you had to pick the papers that formed your h-index, or to make it easier are the top three most cited of your papers, would you pick the same ones.

No offense to my collaborators on those highly cited papers, but I am disappointed that a couple of papers that have had more influence on me have missed the list. There’s a paper I wrote with Jim Herbsleb called “Conceptual Simplicity meets Organizational Complexity“. It was a write up of our research focused on a corporate-wide metrics program.

I think it’s the paper I’ve written about most in this blog. Why? Because I think metrics are pervasive and many of the problems we found in the paper appear in other settings. For example, I wrote about the apparent difficulty of computing University ranking metrics, and it echoes so much of what we saw in our research. Frequently there’s a gulf between those who want and decide the metrics and those who are the object of those metrics, that gulf is responsible for poor metrics. And just like the technically oriented corporation we studied, I’ve seen it in the engineering oriented University I am in. We are seduced by numbers, because they are readily computable, but like the professor who asked the question about quality, I hear far less of whether they are the right things to know. Just because you can know them doesn’t mean that they are the right thing to know.

And so I return to the h-index. The reason that this paper is not on the list is because citations are a measure of something, but they are not the most effective measure of personal-professional development. The paper on metrics has been very influential in my thinking, about my research and about how I navigate academia. So, what would be in your top ranked papers and why?

Using the Right Metrics

An article published today in the Chronicle of Higher Education makes a lot of important points. It examines the relationship and value assigned to teaching as compared to research. But, the reason I wanted to blog about this article is the following. It’s a faculty member describing how he raises 3million dollars a year in support of his research.

“Nobody has ever asked me how good my papers were, and I think you would find that universally true,” he said, “They basically say, Well, how many research dollars are you bringing in?”

Metrics are so seductive, particularly quantifiable ones. It appears easier to measure the quantity of money versus the quality of a paper. It’s desireable to assign University’s a ranking, but as the National Research Council learnt, it’s not easy to assign one that actually accounts for all that takes place in a University. Metrics are not a substitute for assessment, but sometimes I fear that their ease of computation confuses the important distinction between the two, and that can be dangerous for all those who are the object of that calculation.

Anonymous Class Feedback

I am always really nervous about anonymous class feedback (Georgia Tech, like most Universities I am sure, invites students to participate in an anonymous survey about the class). It’s not just because my Dean reads the reports, but also because it’s the final evaluation by the students of my ability (or not) to communicate to them. I dread seeing figures and written comments that could suggest I’ve not done an important part of my job of teaching.

Despite my dread, each year I try to encourage the students to participate in the survey. I explain that it really is anonymous to me. I can’t tell unless you’ve been the student who has complained about X during the semester and in the feedback it looks like you’ve cut and paste that email message into the comments field. I believe that it’s an opportunity for students to help the instructor improve the class, and it’s their voice into their education.

Teaching feedback has definitely helped me though.

Take for example the following

This was the first course instructor has ever taught. Her enthusiasm for the subject matter is bloody fantastic, however the organization of the course seemed rather haphazard.

It was an accurate comment (which continued with more details about the nature of the disorganization), and I was appreciative that the student had called out the fact that it was my first class ever. It (and other comments) helped me to revise parts of the class extensively. The mistakes I had made had not been intentional, I didn’t imagine that they would cause the problems they did. Those sorts of comments, which I have had for other classes, remain the hardest comments to read. I hate reading comments that suggest that a decision I made had unintended negative consequences. Those are also the comments that make me wish that the students would approach me during the class and discuss the issue, but I understand that’s not always possible. When I can (sometimes I am a bit stumped on how to take action), I revise the class and in some cases now, hundreds of other students who have taken the class since then have implicitly benefitted from those changes. Anonymous class feedback does work, if it means that classes evolve accounting for things that could, on reflection, have been done better.

Teaching feedback is also an interesting reflection of me as a person. I get comments about the British slang I use in classes (and I have gotten better about knowing when I’ve said something no-one in the room understands), and my apparent ability to get off topic. One year I was even told that I was being benchmarked against another faculty colleague who also had a similar reputation. The competition was apparently which one of us would veer the farthest off topic, how they measured this remains a mystery to me. I also have an “um” that I need to work on, and I should seriously consider how much I walk about (one time I tried lecturing in high heels which rather than slowing me down resulted in me taking my shoes off and lecturing).

Some of the most rewarding comments have been about my sense of humour. It’s always good to know that people think I have a sense of humour. This is particularly gratifying because I’ve never seen it on a teaching statement, but I decided to include some text in mine on why humour is a useful way to take a mental break during a long lecture of difficult or dry material. Interestingly enough a student told me through the feedback that it’s also a great way to remember a particular concept. Something I’ve been meaning to add to my teaching statement. One source I draw on for humourous tales is the many mistakes I’ve made, and luckily I’ve got a set of instructional material that includes some pretty funny fails.

Another set of rewarding comments are those from students who feel that what they’ve learnt has changed them. One of the classes I teach includes observation methods. I received this comment.

I am different today than before taking this class: I am more aware of my surroundings after taking this class.

I can’t easily put into words what comments like that mean to me. Sometimes people who will not likely use the methods or content of the class again tell me that it helped them or has changed them. Wow. Once I got a hand written letter from a former student telling me that something I had taught him had helped him land a job (the interviewer had asked him questions about a particular HCI method, one that I had taught him, and even though he was going to be focused on systems development he had helped the interviewer understand what the method was and when/why it was valuable). And to whomever wrote this, well yes, you made my day (and not just because you spelt favourite correctly).

This was my favourite– that’s right favourite– course that I have taken at GATech.

So despite my dread, I will be anticipating anonymous class feedback. I will be reading it, taking what’s good and cherishing it, and trying to amend and evolve my classes. And secretly wishing that I’d gotten it all right the first time.

The Politics of Metrics

Today we were discussing some ideas around a new research space.  I suppose we’ve always been engaged in two types of science, the science of intervention (the design and deployment processes) as well as the science of assessment (how do we know whether we’ve succeeded).  But, it was the latter that made me think about assessment… it brought back some memories (including the discovery that a building I worked in is now the Headquarters for a large Office Supplies chain)… wish we’d had their access to stationery when I was there.

Anyway.

Many years ago, a colleague and I conducted a study of a corporate metrics program. Metrics were used by the corporation (as many/most/all do) to understand the overall product portfolio. These metrics gave senior management an overview of where each product stood in terms of its release time, allowing them to make decisions about revenues, and so forth.

From a research perspective what makes metrics so interesting is their role in accounts of the company. Metrics became the distillation of all the nuanced work it takes to make software work, the work of 100′s of people, distilled down to some graphs. Metrics made it possible to compare, to rank, to judge, … and so metrics far from being straightforward numeric assessments of a “ground truth” became far more complex.

We saw how the purpose of metrics became attenuated when crossing organizational boundaries. What seemed so crystal clear to the people who sought the metrics, became increasingly less clear as you moved away from the source of the request, and closer to the people who would fulfil the request, i.e. those who would produce the metrics themselves. Even simple metrics had this feature, they lost their rationale in translation. The loss of the original intention was replaced. It’s very human to infer meaning to action. So metrics acquired other explanations, based on the local culture and also the experiences that those cultures had had with previous metrics initiatives. (Obviously, this is especially bad if metrics have been used to isolate “poor performing” projects and then cancel them).

Our original study took a few months and was an interesting journey into the company. What we then were tasked to do about it was something that took several years. Just proving that if you point out a problem you should always be aware that you might get asked to fix it.

So, why was I reflecting on this. Well today in a conversation about the science of assessment I started to think about these experiences. I started to wonder about the politics in the assessment methods that we use… wondering whether there’s more to be done in examining the politics of these methods. Just a thought…