This is one of those posts that’s been coming on for a while. I’ve written in the past about my concerns about metrics including in academia. I’m beginning to have those concerns about the h-index. I’ve written before about whether I would pick the same papers to be at the top of my h-index but I want to write about two other concerns.
First, I am concerned about what gets lost when we talk about an h-index. The h-index is a measure of a paper’s impact grounded in citation counts. That certainly is one way to measure impact. But one thing that is not counted by this metric is the presence of papers in a class. What is the impact of research publications and results on classroom teaching and activity? That’s just one example of a type of impact that a paper might have that is not captured by the h-index. The h-index is a partial measure of impact, it only measures, impact that can be seen in citation practices.
But that partial nature of the measure is lost when we talk about the h-index as a number. The number becomes significant. And the number is comparable: my h-index is higher than some and lower than others. Reducing impact to a number risks reducing conversations about what impact might be to those that can be conveyed in a number. We lose context when we substitute the numeric h-index for impact.
There’s a list of h-indexes in Computer Science. The very presence of this list is exactly the type of reification of the number as being what matters and using it to rank order people. This is a list of people as measured by that partial measure of their impact, that which can be seen in citation indexes, and we can all ask whether that’s a useful, appropriate ranking, and if so of what and why?
But, back to the list, and my second concern. To get on the list you have to have an h-index of 40. Is that good? I suspect that it was once and soon it will shift. As the discipline matures, more people will probably have h-indexes of 40 making the list not very helpful for narrowing down the field. And herein lies my second concern, I wonder whether the h-index is subject to grade inflation. Do we know what good even is, how long will it take to calibrate not just for the field of Computer Science but also to a field that is changing in composition as it matures. And, what does it mean to routinely use a metric that may be in transition?
And none of this would trouble me if the h-index wasn’t so seductive. Things like the h-index share a lot in common with the dominant beliefs about the nature of truth, particularly for scientists. It’s an externally derivable “fact”. That gives the h-index real power. But asking what kind of fact it is, what truth it represents and what gets lost and how all seem equally as important when dealing with the h-index.