Surprise, surprise, another post about metrics. Its not just the numbers themselves that can be problematic, but here’s a recent encounter with the processes used to compute the numbers.
Some time ago, Paul Krugman wrote this:
As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. (Full article).
I love this phrase, and I tend to concur with him. Perhaps its because he has a Nobel Prize in Economics (2008).
But, I don’t think you even need to have very impressive-looking mathematics. Perhaps in Economics, perhaps to justify research outcomes the nature of the mathematics matters. Not, however, for a variety of metrics.
I was recently asked how I computed my h-index. While many people use tools, I compute mine manually. The mathematics of this is not complicated, although the truth is.
R. Grinter and R. Grinter are two different people. Some of my citations omit the E. that I use (Rebecca E. Grinter). This can, when not caught, lead to grade inflation. One way to catch it is by looking carefully for any focused on spectrosopy. But, even taking those rogues out you can see the consequences of interdisciplinarity on the h-index. Many of the tools let you refine by discipline, but while that removes most of the Chemistry, it doesn’t catch the interdisciplinary pieces that the other R. Grinter worked on. More over, if you get too restrictive, R. E. Grinter and of course my nemesis mis-cite, R. Grinter get lost categorised into other disciplines. Finally, there’s my nick-name. Beki. Happy to be Beki, but people will pick me up as B. Grinter. So, I usually need to pay attention to that. In retrospect I wish I’d just started with B actually, because as far as I can tell there are no other B Grinter’s currently research active.
So that’s why I compute my h-index manually, I do to provide a truth, one that reflects my awareness of the flaws associated with the tools. But, by computing manually I’ve separated my h-index from the one computed by tools, and thus to compare mine to others requires computing all the h-indexs manually. And comparing is frequently at the heart of metrics. And this to me seems like another fine example of the problem of numbers as truth. In the process of trying to supply truth (an accuracy through manual computation) I’ve simultaneously taken away another type of truth, one that is comparative based on using the same process to compute that “truth.”