Beki Grinter

A Future for Academia Driven by Metrics

In academia, academic management, computer science, discipline on January 2, 2014 at 5:08 pm

By now, anyone who knows me knows that I am a *huge* fan of metrics. Particularly when they are used uncritically. So perhaps it was inevitable that I would end up in an environment where metrics play an increasingly ubiquitous role: academia.

I want to introduce three metrics.

Student credit hours: a number that measures by class/faculty the number of students a person has taught. You will have a larger number if you teach larger classes. It’s also the number that is at the beginning of a formula that computes the portion of the Institute’s state budget (and presumably how that is divided, although that part of the budgetting process is a complete mystery to me). Higher is better, and in fairness I can imagine that larger classes can create their own organizational structures that need managing and more potential problem cases.

What’s missing in this metric are some other fundamentals about class.

  1. Smaller might be better for the student experience including but not limited to mentoring, one-on-one time with individuals, managing different learning styles… and that this might be exactly what distinguishes a University education at a bricks and mortar institution from an online experience.
  2. Class preparation time, do classes with more students involve more course preparation time. I taught a class recently that was about 1000 pages of reading for 12 people, but it would have still been 1000 pages if it had been 120 people.
  3. The lack of institutional support for say, grading, that larger classes receive.

Research expenditure. This metric measures the amount of money that the Institute receives when a faculty member spends their grant. Again, bigger is better. But this metric assumes that all research costs the same. Not all research costs the same amount to achieve, and funding agencies know that. It does not account for how much it costs to do research.

H-index. I’ve already written about this.

Imagine my joy when someone suggested that we plot all three against each other for an individual. What would that mean? Someone with a larger class, in an area of research that was more expensive to do, and with a high index does well. So, should we optimize (which is the purpose of metrics, to drive behaviour) for large classes at the sake of not giving students the opportunities that come from small ones? Should we optimize for expensive and popular research, and ignore the intellectual, social and political good that might come from less expensive research areas? Should we give even more legitimacy to the papers of an h-index and not ask about the papers that were potentially unpopular but changed a person’s thinking, deepen their intellect…?

Needless to say this epitomizes all that worries me about metrics. The desire to rank and compare, and use numbers to support that is to think uncritically. Sadly, it’s all too common in academia.

  1. You raise good points but I can already expect what the responses will be. To student credit hours:, yes, smaller is better, but it’s only better for those few students. Credit hours measures the “bang for the buck” of education. Putting fewer students in better/smaller classes increases the (just invented) education-Gini coefficient. To research expenditure: note that expenditure is just an easy way to measure research *income* (ie grants brought in). And on the oh-so-plausible assumption that our research funders are maximizing the bang-for-buck in the research they fund, more money brought in means more research-value generated. As for H-index, yes, that’s an arbitrary metric for value-by-citation, but in your counterexample you seem to be arguing that a paper can be incredibly influential without getting cited—if so, some academics are being sloppy in their citations!

    It’s interesting how credit-hours and expensive research are pulling in opposite directions. Those favoring credit hours are saying that we should give up on “elitist” classes that provide higher quality education to fewer students. But by that perspective, we should also be giving up on funding expensive research on the theory that it’s more equitable to dole out smaller amounts to more people.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: