A few of weeks ago I outed the multi-million-dollar exercise that is the Quality Evaluation component of the performance based research fund (PBRF) as a futile exercise because there was no net gain in research dollars for the NZ academic community.
Having revealed the Emperor’s new clothes, I awaited the call from the Minister in charge to tell me they’d cancelled the round out of futility. When that didn’t come, I pinned my hope on a revolt by the University Vice-Chancellors. Alas, the VCs aren’t revolting. This week, my goal is for there to be mass resignations from the 30 or so committees charged with assessing the evidence portfolios of individual academics and for individual academics to make last minute changes to their portfolios so as to maintain academic integrity.
What is PBRF?
According to Te Ara Encyclopedia of New Zealand,
“The Performance-Based Research Fund (PBRF) is how the government allocates research funding to universities. Every six years each university is ranked according to a number of criteria and given a score, which determines the level of funding each institution receives for the next period. In 2013 the quality scores of all institutions markedly increased over their ratings in 2010, and the rankings had changed, with Victoria University of Wellington marginally ahead of the others.”
Metrics – all they are cut out to be?
I love academic metrics – these ways and means of assessing the relative worth of an individual’s contribution to academia or of the individual impact of a piece of scholarly work are fun. Some are simple, merely the counting of citations to a particular journal article or book chapter, others are more complex such as the various forms of the h-index. It is fun to watch the number of a citations of an article gradually creep up and to think “someone thinks what I wrote worth taking notice of.”
However, these metrics are largely nonsense and should never be used to compare academics. Yet, for PBRF and promotions we are encouraged to talk of citations and other such metrics. Maybe, and only maybe, that’s OK if we are comparing how well we are performing this year against a previous year, but it is not OK if we are comparing one academic against another.
I’ve recently published in both emergency medicine journals and cardiology journals. The emergency medicine field is a small fraction the size of cardiology, and, consequently, there are fewer journals and fewer citations. It would be nonsense to compare citation rates for an emergency medicine academic with that of a cardiology academic.
If the metrics around individual scholars are nonsense, those purporting to assess the relative importance (“rank”) of an academic journal are total $%^!!!!. The most common is the Impact Factor, but there are others like the 5-year H-index for a journal. To promote them, or use them, is to chip away at academic integrity. Much has been written elsewhere about impact factors. They are simply an average of a skewed distribution. I do not allow students to report data in this way. Several Nobel prize winners have spoken against them. Yet, we are encouraged to let the assessing committees know how journals rank.
Even if the citation metrics and impact factors were not dodgy, then there is still a huge problem that faces the assessing committee, and that is they are called on to compare apples with oranges. Not all metrics are created equal. Research Gate, Google Scholar, Scopus and Web of Science all count citations and report h-indices. No two are the same. A cursory glance at some of my own papers sees a more than 20% variation in counts between them.
I’ve even seen papers with citation counts of 37, 42, 0 and 0. Some journals are included, some are not depending on how each company has set up their algorithms. Book chapters are not included by some, but are by others. There are also multiple sites for ranking journals using differing metrics. Expecting assessing committees to work with multiple metrics which all mean something different is like expecting engineers to build a rocket but not to allow them to use a standard metre rule.
To sum up, PBRF Evidence Bases portfolio assessment is a waste of resources, and encourages use of integrity busting metrics that should not be used to rank individual academic impact.