By Peter Griffin 02/10/2015

New Zealand’s leading ‘blue skies’ research fund boosts Kiwi science, but could be tweaked for greater efficiency, says a new study.

An evaluation conducted by researchers at Motu Economic and Public Policy Research institute has found that Marsden funding increases the scientific output of the funded researchers.

Compared to similar groups that do not receive funding, a team that is given Marsden funding shows a 6-12 percent increase in their academic publications and a 13-30 percent increase in the papers that cite their work.

Before a decision is made to fund a project, it must go through two rounds of evaluation. The new study asserts that “the significant resources devoted to the second round evaluation could be reduced without degrading the quality” of funded research.

Over at the Science Media Centre, we collected the following expert commentary from New Zealand researchers.

Prof Juliet Gerrard, Chair of the Marsden Fund Councilsaid in a statement: 

“We welcome the study and the central result that receiving Marsden funding boosts publications, and the impact of those publications. The Marsden Fund has a unique dataset of successful and unsuccessful proposals from many years of operation and was happy for this to be used to see whether funding processes can be improved.

“We are not surprised to find that all the proposals in the second round are of very high quality, and have for many years said that we could double the number funded without dropping quality. This study backs our claim and suggests that the first round process is selecting high quality proposals for full review.

“We continually seek to improve our processes by benchmarking against other funds.  We note that the HRC has trialled a lottery system in their HRC Explorer grants. It would be good to compare this system to the traditional peer review system which they use for their other grants. At the moment, the confidence given by external peer review, particularly in differentiating the ‘truly excellent’ from the ‘very good’ (who might have similar paper output metrics) is critical to the current success of the Marsden Fund.

Prof Harlene Hayne, Vice-Chancellor, University of Otago, comments:

“Since 1994, the Marsden Fund has supported high quality research and has become one of this country’s most prestigious sources of funding for university researchers.  The MOTU report confirms that the Marsden fund has had a significant impact on the amount of internationally-cited research that is produced in this country—research that enhances the international rankings of our universities helping us to succeed in an increasingly competitive international market. Sadly, the report also shows that high quality research proposals are left unfunded simply because the fund just isn’t large enough and clearly demonstrates the return the Government would achieve from increasing its investment in the Marsden Fund.”

Dr Nicola Gaston, President of the New Zealand Association of Scientists and Principal Investigator, MacDiarmid Institute for Advanced Materials and Nanotechnology, comments:

“The Motu study of the effectiveness of the Marsden Fund is a great step towards transparency and the use of evidence in setting policy around science funding in New Zealand.

“What is a little surprising, at first glance, is the conclusion that the rankings made by the panels at the second (and final) stage of decisions does not predict the ultimate impact of the work. However, this is not as surprising at it may seem, given that the panels themselves are broad (for example, spanning Physics, Chemistry, and Biochemistry) and they are heavily oversubscribed, meaning that sometimes panelists may be comparing apples and oranges in terms of the research topic.

“It certainly doesn’t change the conclusion of the researchers – with which I agree wholeheartedly – that this study demonstrates that there would be no diminishing returns if we were to increase – perhaps even to double or triple – the size of the Marsden Fund.

“The second point made by the research team, that the resources invested in the second stage review of proposals may be largely viewed as wasted, I think is rather more nuanced.  From the perspective of the funders this is perhaps true, but the reality is that the expected process of evaluation determines the behaviour of the researchers who apply to the fund. For example, one could imagine that if the second round were perceived to be less rigorous than it currently is, the fund would attract a larger number of less credible applications. So I would recommend caution in any response to this particular conclusion.

“Finally, the simulated trajectories for publication and citation counts, under scenarios with and without Marsden funding, demonstrate one very crucial fact: the baseline slope is negative. Funding for research of the kind available through Marsden is not just a nice-to-have; funding for research is an absolute necessity if we are to sustain a vital and productive research community in New Zealand.”

Declared conflicts of interest: I have no current connection (funding or panelist) with the Marsden Fund though I have been previously funded.

Prof Shaun Hendy, Director of Te Pūnaha Matatini, a New Zealand Centre of Research Excellence, and Professor of Physics at the University of Auckland, comments:

“This is a watershed study. It is the first rigorous evaluation of a New Zealand research funding process ever undertaken and it has thrown up some fascinating insights. Most importantly it shows that receiving a Marsden grant does lead to higher productivity and impact, at least in terms of papers published and the citations those papers receive. This does not surprise me, but it is very exciting to see the benefits of Marsden funding quantified for the first time.

“Perhaps the most interesting finding is that the expert panels used by the MarsdenFund to evaluate proposals are not good at picking winners. Investigators with proposals that are ranked the highest by panelists do not go on to outperform investigators with proposals that are ranked lowest, once you correct for the fact that investigators that are ranked highly are funded. This does not mean that the panels are redundant however, as I would expect that there will still be benefits that accrue from encouraging researchers to plan and develop research plans that can stand up to scrutiny from these panels. However it does suggest that we should be cautious about using success in Marsden as a proxy for research quality, particularly when it comes to career advancement.

“Finally this study demonstrates the benefits of the sustained collection and retention of science and innovation data, and the Royal Society of New Zealand should be commended for its commitment to doing so. Unfortunately, the Ministry of Business, Innovation and Employment and its predecessors have done a very poor job of curating their data, meaning that much of the rest of our funding system will remain opaque for some time to come. The Ministry needs to put in place systems and practises that will allow these types of studies to be undertaken right across our science and innovation system.

Declared conflicts of interest: Adam Jaffe is an investigator in Te Pūnaha Matatini, the Centre of Research Excellence that I lead. I was also a Principal Investigator on two Marsden-funded projects during the period that this study covers (in 2006 and 2008), and I served on the Physics, Chemistry and Biochemistry Panel from 2010-2012.

0 Responses to “Marsden Fund: Benefits quantified for first time”

  • Ummm…..I know I am being picky or missed something here…but where is the 30% quoted from??? I see 26%….
    The exec summary mentions 6-15% and 22-26% for the two measures.

  • Craig Stevens: Crazy science – but is it crazy enough?

    This is an embarrassingly contrived and condescending rhetorical attempt at defending Marsden Fund from criticism that it represents a waste of public money. It doesn’t even make sense:

    “Sometimes the ideas help solve a problem we can all identify with, but more often they seem plain crazy. And that’s the idea”
    And this makes sense, how?

    “…arguing that we needed to walk a line between something that the world needed, but remained niche enough that it wouldn’t just disappear overseas. This perfectly weird stuff must come from somewhere”
    And this makes sense, how? So, let me see, we want to discover or invent something that the world does need, but which is “weird” enough that the world doesn’t steal it from us, this “perfectly weird stuff” which “must come from somewhere”. Yeah, right …

    “In which case a year of Marsden funding would get us about 2km of Transmission Gully, at the cost of getting dumber as a nation”
    So, without Marsden Fund we would get “dumber as a nation”, would we indeed?? Well, at least we might be able to get out of Kaikoura by road! Even with Marsden Fund, it doesn’t appear that the president of the NZ Association of Scientists can write in grammatical or coherent English sentences!