Guest blog – evaluating the New Zealand Science system

By Guest Author 25/09/2009


A personal viewpoint piece* by Garth Carnaby, President of the Royal Society of New Zealand. This paper was presented at the ARMS Conference in Christchurch last week.

A          Questions to be Considered

In presenting this paper, I am not putting myself forward as an expert on science funding systems, or how to evaluate them.  This is a personal perspective and opinion piece based on my own experience rather than a scholarly or researched view.  It is also very preliminary, as I am only just coming to grips with this rather daunting topic.  However what it does try to do is provide some context within which a number of questions can be asked.  These questions include, ’what is the value proposition for public investment in science?’, ’what proportion of public funding should be allocated to which activity?’, and ’how efficient is the process for the allocation of the funding?’.

I have found that in discussing these issues, scientists tend to talk at cross purposes, primarily because they try to hold the discussion whilst assuming that they share a common view of what research actually is.  So the first step I have taken in seeking to approach these questions has been to explore the notion that there is ’a’ science system at all.  To examine this I decided to begin with what constitutes ’excellence’ in RS&T.

B          The Pursuit of Excellence

Very few would argue against the goal of the pursuit of excellence in Research, Science & Technology (RS&T).  RS&T covers a complex and intellectually demanding field in which only the most gifted and highly trained are likely to succeed.  Second rate activity is not likely to have much impact, if any at all.  However, this begs the question as to what constitutes ‘excellence’ and whether there is just one standard of excellence or whether indeed there are multiple dimensions to excellence.

It is the primary hypothesis in this paper that there are indeed multiple dimensions to excellence in the fields of RS&T and that these various forms of excellence are all necessary components of a balanced science system.  In RS&T  there are various activities all of which may be categorized as research and/or development.  These activities are not necessarily connected or even it might be controversially suggested, actually interested in each other. They do however share two areas of commonality.  The first is a reliance on existing internationally published and ’free to all’ scientific knowledge — either as a basis for new discovery, or for application to a new problem.  The second is an appetite for the consumption of taxpayer generated public funds.

Because a significant component of all RS&T activity is publicly funded in most countries, there is generally widespread debate over the optimal allocation of such funding and the system used to ration this resource amongst the various claimants for public support.  Such debates in New Zealand are often coloured by an unattractive element of self-interest and sometimes confused by the generalization from personal experience of what works in one area, or in other countries, to a system for New Zealand as a whole.  Such generalisations are sometimes based on the application of a uni-dimensional set of excellence criteria which work well in fundamental science, but which do not really apply to say, development as opposed to research, or to research targeted at specific public good outcomes which have a parochial local context.  The argument which will be elaborated upon in the next section is that the excellence criterion relevant to a specific activity within the broad envelope of RS&T need to reflect the intent of the investment.  As the intent varies, so too should the excellence criteria.  Furthermore the intent of the research further differentiates the value proposition which proponents of that particular activity might present to the holders of the public purse.  In the analysis which follows, I have not considered special aspects relating to research into the social sciences and humanities, but some of the issues will apply.  My focus in this paper is the traditional science and engineering fields.

C          Co-Existing Research & Development Activities: Their Intent, Their Excellence Criteria, and Their Value Propositions

C(1)        Basic Research

The most widely understood and agreed upon criteria for excellence are those that apply to research aimed at the discovery of new scientific knowledge.  The criteria used to assess excellence in this activity are internationally consistent.  This is hardly surprising since science is international and those seeking to extend scientific knowledge must do so by publishing in journals likely to be seen by others in their field, wherever they might be.  The excellence criteria (not necessarily in order of weighting) include:

* Novelty in relation to the published scientific literature.
* Publication of results in the best international journals.
* Importance of the research gauged by citation rates.
* International science peer review and acclaim.

These criteria are used, for example in New Zealand, to assess research applications to the Marsden fund or to assess academic performance in the PBRF system of the Universities.

The intent of the investment in research aimed at the discovery of new scientific knowledge is multifaceted.  Engagement with and involvement in international science is a cultural imperative for a small country like New Zealand.  New Zealand has direct linkages to establishments at the centre of scientific thought such as the Royal Society of London, the Academy of Science in the USA and the Australian Academies.  We need to remain connected to global science so we can ’spill in’ benefits to New Zealand.  The same applies to other smaller countries such as Australia or even Britain.  The colonization of New Zealand in the 19th century in fact was made possible by and coincided with the exploitation of earlier fundamental scientific knowledge during and following the industrial revolution in Britain.

Modern post-colonial science involves a merging of many knowledge systems and remains central to the creation of new industries worldwide.  It is inconceivable that New Zealand society would fail to invest substantial resources in being involved as fully as possible.  Our investment in this type of research is however not solely justifiable in terms of serendipitous discoveries which may have economic potential.  It is also justifiable in terms of the intellectual stretch it provides to our cleverest individuals, the way in which it informs the tertiary education of new science graduates, the way in which it provides a knowledge resource to be sourced by applied researchers, and how it aids our understanding of our surroundings.  It is widely accepted that a major outcome of academic research is the flow to business, not so much of particular knowledge in the heads of graduates, but of their ability once in business to continue to learn and apply new knowledge to any conceivable situation. Quite simply this type of research investment has enormous leverage on the entire effectiveness of the enterprise, collective effort and wellbeing of all New Zealanders.  It is an ethical obligation for all those who wish to benefit from science to contribute to it.  It has a compelling value proposition for the application of public funds.

C(2)        Applied Research Aimed at the Industrial and Technological Development of Established Industries

Let us put aside for the moment the issue of whether public funding is needed to leverage and hold together the infrastructure needed to support established industries.  The first question I would like to address here is what constitutes excellence in the scientific and technological research conducted in support of such established industries.  Researchers here often have years of experience in a particular sector and may be trying to directly transform agricultural practice, find new uses for milk solids, or to improve power generation.  This type of research is outcome focussed and is concerned with ‘know-how’ as much as ‘know-why’.  Very often, there is no published scientific or technological literature within which the researcher is looking for gaps, inconsistencies or synergies.  Where there are learned journals of published research, they are likely to be specialised and at best to be lowly ranked compared with the mainstream scientific journals.  More likely a project will be selected based on a need identified in the market or in current industrial practice.  A hallmark of excellence is the likely involvement of a major industrial partner or whole industrial sector at the concept stage.  Even more significantly, in relation to excellence assessment, actual publication of research progress, at least in the early stages, is likely to be counter productive to the aims of the research and maybe against the national interest.  The first evidence of its having taken place is likely to be patent activity, or perhaps a surprised market.  Successful unexpected launch after achievement of secret working for several years may be a more suitable excellence yard stick.  Publication in industries where New Zealand holds a world leading technology gap (e.g. dairying) might be seen as more to do with a selfish desire to advance ones own career at the expense of potentially beneficial outcomes to New Zealand.  Success will depend on sales, consequential economic growth and wealth creation.  If taxpayer funding is used, a ‘public’ obligation of targeting the wealth creation towards the domestic economy is likely to apply.  The research is likely to involve much greater teamwork and require the intellectual integration of a wide range of complex competencies, for example of design, finance, engineering, and marketing as well as scientific research or the application of science to the discovery process.  The key brains are likely to be integrating and leading such teams at least one step further removed from the lab bench than in basic research.  The commercialization process for this type of activity is more often than not via licensing or personnel transfer to existing players, often using partnering relationships lasting for decades.

Some of those involved in basic scientific research might argue that what is described here is not science.  And it may indeed be better described as the application of old science to a new problem.  But it certainly is both intellectual in nature and creative of new technological knowledge.  Science or no, those involved would none the less regard it unequivocally as research.  About the only obvious common criterion with basic research for the assessment of excellence would be peer acclaim.  A career of sustained technological innovation based on science is easily recognised by those who are familiar with the contribution.  A researcher working in an industrial context and who has been the key source of intellectual inspiration for the creation of a new industry or some other widely impacting outcome will have wide regard in their own sector.

If one had to choose four principal excellence criteria for this type of research then the list might comprise the following:

* Clarity of the market need.
* Novelty of the proposed solution.
* Consequential economic activity.
* Industrial sector peer review and acclaim.

Some would argue that all the State needs to do is put all of its public funding into the discovery of new scientific knowledge and then leave the rest to the self‑interest of the market and to the enterprise of individuals.  However this ‘laissez faire’ view is a rather naive one, more often than not expressed by those who have not actually been involved in applied research.  Even well established industries need to reinvent themselves every few years if they are to retain their global competitiveness.  And although the domestic economy in New Zealand is relatively diversified and sophisticated, our export sector remains dominated by a small number of agricultural commodities where the country enjoys a comparative advantage in their production, albeit one which is informed by  some of the most advanced scientific knowledge and specialist vocational training of its type in the world.

Quite apart from the pressing need for New Zealand to create new exporting activities, which will be discussed in section B(4), we need first to be sure that we do not lose too many of the successful sectors which we have already got.  The use of public funds for research to support existing industry is not a proposition which is popular with those wishing to appear progressive, but it is fairly basic common sense nonetheless.  It is not self evident that such industries, even those of national importance, will properly maintain themselves if left to their own devices.  There are plenty of concrete examples where they have not.  I could, if encouraged, elaborate on the example of WRONZ and the once iconic NZ Wool Industry, or the parallel situation in Australia, but I will exercise rare restraint on this occasion.  Industry infrastructure, including its research capability, can easily collapse due to any of a number of factors ranging from under-investment, grower politics, incompetent governance, and government indifference and inaction.

The ‘Commodity Levies Act’ in New Zealand does provide a potential State Imposed means of generating ‘industry good’ funding from the many small New Zealand based producers of agricultural commodities.  I will also forbear from writing an essay on the politics of this subject, but suffice it to say that the process has limitations ranging from the difficulties in ensuring value capture, to where the levy is applied, who actually pays it (producer or consumer depending on the effectiveness of the spend), and the difficulties in measuring or demonstrating value created from their use.

A particular anomaly which seems to have developed here in New Zealand is for the CRI’s to be concentrated in the primary and secondary industries (except ESR).  The New Zealand economy comprises 70% provision of services, and these are increasingly exported.  But this largest sector of our economy has no dedicated CRI.  Whilst conventional wisdom may have it that this is not an area suited to scientific advance, it is unquestionably an area in which any innovation will have a disproportionate impact.  An efficiency gain in retailing would be huge.  Is there really no role for dedicated research to support this sector?

The value proposition to government for the use of public funds to leverage individual company or industry levies(on say a $ for $ basis) is based on the likely large economic impact of applied research of this type, its specificivity to the unusual narrowly based New Zealand economy, and the potential for dysfunction and industry underperformance in its absence.  The benefit impact generally extends well beyond just the funding group.  The potential leverage here is huge, with short times to impact, and potentially bankable long term improvements.  A small percentage gain in a $4B industry has the potential to add economic gains in excess of $100M within a few years.  The potential Return on Investment from such a short term outcome, compared to a new larger but higher risk development, which is still ten years away, when both are discounted for both risk and the time cost of money, is compelling in terms of its value proposition for the use of some public funding leverage.

C(3)        Excellence Criteria for Research Committed to the Service of the Public

Not all applied research has economic objectives.

Large parts of New Zealand’s RS&T world remain steadfastly committed to public service provision.  Whether this is the achievement of public health outcomes, an improved environment, or the protection of native fauna and flora, it is the amount of public good which is created which is the principal consideration for those who fund this research and for those who carry it out.  Therefore the scale of the public good which is created by the research must necessarily be a major factor in assessing the excellence of the research.  The way in which the current basic research oriented excellence criteria discriminate against scientists who aspire to these values are subtle, but none-the-less often fatal to their chance of advancement.  It may be that such a researcher has been personally responsible for the research which has preserved a local wildlife species or that they have applied science to save a multitude of citizens from a nasty pandemic.  Because of this, their research may be only of local relevance and hence published in New Zealand journals.  Citation rates and low international impact may just reflect the particular relevance only to our own society.  If those who invest public funds wish to preferentially target such more parochial outcomes, then relevant excellence criteria for assessing the research and those involved are needed to align incentives with the outcomes derived from the investment.  In this case the excellence criteria might include:

* Clarity of the community, social or environmental need and the potential public benefit of a solution.
* Uniqueness of the issue under investigation.
* Public access to the published findings.
* Community peer review and acclaim.

The value proposition to the Crown from those engaged in research of this type is similarly primarily focussed on the social and environmental bottom lines as much as the economic one.

That these value propositions can be compelling is beyond question.  If, for example, ‘bird flu’ or ‘swine flu’ or indeed the next pandemic were to be killing several hundred New Zealanders per week, there would be a very intense concentration on the effectiveness of our science based infrastructure to detect, monitor and treat such a calamity.  At the same time many New Zealanders are passionate about our environment, its fauna, flora and the taonga of the tangata whenua.  The value proposition for public resources to fund research in these areas is based on the unlikely emergence of private funding, and the public benefits to us all

C(4)        Commercialisation of Serendipitous Science Discoveries

This activity is highlighted as yet a fourth area of activity within the RS&T envelope.  It differs from the industry relevant research of section B(2) in that it is disruptive rather than supportive of current economic activity and investments.  Commercialisation of disruptive technologies is usually best pursued by creation of a new start-up company rather than via licensing to an established enterprise.  The opportunity can arise from any of the forgoing research activities, but most likely it will arise from basic research.  The decision to commercialise is complex, and begins with an assessment of market need, just as in industrial research.  But the need is usually harder to quantify and may be both global in scope and very large indeed.  This is actually a form of ’Development’, i.e. ’D’ rather than ’R’ , ’ Research’ and similar excellence criteria apply as in B(2).  But the mix of skills is not only different and possibly even more complex, but importantly often includes a personal appetite to take financial risks and to commit ones own personal assets to the venture.  A list of excellence criteria for this activity might include the following:

* Size of the market need.
* Protectibility of the proposed solution.
* Return achieved on capital invested given the risks.
* Peer review and acclaim.

There seems little debate that there is a relationship between basic research activity in large economies and the commercialisation of technology in those countries.  The question of whether public investment in basic scientific research in a small country like New Zealand will automatically lead to the commercialisation of consequential serendipitous discoveries within New Zealand however deserves further scrutiny.

The total New Zealand investment in scientific research represents much less than 1% of the total public investment in this activity globally.  If it is all immediately published freely in the international science journals without prior patenting, then a New Zealand based developer has no particular advantage in the research actually having been conducted in New Zealand and indeed most general science knowledge needed to solve a particular problem will have been discovered somewhere else some time ago.

Even if a new opportunity, discovered in New Zealand, is patented, and kept secret or protected, then the route to commercialisation may still be difficult.  New Zealand is not necessarily the best place to locate a new industry once it reaches any sort of scale, so the challenge and responsibility for ensuring preferential value capture for the public benefit of New Zealand may be significant.  But many examples do exist of New Zealand technology based start up companies achieving scale whilst both serving a global market need and remaining resident in New Zealand.

The value proposition here for the use of public funds is both strategic and economic.  The New Zealand economy is growth constrained by our past failure to diversify our range of globally competitive industry capable of exporting.  There are environmental limitations on our ability to extract and export more commodities and distance constraints on the export development of our service industries.  This is a higher risk-return investment area where not all the benefit will be captured by private investors.  But it has the potential both to be transformational and to contribute to economic risk mitigation for New Zealand.  As this is clearly a necessary component geared to longer term growth in a balanced portfolio, and the certainty of private capture of benefits is less, it too has a compelling value proposition to make to government as an area for public investment.  This is all the more pertinent at the early stages of transition from state funded activity to angel funded capital burn.  New ideas are emerging to bridge this valley of death.  Examples of these new initiatives with which I am involved include the ’Canterbury Regional Innovation System’ and a possible Canterbury Regional Development Bank.

C(5)        Summary of Activities, Intent, Excellence Criteria and their Value Propositions

The four areas traversed above are summarised in Table 1.  To this table a further column has been added.  In this some of the common criticisms levelled against the use of public funds for a particular activity are described.

D          System Evaluation

Only now, in concluding this presentation, am I ready to begin addressing the three questions posed at the outset.  Firstly, the value proposition for the use of public funds is represented by the sum of the four rather different value propositions described above.  These value propositions would need to be professionally modelled by an economist, before we could get much further here.

Adjudication between the four claimant areas to the public purse is not a simple task either. A rational option would be to distribute the total funding available in such a way that the marginal productivity gain per marginal dollar invested in each was equal.  No attempt is made here to recommend any particular course of action.  The decision is anyway one of political priorities at best informed by the economics.  The purpose of this paper though is to inform and demystify this choice and free it from the clamour of the claimants.  However it is nonetheless asserted that all four areas are deserving of a proportion of the funding as ignoring any one of them would be to invite very sub-optimal outcomes for New Zealand society.  It is also a point raised here, that the four areas discussed are not actually necessarily mutually interdependent.

‘The’ New Zealand science system certainly does attempt currently to support all four areas via quite highly differentiated funding vehicles — all of which have been well described elsewhere.  Their value proposition and excellence criteria have not been so well defined however.  Despite this, successive governments have made the political decisions needed to ration public resources between the areas.

When viewed from top down, as some sort of diversified portfolio of investing, the allocation system for New Zealand science system appears logical, reasonable and balanced.  Funding is allocated to broad strategies covering all four activities described above from basic (Marsden) to very applied (Technology NZ).  We could argue about the size of the cake and its apportionment, but all bases are to some degree at least covered.  Within each investment category competition for funding is vigorous with decisions made by discipline and expert panels on a project by project basis.

None the less, there is plenty of private and public discussion over perceived blockages, inefficiencies and potential improvements.  A particular issue when viewed from bottom up is that scientific specialization is needed to become world class.  But there is only one unreliable purchaser for your skills.  Scientists face periodic redundancy if their skills are too refined.  Institutional leadership is also disempowered by the system.  Strategic accountability for the Institutes ’not for profit’ purpose, and staff development is denied to the executives and Boards which lead them.  However, before those issues can be addressed, it will be necessary in the current environment of scarce public dollars, to first refine the value propositions of the various activities involved.  It would also be helpful to develop some new professionally developed methodology around quantification of the marginal economic productivity gain per marginal public dollar spent on each of the above four value propositions within the New Zealand context.  However that is a possible project for the future.  In the meantime it is clear to the author that if some consensus could be achieved around the issues raised in this essay, then we might at least reduce some of the talking at cross purposes which characterizes some of the discussions at present.

[*] The views expressed in this document are those of the author alone and do not necessarily reflect the views of the RSNZ, NZSG, ESR, CDC or the Australian Synchrotron.