The Challenge of Science Challenges

By Robert Hickson 12/11/2012

Its great to see the government reaching out to the country to get feedback on what the big challenges facing NZ are, and the role that science can play in helping solve these. However, I’m dissatisfied with the possible challenges that they have put up on the website for two reasons. Firstly, many of these seem to be some of the Transformational Research, Science & Technology topics identified by MoRST (one of the grandparents of the Science & Innovation group in MBIE) several years ago. Nothing wrong with that, it was good work. But not a lot of additional thinking about these seems to have gone on since then.

The more unsatisfactory aspect is that these potential challenges cover much of what is already funded under existing schemes – more of the same rather than something novel. Yes, they are examples only, and the Cabinet Paper does refer to the use of “straw men” in the public consultation. But I’d expect more effort to help the public consider what a good science challenge is or what some important specific issues are, rather than just provide examples of a range of research that is currently being undertaken. Who wouldn’t want to support developing cures for cancer, or better pest control? What makes a good challenge and why, and what are the criteria being used to decide? A bit more information please. Maybe the TV advertising will tease this out, I hope so.

On the more positive side, the intent is to get more government agencies aligning policies and other efforts behind the challenges.

I have difficulty is viewing what is currently available as challenges or even “science project” because they lack specificity and measurable outcomes. Minister Joyce in the Cabinet Paper notes that the intention is rather to develop a scheme similar to CSIRO’s National Research Flagships, and that more of government funding will migrate to the selected challenge areas in the longer term. For me a “Challenge” has more the flavour and intent of the DARPA,  Grand Challenges in Global Health and Research UK challenge initiatives, where quite specific problems are being addressed and there are specific milestones and indicators of success along the way.

The Cabinet Paper notes the national science challenges are “aspirational”, so I see the scheme as more PR (not a bad thing) than focussing on some significant and hard specific problems facing NZ. However, time will tell. The process is to take the public responses and attempt to combine them with internal Ministry thinking based on sector consultation and a “Peak panel workshop” to identify the final set of challenges.

I would have preferred that they pick three challenges rather than 10, so that each one has a decent amount of money behind it. Money though isn’t the only “challenge”. Even with recent structural and funding changes, getting some institutions (as distinct from individual researchers) to collaborate nicely rather than compete will take some time (and a new generation of administrators).

It is fortuitous that a recent issue of Technology Review looks at challenges and asks “Why we can’t solve big problems“. The conclusion in the article is that solutions to technological problems require three factors:

  • The public and politicians must care about the problem;
  • There is institutional support for the solution (eg NASA and others supported going to the moon); and
  • The problem is a technological and one that we understand (the latter is an acknowledgement that we don’t yet know enough about, for example, some diseases to consider what an effective technological solution could be).

The DARPA and Global Health challenges tend to operate in this space. The National Science Challenges perhaps won’t meet all these criteria, particularly when it comes to the third bullet point. Reducing obesity or child poverty can have little to do with technological solutions. Improving environmental health may have a technological component, but societal and policy changes will also be necessary.

That isn’t a problem for the National Science Challenges because the initiative is framed as science rather than technological challenges. But the government (and the lucky “winners”) will need to be careful in explaining exactly what they are attempting to achieve to avoid getting a backlash in 4 years or so when the public asks “Has that challenge been solved?”

The European Commission has published a relevant report about Grand Challenges [PDF], noting the preference for the USA to go for technological or industrial challenges while Europe has focussed more on scientific challenges, and some Asian countries are trying to cover the spectrum. The report warns against attempting to integrate or closely align research challenges with innovation ones (ie where the goal is to take an idea or technology through to commercial success). This is because this seldom works where the government doesn’t have a strong influence over industrial decision making. Grand challenges also do better when they have some independence from government, and that the organisation managing the challenges see themselves as “change agents” rather than funding bodies.

So we aren’t going down the Grand Challenges path. But there are some opportunities to stimulate new research and encourage public interest and engagement. I’ll be putting a lot of thought into my proposals for good challenges.

0 Responses to “The Challenge of Science Challenges”

  • Exactly – either they are Science Challenges, in which case they involve additional funding to make novel things happen and have targets and end points, or they are the issues New Zealand needs address with Science, which is what we do now with new labels. Alternatively you could confuse the two and fall neatly between two stools, neither solving any challenges nor investing in the research that needs to be done.

  • For the curious, Nassim Taleb ( of Black Swan repute ) has a free article covering science and technology discovery from his forthcoming book …

    It might help explain the meagre returns for NZ taxpayers from their ongoing and extravagant “investment” in NZ researchers who select their programmes to match resources available to them.

    • Thanks for the link Bruce

      You can’t call New Zealand taxpayer spending on R&D “extravagant” – as a percent of GDP government spending on R&D was 0.59% in 2009, compared with an average of 0.73% across the See OECD Main Science & Technology Indicators OECD

      And unfair to claim researchers select programmes to match the resources. Research programmes are often written in response to extensive guidelines provided by the funding agencies. In some cases quite specific research is requested, in others there is greater scope. Where they request more money than they actually receive, then the programme may need to be modified to the budget.

      It has always been a hard job demonstrating what the taxpayer gets back from investments in research. However, surveys of New Zealander’s attitudes to science make it clear that many consider science important even if there is uncertainty about economic benefits –

      I found Taleb’s paper difficult to follow. He seems to be complicating the issue through mathematical analogies, and simplifying how science and technological developments operate. Science isn’t random, and most does build on previous research. His analogy of funding five options for one year each rather than one for five years will be unlikely to give a better “payoff” in many fields of research due to the long term nature of research.

      He is right, though, in stressing the importance of optionality. A criticism of some of the current funding processes is that the researchers must indicate the outcome at the beginning, and if something else interesting pops up along the way it can be difficult to change the direction of the research.

  • George
    Spot on. There is already endless expert discussion about where the govt should invest its research money, and I doubt the general public will be able to add much to the debate. In fact the risk is that we get seduced into a seemingly attractive area which is already generously funded in other countries.
    And then, of course, if the challenges turn out to be things we’ve been working on for years, there will be great dissappointment in some quarters.
    Still, there will be some value in getting people talking about science, I suppose.

  • One of his points is that “fail fast” is a good strategy, allowing five starts and murdering non-viable research early and then using the money to try something new. The pharmaceutical industry has been trying to move to that model for the last few years, as cost escalate at each phase.

    He also says that practice can eventually produce theory, so we should invest in experimental programmes that aren’t being used validate/enhance theory. He favours simplicity over complexity when evaluating programmes, and purchasing minimal knowledge to achieve material objectives, and not investing in further detailed arcane knowledge. He also suggests funding innovative individuals who have previously produced the goods, but they may be hard to find in NZ currently.

    I disagree, NZ research is extravagant. Taxpayers haven’t been asked whether they want a $500 refund or to fund researchers. So how about some examples that show the current funding model has produced significant returns for taxpayers?. 20+ years should be long enough to have some innovation winners, and please don’t include programmes that came from DSIR, such as supercritical processing.

    Kemo sabe. Experts have been talking for decades, and extracting funding from agencies such as FoRST for over 20 years. Major returns haven’t appeared, some examples – superconductivity, effective pest control for NZ national parks, higher value products from forestry and agriculture, utilisation of agricultural wastes, renewable energy and conservation.

    It may be that the public will have some innovative ideas, but it could be hard for them to compete, given the current providers ( CRIs etc ) are having their own roadshow events about the challenges. Given that funders haven’t previously been able to select winners, it’s unlikely their skill has suddenly improved.

  • Bruce
    “Major returns haven’t appeared, some examples – superconductivity, effective pest control for NZ national parks, higher value products from forestry and agriculture, utilisation of agricultural wastes, renewable energy and conservation.”

    I’m not quite sure what you are saying here. That no progress has been made in these areas? Or it’s too slow?
    There is evidence to the contrary:
    Our exports of processed primary products have grown 122% since 2003. A lot of that is down to R&D.
    Geothermal energy provides 13% of our total at present, and will rise to 20% by 2025.
    And so on. Download the CRI Annual reports for 2012 and you will see lots of other examples.

  • Kemo sabe,

    The request was for financial outcomes from specific NZ taxpayer-funded research that justified the investment.

  • Kiwifruit gold came from a programme initiated in the 1970s, and the Hort 16A strain was identified in 1992 – the time of CRI formation. I regard that timeline as more relevant.

    I pointed to Nassin Taleb’s article, because he proposes funding many programmes and failing fast. That is somewhat different to your original concerns that giving $6 million to 10 programmes may be insufficient funds for each to develop.
    I’d regularly review, kill poor performers quickly, and transfer money to those advancing rapidly on their objectives or to new entrants.

    He also suggests that documentation and explanation of failures is of significant value, something that seldom happens in science – as it mainly benefits potential competitors.

    Also, with regard to your concern about collaboration, one of the larger successful bids ( $15 million ) of this year’s funding round involved a totally new bioresource processing alliance of 4 CRIs and some other entities, lead by IRL, attacking a major topic.

    MoBIE has been stressing that they want to see collaboration, and I assume that successful challenges will feature that as well.

    • I agree that there is a need for more fast failing in some research programmes and, as Taleb discusses, more reporting of negative results. What constitutes a “fast fail” will vary though depending on the nature of the research.

      Pharmaceutical companies are taking a stronger line on failing quickly for drug development. The US National Institute of Mental Health has also introduced a Fast-Fail scheme for clinical trials – The open innovation programs, such as Innocentive and Nine Sigma, can also be seen as taking the fast fail path.

      Defining fail points for more exploratory research is harder. I’d hope that CRIs, now they have more control over some of their own funding, will be continually looking at their research programmes and deciding which ones are worthwhile continuing or modifying. It would be useful to collect this information for later evaluation so we can learn what factors were important in the success and failure of various research programmes, and how this can help subsequent research decisions.

      Adopting a fast fail approach also means that our society has to develop a better understanding about “failures” in research and how these can help improve research outcomes.

      For science challenges the decision about adopting a fast fail approach will depend on whether you are looking for a particular solution, or the objective is to just improve understanding of an issue. As I indicated in the original blog posting, the latter isn’t really a “challenge”, and it would be difficult to define a fail point. But it seems the government is leaning that way at this stage.

      If you are addressing a technological challenge and had, say, $20 million allocated to that over 4 years, then that could fund a variety of competing approaches and so improve finding a good solution quickly (as some of the DARPA challenges do). With just a couple of million then it is more likely only one or two research projects addressing that challenge could be supported and you may not find the best solution.

      I doubt that we have enough research capability in NZ to field multiple teams for any particular challenge. To get the best outcomes the challenges would need to be open to the world (given appropriate protections for IP etc).