SciBlogs

Timing is everything Shaun Hendy Feb 17

No Comments

Today, I will be reflecting on the importance of good science communication at the University of Waikato’s International Symposium on “Transforming Engagement on Controversial Science and Technology”. There is a lot to say, and a lot that has been said, about science communication. In this post, however, I want to reflect on an aspect of science communication that is often overlooked. 

Sir Peter Gluckman, in his role as the Prime Minister’s Chief Science Advisor, wrote last year about scientists, the media and society. In his essay, he warns scientists of the dangers of becoming advocates for a particular cause, instead arguing that scientists need to act as knowledge brokers for society. Sir Peter’s article is well worth reading, but I think it neglects an important aspect of science communication – namely, that of first response.

Scientists as first responders

Both the 2011 and 2013 winners of the Prime Minister’s Science Media Communication Prize have distinguished themselves by their willingness to step forward during a crisis.

On 4 September 2010, Dr Mark Quigley from the University of Canterbury was woken at 4.35am by a 7.1 magnitude earthquake … and in its aftermath became the spokesperson for the New Zealand science community. Mark was not chosen for this role by the Royal Society of New Zealand or the Ministry of Civil Defence. Rather, in the midst this crisis, he stepped up – reacting quickly, calmly and knowledgeably to the unfolding events. Over the coming months, Mark’s face became familiar to many of us, as he explained the science behind what Canterbury was experiencing and the subsequent risks it faced.

Mark was in the right place at the right time to act, and he was prepared. He had been blogging about his research at DrQuigs.com for a number of years prior to the 2010-11 earthquake sequence. When the first earthquake struck, his blog provided a fast, reliable communication platform for getting his science out. After the quakes, the readership of his blog sky-rocketed as the public turned to it for information on unfamiliar phenomena like liquefaction and the risks of aftershocks.

When the 22 February 2011 earthquake struck, it was noticeable how much better the geoscience community was prepared. Mark’s efforts after the September quake had set an important example for the science community of the need for an effective first response. After a major disaster, we have learned that the public and the media have an immediate need for scientific information and analysis to help them understand what is happening and to allow them to make good decisions.

This is something that can only come from well-prepared, articulate scientists, who can think on their feet and who are comfortable using modern social media. Such a response is not something that a corporate communications team or a national academy can provide.

Fonterra gets the bot

If Mark showed us the value of good science communication in a crisis, Fonterra’s recent contamination scare illustrated the costs when science is communicated poorly. In August 2013, New Zealand’s Ministry for Primary Industries ordered a recall of products containing whey protein from several batches produced by Fonterra in 2012, due to the possibility of contamination by Clostridium botulinum. This quickly became international news, as some of the products affected by the recall included infant formula sold around the world.

This was a major crisis for New Zealand that saw our dollar plummet by US$0.03 in just a few days. As the story unfolded, as with the Canterbury earthquakes, the New Zealand public once again expected timely information and analysis from the science community.

Yet little was forthcoming (in part, perhaps, because one of our major research organisations was involved commercially). In the absence of expert comment, the vacuum was filled by speculation (e.g. Vet links botulism to farms not pipes) and misunderstandings (e.g. Fonterra Fallout: Romano quits). As president of the NZ Association of Scientists at the time, I had to field a query from the media as to why the scientific community had gone quiet.

One of the few scientists who did step up was Dr Siouxsie Wiles, a microbiologist at the University of Auckland. [An honourable mention should also go to Prof John Brooks of AUT (see one of his blog posts here).] Like Mark Quigley, Siouxsie is an active blogger, but unlike Mark, her expertise did not directly align with the science behind unfolding crisis. Dismayed by the lack of expert comment, Siouxsie blogged to debunk misinformation and explain the science behind the tests that had been used to detect the contamination.

Sacred cows

The need for the science community to respond to the public need for facts in a crisis undermines one of the ‘sacred cows’ of science communication: that scientists should only speak to the media on areas of their expertise. The problem with this is that real-world crises inevitably stretch the limit of any one scientist’s expertise. Climate change, for instance, is such a complex issue that no one scientist has the in-depth knowledge to cover every angle.

In the past, we had the luxury of specialist science reporters who were able to talk to a range of scientists to deal with this complexity. Today, few journalists have the time, expertise or network of scientific contacts to do this well. While the Science Media Centre plays an important role in connecting media with scientists, the onus now falls much more on these scientists to provide context for their science. This will often require stepping outside the bounds of their expertise.

And when stories develop rapidly, it is even more important to be prepared to push the boundaries of expertise. This is not an easy thing to do, especially in a short time frame, so it is not surprising that scientists are often reluctant to do it. Yet the public need scientific information in such crises, and any scientist who steps up will almost certainly be better than none.

Being prepared

Last October, the report of an external inquiry into the contamination scare (commissioned by Fonterra’s Board) was critical of Fonterra’s external communications and recommended that the organisation adopt greater openness and transparency in its crisis communication. One of the inquiry’s key recommendations was that Fonterra:

“Establish a network of external experts ready to advise in a crisis on key food safety risks (e.g. chemical, microbiological, biological), complementing internal expertise.”

This is good advice not just for Fonterra, but for New Zealand’s entire science community. We must ensure that there are more than just a handful of scientists who are prepared to inform the public in a crisis. These scientists must be comfortable with the new forms of media, including social media like blogging and Twitter. They must understand the pressures that the traditional media face in dealing with complex scientific issues on short deadlines.

And above all, they need to be given proper credit for their work. Science communication takes considerable effort, but despite its obvious importance to society, it often receives little academic recognition. Rewarding individual scientists like Mark and Siouxsie for their efforts after the fact is all very well, but if we want to be ready for the next crisis, we need to ensure that we prepare as a community.

Marsden 2013: Big increase in funding lifts success rate Shaun Hendy Jan 24

5 Comments

This post is late, very late! I have a long list of excuses, many of which involve moving to Auckland and writing a Centre of Research Excellence Proposal. But with the 2014 Marsden round almost upon us, it is well past time to look at the numbers from 2013.

2013 saw a big increase in the funds handed out. In fact the $68m awarded was the largest ever*, only surpassed by the 2009 round ($65m) if you adjust for inflation. In real terms, the Marsden fund has handed out about 18% more each year over the period 2008-2013 than it did over the preceding decade. The average funding awarded to each successful proposal (fast-start and standard) continues to hover just below $600k.

image

If the total investment was high in 2013, while the funding per proposal remained static, then the number of projects that were funded must have risen. This was indeed the case, yet at the same time the number of proposals received by the Royal Society continued to climb. There were a record 1157 first round proposals submitted in 2013, compared to an average of 800 proposals per year over the period 1998-2007. This means that although a record-equalling 109 proposals were funded, the overall success rate of 9.4% remained below its long run average of 10%.

image

The growth in the proportion of funds awarded to fast-start grants for early career researchers (available to researchers within seven years of completing their PhDs) has continued, but the proportion of funds awarded to fast-start grants is still less than the proportion of applications for fast-start grants: in 2013, 22% of the funds Marsden awarded went to fast-start grants while 28% of applicants wrote fast-start proposals**. Would it be fair perhaps to see the share of funding allocated to fast-starts grow to match the proportion of fast-start applicants?

image

Fast-start proposals have had a success rate of just below 13% since they were created, slightly higher than that of standard proposals at 9%. Interestingly, the success rates of fast-start and standard applicants are only weakly correlated. As I noted last year, the fast-start scheme now plays an important role in early career development for scientists now that the FRST post-doctoral fellowship scheme and the International Mobility Fund are gone. The Rutherford Discovery Fellowships also contribute to early career development but are relatively few in number.

img

There was a comment on my 2012 Marsden post that the >1000 proposals rejected annually represented a huge opportunity cost. However the worth of a rejected proposal is not zero. I always tell myself that it is a chance to plan my research several years in advance, and – if you make it through to the second round – it is a chance to get feedback from international experts in the field. Nonetheless the significant growth in rejected proposals that has occurred over the last few years suggests that the opportunity cost of the Marsden Fund may be increasing.

*NB: The figures released by the Marsden Fund in 2013 did not include GST.
** My thanks go to Jason Gush for filling in some holes in my data on fast-starts

The Physics of Santa Shaun Hendy Dec 24

1 Comment

At this time of year, many parents worry about the risks posed to their children from exposure to Santa Claus. We know very little about the science of Santa because the government refuses to fund research into Christmas,as it cannot be linked to direct economic benefit.

Yet, as Colin Craig has been at pains to remind us this year, unless we know the facts, how can we be sure that Santa’s reindeer are chemical free or whether he even exists? Luckily, Radio New Zealand commissioned a small study this year to answer some of New Zealanders most pressing concerns about Santa*.

Many New Zealanders want to know how fast Santa has to travel to deliver all his gifts.

We estimate that Santa has a bit over 30 hours to do this, assuming that he starts around midnight on New Zealand’s side of the international date line, and finishes before people wake up on the other side.

We are less sure about how many children he has to visit: there are about 2 billion children in the world, and when you ask them whether they have been naughty or nice, they inevitably claim they have been nice. As scientists – and some of us were once children ourselves – we just don’t buy this. We think that it is more likely that only about half the children in the world, roughly one billion, have managed to be nice all year.

To deliver presents to these children, Santa has to visit about 5000 homes per second, assuming that there are 2.2 nice kids per household.

When Santa visits New Zealand, however, he has to deal with the fact that Kiwi kids are generally regarded as pretty nice, and this means he has to visit almost all of them**. If he is to reach the children on his list, Santa only three minutes to deliver his presents to the 800,000 Kiwi kids who were not naughty this year.

To fly from Cape Reinga to Rakiura, Santa and his reindeer must travel about 1600km in that three minutes. This works out to be a speed of around 32,000 km/h – 320 times the open road speed limit or about the same speed the space shuttle travels when it orbits the Earth. This is pretty fast.

But how does he power his reindeer? We think that Santa must be sharing the Xmas mince pies and glasses of sherry that are left out for him with his reindeer to keep their energy levels up.

While he travels over New Zealand, his reindeer have to pull him, his sleigh and about 400 metric tonnes of gifts. To achieve their need for speed, the reindeer must supply kinetic energy of about 13 gigajoules. If each child in New Zealand leaves Santa one Xmas mince pie (say, 400 kilojoules) to share with his reindeer, this will only supply around 5% of their energy needs.

Santa’s energy shortfall works out to be roughly the amount of chemical potential energy is stored in two barrels of oil. Hmmm.

More research is clearly needed. We would recommend however that Santa give consideration to delivering his presents in the day time so that he can take advantage of recent advances in solar cell technologies. This would also make life easier for scientists who currently have to write lengthy applications for permission to stay up past their bed time in order to study this mysterious phenomenon.

Declaration of interests: I have been informed that I was bad this year for wasting taxpayers’ money, so Santa is unlikely to be leaving a present for me under the tree. You can rest assured that this has in no way influenced the conclusions drawn in this blog post.

* Radio NZ apparently has no editorial policy on whether the gentleman in question should be referred to as Santa, Santa Claus or Father Christmas. We’ll stick to Santa.

** Anecdotally, my younger sister reports that her two older brothers, despite being Kiwis, were not always nice. This just goes to show how unwise it is to rely on hearsay evidence.

Pounamu returns Thursday Aug 29 Shaun Hendy Aug 25

1 Comment

pounamu-logoThis coming Thursday (Aug 29) from midday we will be running Pounamu again for 24 hours. This is a free, online game set in a future world where all of us can use science as easily as they can use a computer now. We ran the game for the first time last year, in conjunction with the Transit of Venus forum and boy was it addictive. Sciblogger Michael Edmonds wrote a post about his experiences last year. Like Michael, I found it to be one of the most stimulating and exciting forms of science communication I had ever engaged in – I learnt a lot.

You play by posting micro-forecasts (concise ideas – 140 characters, like twitter) of future possibilities, or build on and reshape other players’ ideas. Here’s a micro-forecast from last time by our very own Peter Griffin:

Screen shot 2012-06-07 at 12.45.41 PM_2 (1)

This provocative statement started a conversation that went in several directions:

Screen shot 2012-06-07 at 12.45.41 PM_2

You gain points and move up the game leader-board by posting ideas that create more discussion, contributing interesting ideas to the game and winning awards. This year Auckland University Press are offering copies of Get Off the Grass as prizes. Peter would have scored some points for his fore-cast, but so would have those who built on Peter’s initially card.

You can play for five minutes and share one idea, or play for the whole game and post hundreds of possible futures.  Anyone can play as long as they have an internet connection for their browser – players can register here in advance. There will be some public playing hubs in libraries, museums and other places where you can drop in and get the hang of playing and share the experience with others. I will be playing at Te Papa on Level 4 at one of our public hubs.

The conversation which produced the most discussion last year concerned the teaching of science in te Reo from a Maori perspective. What impressed me most was that the subsequent discussion appeared to change many people’s minds about this idea. The conversation tree is shown below – click here to view the tree on prezi or just click on the image to download:

image

So please join us this Thursday and Friday to play Pounamu! pounamu-pin

Getting Off the Grass Shaun Hendy Aug 06

6 Comments

Fonterra’s discovery of the bacterium that causes botulism in a batch of whey protein concentrate has alarmed many.  As the whey protein is an ingredient in popular infant formulas, many parents will be worried that they have inadvertently exposed their children to potentially fatal bacteria.  Hopefully, the recall of  products that use Fonterra’s whey ingredient will prevent any illness, even if it appears that these recalls may have been tardy.  One would also hope that Fonterra learns from the experience, because when Fonterra stumbles, so does the rest of the country.

Get-Off-The-Grass-08.inddThis latest incident illustrates once again how important it is that New Zealand diversify its economy.  In fact, this is the subject that I address in my upcoming book, Get Off the Grass, co-authored with the late Sir Paul Callaghan.  I’ll be launching the book with a public talk at Victoria University of Wellington on August 15th (register here if you would like to come along), with the paperback hitting bookstores the following day.

In Get Off the Grass, Sir Paul and I investigate why New Zealanders work harder and earn less than most other people in the developed world.  In Sir Paul’s previous book, Wool to Weta, this was framed as a choice:  we choose to be poor because of the types of industries that we prioritise, such as farming and tourism, earn us relatively little per hour worked. In Get Off the Grass, we use ideas from economic geography and the study of complex systems to investigate why it has been so hard to innovate our way out of these low productivity industries.

To illustrate just how specialised New Zealand’s economy is, I have borrowed a figure from Dr Helen Anderson, which compares the diversity of our exports with those of Denmark.  Like New Zealand, Denmark has a strong agricultural base.  Unlike New Zealand, Denmark has made concerted efforts to diversify its economy over the last few decades.  We are constantly told that New Zealand is too small to do everything, yet Denmark, a country with a population of only 5.5 million people, manages to do a heck of a lot more.

image(Figure courtesy of Dr Helen Anderson; Sources: Statistics NZ, StatBank Denmark, DataStream, NZIER, July 2012)

With a title like Get Off the Grass, it won’t surprise you that we argue that New Zealand can and should look to do an awful lot more than just agriculture.  Some of the points we make in the book are:

  • There is a deep flaw in our reliance on the 100% Pure brand.  We need the edge our clean, green brand gives us to sell our agricultural commodities at good prices, yet the production of these commodities actually damages the environment.  See this piece I wrote for Unlimited magazine last year.
  • Economic diversity is crucial for long-term economic stability, and this in turn is crucial for growth.  The fluctuations in our dollar caused by the contamination of one of our major exports illustrates why.  The volatility caused by such crises in turn hurts other export sectors, making it even harder to get off the grass.
  • Diversity is regarded as a crucial ingredient for innovation, so our strong focus on agricultural research actually makes us less innovative as a nation, whether in agriculture or otherwise. Physics and chemistry have contributed an awful lot to agriculture, but agricultural science has not returned the favour.
  • Specialisation in a single industry is just not a good long term strategy.  No industry stays on top forever, and if your favoured industry becomes too important to fail, it will prevent you moving into other industries before it’s too late.

Detroit, with its dependence on car manufacturing, is a classic example.  Although Detroit’s car industry has vast scale with the three biggest car makers in the US, the city is now a basket case because its mono-cultural manufacturing sector has failed to reinvent itself in the face of strong competition from overseas manufacturers.

As I said when the National Science Challenges were announced, our dependence on the primary sector leaves our economy perilously exposed to volatile commodity markets. Jacqueline Rowarth told Radio New Zealand that previous attempts to diversify our economy had failed.  Get off the grass – we’ve yet to make a serious attempt!

Complexity, emergence and networks Shaun Hendy Jul 11

No Comments

What do magnets, stock markets, and Facebook all have in common? With Get Off the Grass off to the printers, I now have some time to ponder such important questions. So tonight at 8.40pm, I’ll be back talking to Bryan Crump on Radio NZ Nights about what it is that these things share: namely, complexity. (You can listen the interview here.)

It’s complicated
We are surrounded by complicated things. It seems obvious that both the behaviour of the stock market, which is a result of many individual investment decisions made by thousands of investors, and the behaviour of a magnet, which is the aggregate of the magnetic properties of a very, very large number of individual atoms, are complicated.

What is much less obvious is that the stock market and a magnet should behave anything like each other. Yet this is what scientists have found: in certain circumstances, complicated systems that consist of many entities that interact with each other often exhibit similar patterns of collective behaviour.

What are these similarities? It turns out that statistically, the ups and downs of the stock market are similar to the microscopic fluctuations of the strength of a magnet like iron. Because individual atoms will occasionally flip the orientation of their own magnetic field, the net magnetic field of a collection of magnetic atoms will fluctuate. These fluctuations tend to be small, because if an individual atom flips, it will then experience a magnetic force from the other atoms that will eventually make it flip back again to line up with all the others.

If you heat the magnet up though, each atom in the magnet will jiggle more and is more likely to flip. The hotter the magnet becomes, the more the strength of its magnetic field will fluctuate. But if the magnet becomes too hot, it can actually lose its magnetic field altogether, because the flipping becomes so random that the tiny magnetic fields of each of the atoms cancel each other out.

Sell-offs, seagulls and sand
What has this got to do with the stock market? It turns out that investors can behave a little bit like atoms. Most of the time, investors tend to invest independently of each other. They make their own decisions to buy and sell stocks based on the prospects and performance of individual companies, without worrying too much about what everyone else is doing.

Just prior to a stock market crash, this behaviour changes. If you buy a stock at the point where everyone else is selling, then you will soon see the price of that stock drop below what you have just paid for it. This can look like a strong incentive to sell your stock before its price drops any further. If this starts happening across too many stocks, then investors will see the value of their stock portfolios falling and this can trigger an even larger sell-off. The value of the market plummets.

When the stock market behaves normally, investors act independently, just like atoms in a piece of iron that is too hot to have a magnetic field. When it crashes, investors all start doing the same thing – selling -  just like the atoms in a magnet that have all aligned their magnetic fields. You can also make mathematical analogies with flocks of birds, when they all fly together in the same direction. Even the avalanches that occur on the slopes of sand dunes have things in common with the movement of stock prices during a crash.

Systems that are normally so complicated that we might think of them as nearly random can, on occasion, start to act collectively. Stock markets can plummet in minutes, birds of a feather flock together, while atoms can align to produce powerful magnets. When systems start to behave coherently, scientists see complexity, not just complication. In other words, complexity is what results when the components of a complicated system start to behave in a collective, self-organised fashion. And remarkably, these complicated systems exhibit very similar behaviour when they self-organise.

Breaking symmetry
Despite examples like these, complexity remains a tricky concept to nail down. You know it when you see it, but it’s hard to come up with a single definition that encompasses all aspects of complexity that we see in nature and human society. Nonetheless, complexity has become an increasingly important concept in science over the last few decades.

One of the seminal articles in the field was written by theoretical physicist Philip W Anderson in 1972. Anderson noted that surprising behaviour can arise in systems that contain many interacting components, like the atoms in a magnet or investors in the market.  He pointed out that we can’t always understand such complex systems by focussing on their individual components.

When New Zealanders travel to Europe or North America, they often find themselves bumping into other people when they walk down a busy footpath. Kiwis tend to pass people on the left, while people overseas often pass on the right. At least for the first few days, this means we are constantly walking into people. It probably has something to do with the side of the road that we drive on, but this is not universal. In my experience, when the British are on foot, they seem to want to pass each other on the right despite the fact they drive on the left.

This is an example of what physicists call spontaneous symmetry breaking. It’s really only possible to walk down a busy street if we all agree on the way in which we’ll pass each other. Kiwis have made one choice, while people in other countries have made others, yet there is nothing in particular about any of us individually that says it has to be the left or the right – we just have to agree with those we walk past on a daily basis.

Something very similar happens in biology. Bio-molecules that are mirror images of each other are said to be left- or right-handed, by analogy with the way your left hand becomes your right when you look at yourself in the mirror. The chemistry of left or right-handed molecules is identical, at least when those molecules are in isolation or interact with chemicals that don’t have a handedness. However, in much the same way as you would find it hard to shake someone’s left hand with your right, left-handed molecules are not always able to react with right-handed molecules.

So biology has to make a choice. If life is going to work properly, it needs to stick to either left- or right-handed molecules, neither of which is preferred by the chemistry of the individual molecule. On Earth at least, life chose to be left-handed. So despite the fact that the building blocks of biology are chemicals, biology is not just applied biochemistry.

Complex societies
In other words, complex systems cannot be completely understood by studying their components in isolation. Understanding how one investor or molecule behaves in isolation won’t necessarily tell you why stock markets crash or how life works. The properties of complex systems, like the biosphere or the stock market, only emerge when the components of the system have to interact with each other.

Networks have become very important these days in our increasingly connected world. If you have read this far, it won’t surprise you that the networks that underpin both society and the economy also show complex, emergent behaviour. In recent times, studies of social networks like Facebook have led to some of the biggest advances in understanding complexity, but as with other complex systems, it is impossible to understand a network by considering just a single person in that network.

We have a lot more to learn about our society and the economy, but the lessons we should take from the study of complex systems is that we are not just a collection of individuals. Society is more than just the sum of its parts.

Valuing Science in New Zealand Shaun Hendy Mar 20

5 Comments

On April 3rd, the New Zealand Association of Scientists is holding its annual conference to ask “What is the value of science in NZ?” (you can register here). As the conference chair, Dr Nicola Gaston, puts it:

When scientists are asked to describe scientific research that isn’t done for short-term economic benefit, they call it blue-skies research, basic, fundamental, or sometimes investigator-led. But what do these terms mean to non-scientists? Is it perhaps time to discuss the value of the science that we do more explicitly, without necessarily resorting to economic jargon?

Nicola will have to grit her teeth, because I am going to make excessive use of economic jargon in this post. How do you put a value on science?

This is actually a very difficult question. If I asked you to place a value on a car, you might well go to Trade Me and see what cars of that particular make, model and year were selling for. This will give you an estimate of the market value of the car, and for many goods, this is a very good way of determining their value.

Unfortunately, it turns out that scientific knowledge cannot be valued this way. Unlike a car, many people can possess the same piece of knowledge and once this knowledge exists, it is hard to stop it spreading. Because it is difficult to have exclusive ownership of an idea, the market will pay less for that idea than it is worth to society as a whole. In fact, because the market undervalues knowledge in this way, a free market economy produces less scientific knowledge than society would like.

In other words, the social value of scientific knowledge is typically greater than its market value.

This is true even for scientific knowledge that has direct economic value. Economists have found that scientific knowledge produced by firms often spills over into others. Firms that develop valuable new technologies or products will soon find that others begin to copy them, forcing them to share the benefits of their discovery with others. As the market value of new knowledge represents its worth to an individual firm rather than to the economy as a whole, the economic value of scientific knowledge will often be greater than its market value.

Should governments do science?
This is not good news because markets are amongst the best tools we have for allocating resources in society. If markets are poor at valuing knowledge, how should we go about allocating resources to scientific research?

Most of us look first to our government to redress the market’s undervaluation of science. Indeed, governments have developed all sorts of tricks to deal with this. Patents, tax credits and R&D grants are all mechanisms that governments use to stimulate scientific research over and above that which the market will deliver.

Yet many of these tools rely on the government being able to determine the economic or social benefits of scientific knowledge, often in advance of the research itself. What then are Kiwis to make of their government, which funds far less science than the governments of most other advanced countries (see below) and often tries its best to rely on the market for estimates of the value of science?

image(Source: OECD, 2006)

A culture of knowledge
Well, you get the government you vote for. Our government’s reluctance to spend on research and development mirrors that of our private sector. Frankly, I think that New Zealanders place less value on scientific knowledge than the citizens of other countries. Attempts at getting Kiwis to place a value on science through initiatives like the National Science Challenges have met with a lukewarm response. Sadly, our politicians are well aware that the New Zealand public is ambivalent about science. Why campaign on increasing spending on science if no one cares?

I suspect that it is the countries that place a higher cultural value on scientific knowledge that vote in governments that are prepared to fund science generously. That these countries are also richer is perhaps not surprising – their cultural values compensate for the market’s underestimation of the value of knowledge.

So how could we create a country that places a higher value on knowledge? And would such a society be healthier, wealthier, and happier? Join us on April 3 in Wellington to discuss this further.

An unfortunate experiment in peer review Shaun Hendy Mar 10

8 Comments

A few months ago, Sir Peter Gluckman made the observation in a discussion paper (“Which science to fund: is it time to review peer review?”) that

While scientists pride themselves on objectivity, there is surprisingly little in the way of objective assessment of the nature and quality of peer review processes for grant allocation.

Ironically, under-resourcing at the Ministry of Business, Innovation and Employment last year has provided us with an opportunity to put one aspect of the peer review of grant proposals to the test. In the midst of yet another restructuring, the Ministry was unable to run a complete peer review process for the 299 proposals it received last year. The results of this incomplete process allow us to put peer review to the test.

Sir Peter’s paper is particularly concerned that the peer review process used in allocating grants may lead to an overly conservative decisions being made:

… the most innovative research tends to involve intellectual risk and thus can invite criticism, it is generally accepted that the general processes of grant awarding bias decisions towards conservatism …

Sir Peter suggests that bias can arise because:

… simple but positive reviews are often discounted as if the reviewer has not been serious in his/her evaluation. Conversely, simple but negative reviews carry extra weight in tight funding systems with low success rates.

The numbers

Last week, Radio New Zealand’s William Ray received information from the Ministry concerning the distribution of the number of reviews received by each proposal and the corresponding success rates. This data is shown in the plot below. In total, 298 proposals were sent out for review (one was ruled ineligible for other reasons), with each being subject to at least one review. The Ministry had aimed to obtain five reviews per proposal (three scientific reviews and two end user reviews), but in the end it only managed an average of 2.7.

image

Sir Peter’s hypothesis that negative reviews are weighted by assessment panels more heavily than those that are positive would mean that the more reviews a proposal receives, the less likely it is that it will be funded (all else being equal). The data is consistent with this, as nearly 35% of proposals that received one review were funded as opposed to around 25% of those that received more than one review. But is this difference statistically significant?

Because I am doing this at home on a Sunday, I will just use a one-sided 2-by-2 Fisher test. My null hypothesis is that there is no difference in the chances of success between proposals that receive one review and those that receive two or more. Applying the Fisher test gives me a p-value of 0.125, telling me that in the absence of bias, roughly one in eight funding rounds would produce a skew in success rate of the observed amount or more. This tells us that the observed difference cannot be regarded as statistically significant.

The conclusion

This does not allow us to rule out bias in peer review, of course, it just means that we can’t reject the hypothesis that it is absent. Thankfully we are told that the Ministry is better prepared this year to deal with the peer review process. With such a spread in the distribution of peer reviews, the process last year was very vulnerable to any bias in the way that panels weight peer reviews. If a peer review process is to be run, then the Ministry should strive to achieve a consistent number of reviews per proposal.

However, this does illustrate the possibility of conducting deliberate experiments that might allow us to test further for bias. Although not without difficulty and expense, it would be very interesting to compare the decision making of panels that received different numbers of peer reviews for the same set of proposals for instance. Let’s just hope we are not subject to further unplanned experiments by our Ministry!

 

Disclosure: I was a successful applicant in the funding round last year.

Marsden 2012: Success rate continues to fall Shaun Hendy Oct 25

10 Comments

This year’s Marsden Fund results were announced this morning.  The full list of successful proposals is available on the Royal Society of New Zealand website, or, if you prefer, you can get a sampling of what the media made of the lucky winners via the Dom Post and the Herald.  This year the success rate has dropped  to 7.7%, a half percent lower than last year and the first time it has been below 8%.

Although many of us would like to see the Marsden fund substantially increased, the figure below shows that the historically low success rates of the last three years have been driven by a large increase in the number of proposals received rather than a loss of funding. This increase in the number of proposals may reflect a reduction in the amount of funding available for investigator-led research across the system (note to self: see if it’s possible to use http://www.msi.govt.nz/update-me/who-got-funded/ to get some hard numbers on this!). However, I think we have also seen an increase in research activity within the scientific community, possibly driven by the Performance Based Research Fund.  A successful Marsden grant is now worth more to a university than its nominal book value.

While the number of applications has increased from last year, the total amount of funding available has remained essentially the same (see below). In fact, in real terms, the total funding awarded is about 16% more than it was a decade ago. However, over this decade, the amount of funding awarded for a full proposal has increased by 22%.  Although there are more of them, the Marsden fund’s dollars buy less science these days.

The last decade has also seen a steady increase in the proportion of funding awarded through the fast-start scheme for early career researchers (defined as those who were granted their PhD in the last seven years).  However, the share of the funding allocated to the fast-start scheme flattened off this year as is shown below.  As I noted last year, the loss of the FRST post-doctoral fellowship scheme and the International Mobility Fund means that young researchers are even more dependent on fast-start funding as they establish their careers.  I suspect the Marsden fund is now playing a much bigger role in supporting the vitality of the science system than it did a decade ago.  Is the drop in success rate over the last few years a sign of other stresses in the science and innovation system?

This year was my last as a panellist on the Physics, Chemistry and Biochemistry panel.  Being a panellist is hard work.  Reading on the order of 100 proposals in the first round (and later another 20 or so in the second round) is a time consuming process, and of this 100, less than 10 will eventually be funded.  As an applicant, you can do both yourself and the panel a favour by making your proposal as readable as possible – make use of your colleagues to help you do this, but not just those closest to you.  Perhaps try to select a spread of proof-readers that reflects the expertise of the panel. In the end, however, it is always very evident that there are more great proposals than there are dollars.  If you were unsuccessful this year, see if you can obtain internal funding to develop your ideas further.

(Disclosure: I was a Principal Investigator on one Marsden funded project that finished this year).

Pounamu Shaun Hendy Jun 01

1 Comment

HomeNext week on June 7-8, we will be running an on-line game called Pounamu using the Institute for the Future’s foresight engine.  The engine brings a large community of people to come together to investigate, explore and discuss a future scenario.  Pounamu invokes a future New Zealand that is trying to make its way in the world by drawing on its wits.  Not so different to today, maybe, except that the growth of the web information has become much more accessible, while the skills that it takes to turn this torrent of information into useful knowledge have become more highly prized.  In Pounamu, New Zealand must learn to export knowledge not nature.

We will be running the game in conjunction with the Transit of Venus Forum that take will take place in Gisborne at the same time.  While the Forum will be webcast live, the game will allow a much broader cross-section of New Zealanders to participate actively in the discussion of New Zealand’s future.

I am really looking forward to it – a few of us had a blast last week putting the game through its paces.  My forecast that everyone would be sharing their DNA profile on facebook was hotly debated.  The pros: by sharing your DNA data openly, you’ll be helping researchers understand health and disease, and maybe you’ll get advice on how to change your lifestyle to avoid triggering a genetic predisposition to a disease.  The cons: what will your health insurance provider do with that information? And some people didn’t want to know whether they had a genetic condition, even if this knowledge meant they could do something about it.

What do you think? Tell us by registering and then playing Pounamu on June 7-8!

Network-wide options by YD - Freelance Wordpress Developer