How did the pollsters go?

By David Winter 28/11/2011

A couple of weeks ago I wrote a post comparing the various polling companies that work in New Zealand, trying to see if there were any systematic biases in the results each company produces. In that post, it wasn’t possible to compare each company’s estimates to an external standard because, without an election its impossible to know exactly what the population is thinking. On Saturday night we got to know just what the electorate was thinking, and there were a couple of surprises. 
So, here’s the last estimate of each polling company compared with the results from Saturday night for each of the four parties that made the 5% cut:

The solid line is the election result for each party. Dashed lines are a 95% margin of error for a poll with 1 000 respondents and the given level of support

As you can see, the polls had National too high and New Zealand First too low more or less across the board. It’s hard to say any company did better than the others, Roy Morgan got New Zealand First about right and were the closest to National’s final result, but they made up for that by over-estimating the Green Party and under-estimating Labour.

So, why did most of the companies get New Zealand First and National wrong? Of course, it’s possible that the polling companies had it right, and that people across the nation changed their minds once they had the orange felt-tip pen in their hands. It could also have something to do with who bothered to turn up on Saturday. In the run up to the election, polling companies generally reported around 15% of the electorate were undecided but about 25% didn’t vote. So, some of the people who recorded a preference in a poll didn’t bother to have that preference recorded in the election. As far as I know, New Zealand polling companies don’t ask any questions to determine how likely a respondent is to vote, and ignoring this information could bias their results.

It’s not possible to reconstruct the motivations of those people that didn’t show up on Saturday, but we can look at some data at a coarser grain. If people that express a preference for New Zealand First were more likely to vote than the electorate at large (and so, not including a ‘likely voter’ question in polling would lower their estimate in a poll) then you might expect electorates that really pulled for that party to have higher turnout. It’s not perfect, but it’s all we’ve got, and it doesn’t really fit the story. National actually had stronger support in electorates with high turnout and there isn’t much going on with New Zealand First (I estimate about a 0.8% increase in support votes per 1 000 voters, but that’s not near to significantly different from no increase):

Percentage turnout is calculated relative to total enrollment for electorate (special votes not yet included)

By the way, that graph, with such poor turnout in the electorates that with the largest proportion of Labour voters, might be depressing viewing for Labour supporters. Indeed, I’m sure they’ll want to work on “getting out the vote” in the Māori electorates and South Auckland, but that pattern doesn’t explain the erosion of Labour’s support in this election. Turnout was low in these electorates in 2008. Here’s the support for each party plotted against change in turnout for each electorate (not including special votes):

0 Responses to “How did the pollsters go?”

  • Hi David,

    Interesting stuff! Could you please upload the party vote, percent turnout, change in turnout dataset? I’ll be traveling and it is always cool to have a dataset to play around (Does it sound too geeky)?

    The analysis may need to consider the size of the electorates, because it could explain part of the variation you are observing.


  • Cheers Luis,

    Just emailed you the files i have (if anyone else wants them, I’d be happy to send them, and might chuck them up on dropbox or similar)

  • Nicely done David!

    I was planning on doing much the same thing for the voter turnout data and you’ve kindly gone and saved me the effort 🙂

    Interesting to note the Greens increased their party vote share despite the negative trend in the final figure (albeit a weak one). It seems to indicate something of a genuine shift towards them since last election.