For a country that takes sport too seriously, lives or dies on the international price of dairy products, and has a Prime Minister so weird that international TV hosts are fascinated by him, it is unusual for a social science research project to be one of the stories of the week.
But that happened last week, with a barrage of criticism of a survey hosted by TVNZ and conducted in conjunction with Vox Pop labs from Canada and a number of academics from Victoria University of Wellington and the University of Auckland. Critics include politicians I admire, comedians that used to beat me in debating competitions, and even well-loved cousins (who happen to be nationally significant political scientists). A great summary of the issues is provided by Bryce Edwards.
There were a number of concerns that were raised about Kiwimeter but the most central is that some of the questions are racist. In particular, many critics singled out the item:
Māori should not receive any special treatment
Many critics have argued that this statement implies that Māori do receive something that can be called “special treatment,” and respondents are being asked if they believe that this state of affairs should or should not be the case. Given that we as a nation have the Treaty of Waitangi as our founding document, it is surely inappropriate to view the rights of either Māori or Pākehā as “special treatment.”
Guided by data
I’m a Professor of Psychology. My expertise is not in survey methodology or multivariate statistical modelling. However let me give a view on some of the issues surrounding Kiwimeter. Full disclosure: one of the researchers is my colleague in the School of Psychology at the University of Auckland.
The purpose of a survey like Kiwimeter is to get an accurate representation of how people actually think about their world (rather than how they might like to pretend that they do). This turns out to be an incredibly difficult problem to solve. My colleagues who run the NZ Attitudes and Values Survey, a similar type of survey, have a great paper out on exactly how this process works.
One important thing is that to be successful, you have to be guided by data rather than personal preference. So you create thousands of questions and then test them to find the ones that give you the most stable estimates of people’s attitudes; these are also questions that best differentiate people from each other.
Equally, you end up with a group of items that fit together like a jigsaw. As a group they provide the most stable measurement of attitudes, and they allow you to see how different groups of people relate to each other.
Items like these come from all sorts of places. A lot of them come from speeches and writings of prominent New Zealanders. But if you want to accurately estimate attitudes to different groups and views of inclusion/exclusion, you need statements from across the political spectrum. Not all of these statements will be endorsed by legal frameworks and points of view that value indigenous and multicultural perspectives. But that’s likely to be necessary since not all individuals in New Zealand endorse these views.
Another criticism is why the questions tended to ask about attitudes in relation to Māori and immigrants, but not in relation to Pākeha. Giovanni Tiso asks why:
we are never asked if ‘a history of discrimination has created conditions that make it disproportionately easy for Pākehā to be successful’
As far as I know, there is no reason that this question shouldn’t be asked. I don’t know if it, or similar questions, was tested. But if it was, there is no guarantee that it would have emerged as one that provided stable measurement. A researcher simply can’t decide to ask a specific question in a survey; the question itself has to function effectively. It is likely that the question regarding “special treatment” for Māori functioned very well to separate different groups of people, based upon their responses (and if it didn’t, it wouldn’t have been there).
Academics and the media
Bronwyn Hayward raises the question of the relationship between a media organization and the academic researchers. While questions relating to racism are valid issues of academic investigation, are they compromised when the vehicle through which they are conveyed is a national media channel with a vexed history of presenting interests from non-dominant populations? This is a complex problem that gets directly to the issue of how academics engage with external organizations to facilitate research. The benefits of the relationship with TVNZ come from the fact that more than 100,000 people have taken the survey; getting such a large sample is no doubt extremely valuable to the academics involved. To study issues of racism and openness to immigration you need to sample as widely as you possibly can. But one can reasonably understand why someone of any ethnicity, and particularly Māori, might find this statement offensive. At the very least, the survey could have given a better context for the questions, equipping people with the knowledge that they might find questions worded in an objectionable way.
Tze Ming Mok points out that these effects are knowable, and can be assessed by pre-testing of the survey. While this is true, again the problem is that all the items fit together to create its overall reliability. And while some people clearly have found at least some items offensive, many more have completed the survey and so may not have felt the same way. Tze Ming is more knowledgeable about these matters than me but I’m not convinced that the Kiwimeter team would have gained this evidence as easily as she argues.
The discussion around Kiwimeter has been really interesting since it is unusual to have a wide societal discourse about psychometrics. What we should keep in mind is the importance of measuring psychological constructs that some members of society find offensive. Last year I blogged about a study using very similar methodology showing that Māori who were “visibly Māori” (e.g., endorsed items such as “people are not surprised to learn that I’m Māori”) were much less likely to own their home than Māori who self-reported fewer Māori features.
This work risked offense to some of those who took the survey but ultimately found evidence for institutional racism in the NZ banking sector. Hidden under its slick but dorky exterior, Kiwimeter offers a rich set of data that can help inform us about ways to support our most disenfranchised citizens.