Data data everywhere

By Aaron Schiff 30/03/2015


Wiki New Zealand launched its redesigned website yesterday and it looks fantastic. I know that a huge amount of effort by some very talented people went into the new site, and that certainly shows in the final result.

WikiNZ describes its mission as “democratising” data by “pulling together New Zealand’s public sector, private sector and academic data in one place and making it easy for people to use in simple graphical form for free”.

Given the amount and variety of data out there, this is a challenging mission. The way I see it there are two main challenges — helping people to find and discover data and presenting it to them, and collating all that data in the first place.

On the discovery problem, I like how WikiNZ has basically embraced the diversity of data and uses a Pinterest-style presentation on its front page by showing a random collection of charts.

wikinz1

This makes it quick and easy to browse the charts, and you can find interesting stuff without much effort. For example this pair of charts got shared a lot on social media yesterday:

wikinz2

In terms of presentation of data, WikiNZ is strictly about the facts – you get the chart and an option to download the data but there is no gloss or commentary. This is in keeping with WikiNZ’s character, although it does mean you need to supply your own data interpretation skills.

In harvesting data for the site, the key challenge is that a lot of public data is in various formats that are not always clean and ready to use. Spreadsheets are common, and within that category there is a huge variety of layouts and styles for presenting the data. Generally these are designed for humans to use directly, without consideration for how easy or hard it is to extract the data with software.

WikiNZ’s approach to this problem is very pragmatic – they have embraced the chaos and developed their own tools for extracting data from all kinds of formats and layouts and turning it into something simple and useful. This video demonstrates and explains their system “Grace”:

I was lucky enough to be given access to a beta version of Grace and I can attest that it works very well and saved me a lot of time. I’m hoping WikiNZ will license it for commercial use 🙂

All up, a super effort by the WikiNZ team, and as they say this is just the beginning.

Editor’s note: we continue to work through Aaron’s excellent back-catalogue of posts.