# Seeing Data

## Whale Fall (after life of a whale)Chris McDowallOct 15

1 Comment

I find this video both informative and astonishingly beautiful. The film-maker uses paper cutout puppets to describe what happens to a whale’s carcass after the creature dies. Music provided by the excellent Rachel’s.

Enjoy in full-screen with a cup of coffee.

## Plotting geographic data on a world map with PythonChris McDowallAug 12

Last week I promised to write a blog post detailing how I created this public transport animation. On reflection, it’s a topic best dealt with over a few sessions. Let’s start simple. How might you plot lots of geographic data on a map? In this post I will show you how to programmatically create a map of the World’s top ten most populated cities. It will end up looking something like this.

This tutorial involves programming with the Python language and some additional modules. If you have never programmed before it might be a tad confusing. I suggest you first read one of the many fine introductory Python tutorials to get your head around the language. The rest of this post will assume that you understand how to install and import modules, how to write and run a script and the fundamentals of Python data structures.

Before we begin you will need to install the following:

• Python 2.5, 2.6 or 2.7.
• NumPy (a Python extension that adds support for multi-dimensional arrays and a host of whiz-bang high-level mathematical operations).
• Matplotlib (a flexible 2D plotting library for Python that “tries to make easy things easy and hard things possible”).
• Basemap (a Matplotlib extension for plotting data on geographic projections).

Right, let’s make a map. Wikipedia has a list of World metropolitan areas by population that can serve as a nice test dataset. I will demonstrate how to turn this data into a labelled proportional symbol world map.

First we need to import our various libraries:

```from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import numpy as np```

Next we will set up our map. Basemap supports many different geographic projections. For this exercise I am using the Robinson projection.

```# lon_0 is central longitude of robinson projection.
# resolution = 'c' means use crude resolution coastlines.
m = Basemap(projection='robin',lon_0=0,resolution='c')
#set a background colour
m.drawmapboundary(fill_color='#85A6D9')```

Basemap comes packaged with some global geographic data at four resolutions: ‘crude’, ‘low’, ‘medium’ and ‘high’. Let’s draw the continent and country datasets using the ‘crude’ outlines.

```# draw coastlines, country boundaries, fill continents.
m.fillcontinents(color='white',lake_color='#85A6D9')
m.drawcoastlines(color='#6D5F47', linewidth=.4)
m.drawcountries(color='#6D5F47', linewidth=.4)```

We can also ask Basemap to draw lines of longitude and latitude.

```# draw lat/lon grid lines every 30 degrees.
m.drawmeridians(np.arange(-180, 180, 30), color='#bbbbbb')
m.drawparallels(np.arange(-90, 90, 30), color='#bbbbbb')```

Populate three arrays of equal length with the latitude, longitude and population values for each city. Normally we would read the data from a file, database or service but in the interest of simplicity I have typed them directly into the script.

```# lat/lon coordinates of top ten world cities
lats = [35.69,37.569,19.433,40.809,18.975,-6.175,-23.55,28.61,34.694,31.2]
lngs = [139.692,126.977,-99.133,-74.02,72.825,106.828,-46.633,77.23,135.502,121.5]
populations = [32.45,20.55,20.45,19.75,19.2,18.9,18.85,18.6,17.375,16.65] #millions```

Use our basemap object to convert the latitude/longitude values into map display coordinates.

```# compute the native map projection coordinates for cities
x,y = m(lngs,lats)```

Multiply each population by itself to create a scaled list of values. These will be our circle display sizes.

```#scale populations to emphasise different relative pop sizes
s_populations = [p * p for p in populations]```

Use the matplotlib scatter function to plot the circles. Note the use of the zorder parameter. This ensures that the scattered circles will be rendered on top of the continents.

```#scatter scaled circles at the city locations
m.scatter(
x,
y,
s=s_populations, #size
c='blue', #color
marker='o', #symbol
alpha=0.25, #transparency
zorder = 2, #plotting order
)```

Loop though the unscaled population values and the display coordinates. Label each circle with the city population rounded to the nearest million people.

```# plot population labels of the ten cities.
for population, xpt, ypt in zip(populations, x, y):
label_txt = int(round(population, 0)) #round to 0 dp and display as integer
plt.text(
xpt,
ypt,
label_txt,
color = 'blue',
size='small',
horizontalalignment='center',
verticalalignment='center',
zorder = 3,
)```

```#add a title and display the map on screen
plt.title('Top Ten World Metropolitan Areas By Population')
plt.show()```

Run your script and you should see a map. Lovely. I recommend you experiment further by tweaking parameters, importing other datasets and using alternative plotting methods.

This is the basic overview of how I typically plot geographic data in Python. Next time I’ll take this a step further and show you how to map, export and animate temporal geographic data.

## Animating Auckland’s public transport network – Take TwoChris McDowallAug 04

In late January I created an animation of Auckland’s public transport network with data from the MAXX Auckland transport Google transit feed. As I noted in the post, there were several issues with the video. I carved out a little time this week to revise the animation and I am happy with the progress. The new video is embedded below.

There is a lot of detail in this animation. It’s best viewed in high definition with fullscreen mode on.

Version Two distinguishes between buses (teal), ferries (blue) and trains (red). I also tidied some of the more obvious errors with the ferry route geometry data. This involved manually tweaking the route geometries stored in the transit feed “shapes.txt” table for many of the ferries. I still need to adjust a few (I’m looking at you, Rangitoto Island). I have not corrected the bus data. The main issue with bus data are the erroneous harbour crossings from the North Shore to the Auckland CBD.

Next week I will provide an overview of how the animation was created.

## Asteroid 2011 MD passes through the Earth’s GPS satellite networkChris McDowallJun 27

Tomorrow the ten metre wide Asteroid 2011 MD will pass within 12,000 km of Earth. How close is that? Well, it’s close enough that it will travel through the network of GPS satellites that circle the Earth. Pasquale Tricarico, a Research Scientist at the Planetary Science Institute in Tucson, Arizona, has created a series of animations describing the event’s geometry.

I have embedded my favourite visualisation from the series below. It depicts the encounter from the point of view of the asteroid. Amazing.

Thanks to Michele Bannister for sharing the link.

## How do you Visualise a Conversation?Chris McDowallJun 24

Over the next couple of days I am helping run an online event called Magnetic South – an online game about the long term future of Christchurch. It is part of Christchurch City Council’s Share an Idea suite of initiatives and will input to the development of the Central City Plan. The game is being run on the ‘Foresight Engine’, a platform created by the Institute for the Future to support people thinking together about issues that are important to them in a way that is both productive and fun. If you want to explore possible futures for Christchurch then please create a player and get started!

A big challenge for the Magnetic South team involves making sense of the many micro-forecasts people contribute during the event. Previous Foresight Engine events have produced thousands of individual contributions. The nature of the game is such that these snippets of texts should not be read in isolation. Most player contributions build off earlier forecasts and, in turn, prompt additional responses. To unpack what happened during the Magnetic South event the various micro-forecasts need to be contextualised.

To assist our analysts I have written some code to visualise the structure of the many conversations that occur. My initial design represented each conversation as a sideways tree constructed from squares and lines. Each square represents a contribution someone made to the conversation. Filled squares represent contributions from the person who started the conversation. Unfilled squares represent contributions made by other participants. Lines connect related contributions — the square on the right being a response to the square on the left.

Here is a very simple example based on data created during an earlier Foresight Engine event that explored a future where water is as expensive as energy. The example conversation consists of just three comments. Read from left to right, someone kicks off the conversation with a Dark Imagination forecast. The unfilled square represents someone else responding to the original forecast at some point in the event. The rightmost square signifies original forecaster replying. There were no further contributions to this conversation.

Below is a slightly more complex example. Someone makes a micro-forecast concerning Capturing all sources of energy that are currently not very considered: e.g. muscle power at gyms (rowers etc.), heat from car engines…’ This prompts two responses, one of which (‘They are here and now. The Piezoelectric powered Disco in London is my favourite example’) leads to an extended back and forward exploration of the idea.

Although they showed the scale of the conversation, there was no way to know how many people took part. Yesterday afternoon I added some new code to add numeric codes into the boxes. Each time a new person enters a conversation they get a number. The person who started the conversation is number 1. The first person to respond is number 2… and so on. This makes it possible to see things like “how many people were involved in the conversation” or “was this conversation dominated by a couple of people, or are there are a whole heap of folks chipping in”? Here is the same conversation rendered with the new code.

It is now possible to see that that five people took part in the conversation. Most of the exchanges actually occurred between the second and the third participants. A couple of other people (numbers 4 and 5) made a single contribution.

I want to emphasise that these figures are not intended to be read in isolation from text generated during the game. Instead they are tools to provide researchers with an overview of the structure of the many conversations that occur to help them navigate and contextualise the raw text. Specifically, I am attempting to convey the scale, complexity and rhythm of the various exchanges that happen inside the event. Throughout today’s Magnetic South event I will create similar diagrams and combine them with textual descriptions and share them through the game blog. Please get involved  we would love to hear your ideas on the future of Christchurch!

Here are some more diagrams that I created from the E=H20 event data. Even in the absence of text, they convey a sense of what happened.

## How do you explain a city?Chris McDowallJun 04

```"To tell you about Penthesilea I should begin by describing the entrance to the city. You, no doubt, imagine seeing a girdle of walls rising from the dusty plain as you slowly approach the gate, guarded by customs men who are already casting oblique glances at your bundles. Until you have reached it you are outside it; you pass beneath an archway and you find yourself within the city; its compact thickness surrounds you; carved in its stone there is a pattern that will be revealed to you if you follow its jagged outline."```

` `

```"If this is what you believe, you are wrong: Penthesilea is different. You advance for hours and it is not clear to you whether you are already in the city's midst or still outside it. Like a lake with low shores lost in swamps, so Penthesilea spreads for miles around, a soupy city diluted in the plain; pale buildings back to back in mangy fields, among plank fences and corrugated-iron sheds."```
Italo Calvino, Invisible Cities

Urban Earth is based on a simple idea. Walk a transect across a major city and take a photograph every eight steps. The end result is a set of silent miniature video documentaries that provide impressions of everyday life in a broad range of urban environments. They also convey a sense of the scale and complexity of the cities and suburbs we have collectively constructed. I half-wish there were annotations, mini-maps and other snippets of information layered over the imagery, but I love it.

LONDON

A photograph is taken every 8 steps (roughly) across each city and then edited together to make films which reveal an alternative way of experiencing cities. Each photo is always taken looking directly forward without bias, presenting an urban view which is emotionally challenging for the photographer whose gaze is drawn towards specific people, objects and places.Urban Earth

MEXICO CITY

Geography is more important than many people think. A random route across a city may expose many things, but an URBAN EARTH walk is special because it attempts to reveal what a city is like for the people who live in it. URBAN EARTH is not about following the tourist trail or tracking down the most extreme places… it is about finding normality.Urban Earth

Thanks to Matt Ball for sharing.

## Visualising the New Zealand Budget 2011 with TreemapsChris McDowallMay 20

It is really difficult to grasp the significance of lots of big numbers. It is even trickier when the numbers are organised in a hierarchy. For example, yesterday afternoon Bill English, the Minister of Finance, delivered his third budget, outlining the nation’s revenues and expenses. The budget includes details such as how the government plans to spend \$21 billion dollars on social development in the coming year of which \$9.5 billion will be spent on superannuation, almost \$1.9 billion on the domestic purposes benefit, nearly \$1.6 billion on accommodation assistance… and I’m already lost.

Treemapping is a technique for visually comparing groupings of numbers. A treemap represents a hierarchy of numeric values as a set of nested shapes – usually rectangles. I’m fond of this technique for several reasons, not least because it plays on existing associations. Big rectangles represent big numbers. Rectangles inside other rectangles indicate that one thing is part of another thing. Rectangles can even be shaded to depict an additional variable, for example, an increase or decrease over time.

Last night, with the assistance of Keith Ng, I created an interactive treemap of the New Zealand budget. It is available here. I encourage you to have a play with it. Unfortunately, given the large volume of data that needs to be processed and rendered, the visualisation struggles on every browser except Google Chrome. If you have Chrome on your computer, I highly recommend that you use it to view the visualisation.

There are two levels to the treemap. The top level shows the breakdown of various expenditure areas. The bigger the rectangle, the more money is being spent. Green rectangles indicate that spending has significantly increased since the last budget; red rectangles indicate a decrease. When you click on any spending area, the visualisation will dive into that area and show a detailed breakdown.

If you are interested in learning more about treemapping, I suggest you read Ben Shneiderman’s account of how he developed this class of visualisation and Thomas Kerwin’s survey of treemap techniques. Wikipedia has handy list of software if you want to create your own treemaps.

## Ring of Fire – Animated Map of World Earthquakes (Jan 1 – Mar 12, 2011 GMT)Chris McDowallMar 13

This video was made very quickly and could use some work. I post it here in case you find it interesting. I suggest you watch it full-screen, in high definition with scaling off.

The animation depicts two and half months of 2011 USGS earthquake data. Blue circles represent deep seismic activity recordings (>= 40km deep). Red circles represent shallow seismic events (<40km deep). Each event leaves behind a red dot to show the overall pattern. The animation ends the day after the 8.9 quake that hit Japan on March 11 and includes the shallow 6.3 Christchurch quake.

I am keen to spend some time improving this animation. Perhaps I will find some time in the coming weeks. I intend to release the programming code as an open source project – I would love to see the community build on this stuff.

## New York subway system as a stringed instrumentChris McDowallFeb 03

A short post to draw your attention to some lovely work by Alexander Chen. Conductor: mta.me pulls subway schedule data from New York City’s Metropolitan Transportation Authority and transforms it into dynamic musical animation. The end result is wonderful.

Thanks to commenter Kate for sending me the link.

Here’s Chen’s description of the work:

“The piece follows some rules. Every minute, it checks for new trains launched from their end stations. The train then moves towards the end of the line, with its speed set by the schedule’s estimated trip duration. Some decisions were made for musical, aesthetic, and technical reasons, such as fading out routes over time, the gradual time acceleration, and limiting the number of concurrent trains. Also, I used the weekday schedule. Some of these limitations result in subtle variations, as different trains are chosen during each 24-hour loop.

The system has changed since 1972, and some lines no longer exist. For example, the 8 train, or the Third Ave El, was shut down in 1973. The former K train was merged into other routes. I decided to run these ghost trains between 12am-2am.”

## An animated map of Auckland’s public transport networkChris McDowallJan 20

I grew up in Auckland and spent many years using the city’s public transport system to get from place to place. While sitting on a bus, I sometimes wondered what the transportation network would look like if we could see the movements of the individual vehicles from the air. I would try to visualise the aggregate trajectories as each vehicle carved a path through space-time. After a few moments I would get hopelessly overwhelmed and go back to reading my book.

Last year Auckland Transport published its Google Transit Feed data on the MAXX website and I realised it provided the information I needed to make the map I used to daydream about. The data is a series of (large) spreadsheets, each describing a different aspect of the bus, train and ferry network. Last week I downloaded the spreadsheets and wrote some software to transform the data into an animated map.

As usual, I recommend you full screen the video and watch it in high definition.

The animation begins at 3am on a typical Monday morning. A pair of blue squiggles depict the Airport buses shuttling late night travellers between the Downtown Ferry Terminal and Auckland International. From 5am, a skeleton service of local buses begins making trips from the outer suburbs to the inner city and the first ferry departs for Waiheke Island. Over the next few hours the volume and frequency of vehicles steadily increases until we reach peak morning rush hour. By 8am the city’s major transportation corridors are clearly delineated by a stream of buses filled with commuters. After 9am the volume of vehicles drops a little and stays steady until the schools get out and the evening commute begins. The animation ends at midnight with just a few night buses moving passengers away from the central city.

Some things to note:

• The steady pulse of the Devonport Ferry.
• The speed at which buses hurtle down the Northern Motorway’s new bus lanes.
• The interplay between buses and ferries on Waiheke Island.
• The sheer mind-boggling complexity of the system.

Please note that there are a few quirks in the animation. A couple of ferry services pass across the land and there is at least one erroneous harbour crossing. These errors are not problems with the MAXX schedule – they are errors introduced by missing geometries. I will attempt to rectify these issues at a later date.

Network-wide options by YD - Freelance Wordpress Developer