Last week I promised to write a blog post detailing how I created this public transport animation. On reflection, it’s a topic best dealt with over a few sessions. Let’s start simple. How might you plot lots of geographic data on a map? In this post I will show you how to programmatically create a map of the World’s top ten most populated cities. It will end up looking something like this.
This tutorial involves programming with the Python language and some additional modules. If you have never programmed before it might be a tad confusing. I suggest you first read one of the many fine introductory Python tutorials to get your head around the language. The rest of this post will assume that you understand how to install and import modules, how to write and run a script and the fundamentals of Python data structures.
Before we begin you will need to install the following:
- Python 2.5, 2.6 or 2.7.
- NumPy (a Python extension that adds support for multi-dimensional arrays and a host of whiz-bang high-level mathematical operations).
- Matplotlib (a flexible 2D plotting library for Python that “tries to make easy things easy and hard things possible”).
- Basemap (a Matplotlib extension for plotting data on geographic projections).
Right, let’s make a map. Wikipedia has a list of World metropolitan areas by population that can serve as a nice test dataset. I will demonstrate how to turn this data into a labelled proportional symbol world map.
First we need to import our various libraries:
from mpl_toolkits.basemap import Basemap import matplotlib.pyplot as plt import numpy as np
# lon_0 is central longitude of robinson projection. # resolution = 'c' means use crude resolution coastlines. m = Basemap(projection='robin',lon_0=0,resolution='c') #set a background colour m.drawmapboundary(fill_color='#85A6D9')
Basemap comes packaged with some global geographic data at four resolutions: ‘crude’, ‘low’, ‘medium’ and ‘high’. Let’s draw the continent and country datasets using the ‘crude’ outlines.
# draw coastlines, country boundaries, fill continents. m.fillcontinents(color='white',lake_color='#85A6D9') m.drawcoastlines(color='#6D5F47', linewidth=.4) m.drawcountries(color='#6D5F47', linewidth=.4)
We can also ask Basemap to draw lines of longitude and latitude.
# draw lat/lon grid lines every 30 degrees. m.drawmeridians(np.arange(-180, 180, 30), color='#bbbbbb') m.drawparallels(np.arange(-90, 90, 30), color='#bbbbbb')
Populate three arrays of equal length with the latitude, longitude and population values for each city. Normally we would read the data from a file, database or service but in the interest of simplicity I have typed them directly into the script.
# lat/lon coordinates of top ten world cities lats = [35.69,37.569,19.433,40.809,18.975,-6.175,-23.55,28.61,34.694,31.2] lngs = [139.692,126.977,-99.133,-74.02,72.825,106.828,-46.633,77.23,135.502,121.5] populations = [32.45,20.55,20.45,19.75,19.2,18.9,18.85,18.6,17.375,16.65] #millions
Use our basemap object to convert the latitude/longitude values into map display coordinates.
# compute the native map projection coordinates for cities x,y = m(lngs,lats)
Multiply each population by itself to create a scaled list of values. These will be our circle display sizes.
#scale populations to emphasise different relative pop sizes s_populations = [p * p for p in populations]
Use the matplotlib scatter function to plot the circles. Note the use of the zorder parameter. This ensures that the scattered circles will be rendered on top of the continents.
#scatter scaled circles at the city locations m.scatter( x, y, s=s_populations, #size c='blue', #color marker='o', #symbol alpha=0.25, #transparency zorder = 2, #plotting order )
Loop though the unscaled population values and the display coordinates. Label each circle with the city population rounded to the nearest million people.
# plot population labels of the ten cities. for population, xpt, ypt in zip(populations, x, y): label_txt = int(round(population, 0)) #round to 0 dp and display as integer plt.text( xpt, ypt, label_txt, color = 'blue', size='small', horizontalalignment='center', verticalalignment='center', zorder = 3, )
Finally, add a title and display the map on your screen.
#add a title and display the map on screen plt.title('Top Ten World Metropolitan Areas By Population') plt.show()
Run your script and you should see a map. Lovely. I recommend you experiment further by tweaking parameters, importing other datasets and using alternative plotting methods.
This is the basic overview of how I typically plot geographic data in Python. Next time I’ll take this a step further and show you how to map, export and animate temporal geographic data.
In late January I created an animation of Auckland’s public transport network with data from the MAXX Auckland transport Google transit feed. As I noted in the post, there were several issues with the video. I carved out a little time this week to revise the animation and I am happy with the progress. The new video is embedded below.
There is a lot of detail in this animation. It’s best viewed in high definition with fullscreen mode on.
Version Two distinguishes between buses (teal), ferries (blue) and trains (red). I also tidied some of the more obvious errors with the ferry route geometry data. This involved manually tweaking the route geometries stored in the transit feed “shapes.txt” table for many of the ferries. I still need to adjust a few (I’m looking at you, Rangitoto Island). I have not corrected the bus data. The main issue with bus data are the erroneous harbour crossings from the North Shore to the Auckland CBD.
Next week I will provide an overview of how the animation was created.
Tomorrow the ten metre wide Asteroid 2011 MD will pass within 12,000 km of Earth. How close is that? Well, it’s close enough that it will travel through the network of GPS satellites that circle the Earth. Pasquale Tricarico, a Research Scientist at the Planetary Science Institute in Tucson, Arizona, has created a series of animations describing the event’s geometry.
I have embedded my favourite visualisation from the series below. It depicts the encounter from the point of view of the asteroid. Amazing.
Thanks to Michele Bannister for sharing the link.
How do you explain a city? Jun 04
"To tell you about Penthesilea I should begin by describing the entrance
to the city. You, no doubt, imagine seeing a girdle of walls rising from
the dusty plain as you slowly approach the gate, guarded by customs men
who are already casting oblique glances at your bundles. Until you have
reached it you are outside it; you pass beneath an archway and you find yourself
within the city; its compact thickness surrounds you; carved in its stone there is
a pattern that will be revealed to you if you follow its jagged outline."
"If this is what you believe, you are wrong: Penthesilea is different. You
advance for hours and it is not clear to you whether you are already in the city's
midst or still outside it. Like a lake with low shores lost in swamps, so
Penthesilea spreads for miles around, a soupy city diluted in the plain; pale
buildings back to back in mangy fields, among plank fences and corrugated-iron
Italo Calvino, Invisible Cities
Urban Earth is based on a simple idea. Walk a transect across a major city and take a photograph every eight steps. The end result is a set of silent miniature video documentaries that provide impressions of everyday life in a broad range of urban environments. They also convey a sense of the scale and complexity of the cities and suburbs we have collectively constructed. I half-wish there were annotations, mini-maps and other snippets of information layered over the imagery, but I love it.
“A photograph is taken every 8 steps (roughly) across each city and then edited together to make films which reveal an alternative way of experiencing cities. Each photo is always taken looking directly forward without bias, presenting an urban view which is emotionally challenging for the photographer whose gaze is drawn towards specific people, objects and places.” Urban Earth
“Geography is more important than many people think. A random route across a city may expose many things, but an URBAN EARTH walk is special because it attempts to reveal what a city is like for the people who live in it. URBAN EARTH is not about following the tourist trail or tracking down the most extreme places… it is about finding normality.” Urban Earth
Thanks to Matt Ball for sharing.
It is really difficult to grasp the significance of lots of big numbers. It is even trickier when the numbers are organised in a hierarchy. For example, yesterday afternoon Bill English, the Minister of Finance, delivered his third budget, outlining the nation’s revenues and expenses. The budget includes details such as how the government plans to spend $21 billion dollars on social development in the coming year of which $9.5 billion will be spent on superannuation, almost $1.9 billion on the domestic purposes benefit, nearly $1.6 billion on accommodation assistance… and I’m already lost.
Treemapping is a technique for visually comparing groupings of numbers. A treemap represents a hierarchy of numeric values as a set of nested shapes – usually rectangles. I’m fond of this technique for several reasons, not least because it plays on existing associations. Big rectangles represent big numbers. Rectangles inside other rectangles indicate that one thing is part of another thing. Rectangles can even be shaded to depict an additional variable, for example, an increase or decrease over time.
Last night, with the assistance of Keith Ng, I created an interactive treemap of the New Zealand budget. It is available here. I encourage you to have a play with it. Unfortunately, given the large volume of data that needs to be processed and rendered, the visualisation struggles on every browser except Google Chrome. If you have Chrome on your computer, I highly recommend that you use it to view the visualisation.
There are two levels to the treemap. The top level shows the breakdown of various expenditure areas. The bigger the rectangle, the more money is being spent. Green rectangles indicate that spending has significantly increased since the last budget; red rectangles indicate a decrease. When you click on any spending area, the visualisation will dive into that area and show a detailed breakdown.
If you are interested in learning more about treemapping, I suggest you read Ben Shneiderman’s account of how he developed this class of visualisation and Thomas Kerwin’s survey of treemap techniques. Wikipedia has handy list of software if you want to create your own treemaps.
This video was made very quickly and could use some work. I post it here in case you find it interesting. I suggest you watch it full-screen, in high definition with scaling off.
The animation depicts two and half months of 2011 USGS earthquake data. Blue circles represent deep seismic activity recordings (>= 40km deep). Red circles represent shallow seismic events (<40km deep). Each event leaves behind a red dot to show the overall pattern. The animation ends the day after the 8.9 quake that hit Japan on March 11 and includes the shallow 6.3 Christchurch quake.
I am keen to spend some time improving this animation. Perhaps I will find some time in the coming weeks. I intend to release the programming code as an open source project – I would love to see the community build on this stuff.
A short post to draw your attention to some lovely work by Alexander Chen. Conductor: mta.me pulls subway schedule data from New York City’s Metropolitan Transportation Authority and transforms it into dynamic musical animation. The end result is wonderful.
Thanks to commenter Kate for sending me the link.
Here’s Chen’s description of the work:
“The piece follows some rules. Every minute, it checks for new trains launched from their end stations. The train then moves towards the end of the line, with its speed set by the schedule’s estimated trip duration. Some decisions were made for musical, aesthetic, and technical reasons, such as fading out routes over time, the gradual time acceleration, and limiting the number of concurrent trains. Also, I used the weekday schedule. Some of these limitations result in subtle variations, as different trains are chosen during each 24-hour loop.
The system has changed since 1972, and some lines no longer exist. For example, the 8 train, or the Third Ave El, was shut down in 1973. The former K train was merged into other routes. I decided to run these ghost trains between 12am-2am.”
I grew up in Auckland and spent many years using the city’s public transport system to get from place to place. While sitting on a bus, I sometimes wondered what the transportation network would look like if we could see the movements of the individual vehicles from the air. I would try to visualise the aggregate trajectories as each vehicle carved a path through space-time. After a few moments I would get hopelessly overwhelmed and go back to reading my book.
Last year Auckland Transport published its Google Transit Feed data on the MAXX website and I realised it provided the information I needed to make the map I used to daydream about. The data is a series of (large) spreadsheets, each describing a different aspect of the bus, train and ferry network. Last week I downloaded the spreadsheets and wrote some software to transform the data into an animated map.
As usual, I recommend you full screen the video and watch it in high definition.
The animation begins at 3am on a typical Monday morning. A pair of blue squiggles depict the Airport buses shuttling late night travellers between the Downtown Ferry Terminal and Auckland International. From 5am, a skeleton service of local buses begins making trips from the outer suburbs to the inner city and the first ferry departs for Waiheke Island. Over the next few hours the volume and frequency of vehicles steadily increases until we reach peak morning rush hour. By 8am the city’s major transportation corridors are clearly delineated by a stream of buses filled with commuters. After 9am the volume of vehicles drops a little and stays steady until the schools get out and the evening commute begins. The animation ends at midnight with just a few night buses moving passengers away from the central city.
Some things to note:
- The steady pulse of the Devonport Ferry.
- The speed at which buses hurtle down the Northern Motorway’s new bus lanes.
- The interplay between buses and ferries on Waiheke Island.
- The sheer mind-boggling complexity of the system.
Please note that there are a few quirks in the animation. A couple of ferry services pass across the land and there is at least one erroneous harbour crossing. These errors are not problems with the MAXX schedule – they are errors introduced by missing geometries. I will attempt to rectify these issues at a later date.