PfAFing About : opening UK address data

The Postcode Address File doesn't work all the time.

I recently mentioned that the Royal Mail's Postcode Address File (PAF) had elicited a lot of recent controversy. Well the controversy continues as the UK government has announced that the PAF business will be privatised along with the Royal Mail despite protests from some well-known people:

RT @drbobbarr: URGENT If you don't want to be charged 4 UK addresses and postcodes forever don't let PAF monopoly be privatised. #OpenPAFnow
— Tim Berners-Lee (@timberners_lee) March 21, 2013

Postcode address file (PAF) to remain as state monopoly bit.ly/11naOao. The value destruction of PAF starts here...
— Jonathan Raper (@MadProf) April 22, 2013

Everyone's postcodes to be privatised in Royal Mail flotation, despite objections from Sir Tim Berners-Lee: Telegraph soa.li/Gt75z0d
— Robert Barr (@DrBobBarr) April 19, 2013

Naturally this has led to discussion on twitter as to whether one can create a fully open alternative to PAF.

The rest of this post is probably mainly of interest to mappers and OpenData advocates in the UK. To the outside world UK OpenStreetMappers seem to be oddly obsessed with postcodes. I hope some of the reasons for this may become apparent from this post.

Completeness of post box mapping in Britain

I'm always on the look-out for ways to visualise the degree of completeness of OpenStreetMap data. Although it's often possible to perform quantitative analysis, such as the work done by Peter Reed on supermarkets in the UK, it's usually a lot harder to show the geographic element, even when a comparative data set is available.

Thinking about this last night I realised that Geolytix's Postcode Sector open data would allow comparison between the list of post boxes released by the Royal Mail in 2009 and OpenStreetMap data. Although the Royal Mail data is not geocoded it does contain a reference for each postbox which includes the postal district  in which the box is located. OSM data can be directly matched to postal districts with a point within polygon query. The result is here:

Percentage of Royal Mail postboxes mapped on OSM by Postal District

Note that the Royal Mail data is available as the result of a Freedom of Information request: copyright remains with the Royal Mail. I merely counted the number of rows in the tab-separated file for each postal district.

Producing the image took me way longer than I hoped. Reasons included bugs in QGIS regarding handling smallint and int columns from Postgres, a weird bug where the QGIS Print Composer refused to do anything sensible with a scale of 1:4,000,000, grappling with the new label formatter in QGIS, query performance; and adding attribution statements. In the end I did all the analytic processing in PostGIS and the presentation in QGIS. Accordingly I'm more interested than ever in the ideas of the guys at Mapsdata who I met a couple of weeks ago at a London OpenStreetMap gathering at the Blue Posts.

I hope the data speak for themselves. Very incomplete mapping in Wales, the South-West Peninsula, and Lincolnshire are no surprise, but I didn't expect the pretty good data in most areas of the Scottish Highlands. I suspect that much of this may down to the activities of a single mapper. A number of areas have more postboxes than recorded by Royal Mail in 2009: this may be redundant entries in OSM, single entries for post-box pairs in the Royal Mail data, or changes in the number of postboxes between 2009 and 2013.

Even in areas we know to be fairly well covered by on-the-ground mapping, coverage is patchy. I can only conclude that some mappers DO NOT MAP POSTBOXES.