How accurately have Townlands in Northern Ireland been mapped?

From time-to-time newly released Open Data provides a nice opportunity to check OpenStreetMap for its accuracy in all its forms (see Hakaly (2008) for a breakdown of what this can mean).

Coastal Townlands, Counties Derry/Londonderry and Antrim.
Boundary lines see below. The deeper the colour of the area, the greater discrepancy in the area of the OSM polygon and the OSNI one. The pale base colour represents a divergence of under 2%. Townlands on the coast and on the UK/Ireland border seem to  be most likely to diverge in size. The small cluster centre right is caused by different ways of handling townlands which cross a Civil Parish boundary (OSM & the original source GSGS 3906 split these, the OSNI data does not).

We have known for a while that both the Ordnance Survey of Northern Ireland and the Ordnance Survey of Ireland were planning OpenData releases. When they came it was all in a rush. For now the hard work starts of checking license conditions for suitability for use in OSM and other places, as well as then working out what is really useful. However, because the townland boundaries of Northern Ireland are complete, it was an ideal opportunity to look at accuracy.

View along N side of MacGilligan Peninsula towards Inishown from Umbra

My reasons for doing this are not just pure interest. The usefulness of the Irish Vice County boundaries depends of their positional accuracy. Earlier my prediction was that such boundaries ought to be within 10 metres of their true location on the ground where they were based on townland boundaries, but this was largely based on experience with other OSM data rather than an objective statement. Thus investigating the accuracy using an independent data set provides an excellent way of testing this statement. The tests need to be done now, because (as we shall see) the nature of OSM is to fix issues spotted very quickly, and thus datasets become loosely coupled.

I adopted two approaches:

1. A straight comparison of areas (or their ratios).
2. Using a series of buffered boundaries from one source (OSNI) and seeing what proportion of the other source (OSM) was included in each buffer.

To choose which townlands to compare I followed a suggestion of Rory McCann and for each OSM townland selected the one which shared the most area in common from the OSNI data set. (I have also done it on matching names for a smaller set of data & get similar results). Note that I am comparing townland with townland, not boundary segment with boundary segment. This means that each boundary segment (other than coastal, lacustrine or riverine ones) will be included twice.

Buffering approach to investigating boundary accuracy.
Demonstrated with Umbra townland in County Derry/Londonderry.
This is predominantly coastal sand dunes, with a small river running along its S boundary.

Northern Ireland using the same colouring.
At this scale very few boundary mismatches are apparent.

The buffering approach is based on that described by Hakaly (2008). I used buffers of 5, 10, 15 and 20 m, and then clipped the initial OSM way be each in turn.  On the scale of the whole country it is clear that most boundaries match closely. This is confirmed by checking what proportion of the boundaries fall into each buffer class: over 80% are within 5m, over 90% within 10m and nearly 95% within 20m.

Closer inspection (as with the Umbra) shows much of the discrepancy to be present along the coast. This is not surprising, coastlines on OSM were originally derived automatically, and even when refined by hand are unlikely to accord with Mean High Water (MHW). Certainly, for my purposes, it is merely important that the OSM coastlines do not stray above MHW.

OSM townland Boundaries within 5m of OSNI data

The analysis described so far focusses on positional accuracy. Looking at areas highlights a range of other accuracy issues.

Area comparison. Townlands are coloured according to absolute variance of ratio of areas from 1.
The redder they are the further the ratio is from 1.

Area discrepancies of over, say 5%, may be the result of any of the following:

• Boundary discrepancy (such as coastlines). Mainly caused by coastlines, or difficulty of delineating some boundary feature, such as the course of the Umbra river above) 
• Erroneous interpretation of the boundary on old maps causing selection of the wrong feature. This transfers land from one townland to another, therefore these should cluster. 
• Missing townlands. When a single townland has been created without noticing one or more others inside it (Town Parks townland at Ballymoney is an example). 
• Different treatment of townlands bisected by a Civil Parish. See caption of first image above. Incorrect tagging. 
• Higher level administrative units having tags appropriate to a townland. I've noted two cases of this one of which was Ballyphilip CP on the Ards peninsula in County Down. 
• Islands. Some offshore islands appear to be missing from the OSNI data (see The Skerries N of Portrush)

We've already caught a few examples in each of these classes through this analysis, and no doubt will find a few more. I have not yet investigated the very apparent discrepancy along the borders.

To conclude, townland boundaries show exactly the kind of positional accuracy we expected (or perhaps hoped). Perhaps 1% of the total data (90-100 townlands from about 9000) may need some form of correction. I'm biased, but this seems pretty good, for a project principally relying on rectified photo-reduced maps from 1939! It's also worth remembering, that unlike road comparisons, there is no widely available sensor data (ie GPS tracks/point) to help boundary alignments.

When time permits I'll extend this to include OSI Open Data too. A big thanks to both organisations for releasing their Open Data. OSNI staff have been contributors to OSM for a while: they host Missing Maps lunchtime sessions in their offices.

How accurately have Townlands in Northern Ireland been mapped?

From time-to-time newly released Open Data provides a nice opportunity to check OpenStreetMap for its accuracy in all its forms (see Hakaly (2008) for a breakdown of what this can mean).

Coastal Townlands, Counties Derry/Londonderry and Antrim.
Boundary lines see below. The deeper the colour of the area, the greater discrepancy in the area of the OSM polygon and the OSNI one. The pale base colour represents a divergence of under 2%. Townlands on the coast and on the UK/Ireland border seem to  be most likely to diverge in size. The small cluster centre right is caused by different ways of handling townlands which cross a Civil Parish boundary (OSM & the original source GSGS 3906 split these, the OSNI data does not).

We have known for a while that both the Ordnance Survey of Northern Ireland and the Ordnance Survey of Ireland were planning OpenData releases. When they came it was all in a rush. For now the hard work starts of checking license conditions for suitability for use in OSM and other places, as well as then working out what is really useful. However, because the townland boundaries of Northern Ireland are complete, it was an ideal opportunity to look at accuracy.

View along N side of MacGilligan Peninsula towards Inishown from Umbra

My reasons for doing this are not just pure interest. The usefulness of the Irish Vice County boundaries depends of their positional accuracy. Earlier my prediction was that such boundaries ought to be within 10 metres of their true location on the ground where they were based on townland boundaries, but this was largely based on experience with other OSM data rather than an objective statement. Thus investigating the accuracy using an independent data set provides an excellent way of testing this statement. The tests need to be done now, because (as we shall see) the nature of OSM is to fix issues spotted very quickly, and thus datasets become loosely coupled.

I adopted two approaches:

1. A straight comparison of areas (or their ratios).
2. Using a series of buffered boundaries from one source (OSNI) and seeing what proportion of the other source (OSM) was included in each buffer.

To choose which townlands to compare I followed a suggestion of Rory McCann and for each OSM townland selected the one which shared the most area in common from the OSNI data set. (I have also done it on matching names for a smaller set of data & get similar results). Note that I am comparing townland with townland, not boundary segment with boundary segment. This means that each boundary segment (other than coastal, lacustrine or riverine ones) will be included twice.

Buffering approach to investigating boundary accuracy.
Demonstrated with Umbra townland in County Derry/Londonderry.
This is predominantly coastal sand dunes, with a small river running along its S boundary.

Northern Ireland using the same colouring.
At this scale very few boundary mismatches are apparent.

The buffering approach is based on that described by Hakaly (2008). I used buffers of 5, 10, 15 and 20 m, and then clipped the initial OSM way be each in turn.  On the scale of the whole country it is clear that most boundaries match closely. This is confirmed by checking what proportion of the boundaries fall into each buffer class: over 80% are within 5m, over 90% within 10m and nearly 95% within 20m.

Closer inspection (as with the Umbra) shows much of the discrepancy to be present along the coast. This is not surprising, coastlines on OSM were originally derived automatically, and even when refined by hand are unlikely to accord with Mean High Water (MHW). Certainly, for my purposes, it is merely important that the OSM coastlines do not stray above MHW.

OSM townland Boundaries within 5m of OSNI data

The analysis described so far focusses on positional accuracy. Looking at areas highlights a range of other accuracy issues.

Area comparison. Townlands are coloured according to absolute variance of ratio of areas from 1.
The redder they are the further the ratio is from 1.

Area discrepancies of over, say 5%, may be the result of any of the following:

• Boundary discrepancy (such as coastlines). Mainly caused by coastlines, or difficulty of delineating some boundary feature, such as the course of the Umbra river above) 
• Erroneous interpretation of the boundary on old maps causing selection of the wrong feature. This transfers land from one townland to another, therefore these should cluster. 
• Missing townlands. When a single townland has been created without noticing one or more others inside it (Town Parks townland at Ballymoney is an example). 
• Different treatment of townlands bisected by a Civil Parish. See caption of first image above. Incorrect tagging. 
• Higher level administrative units having tags appropriate to a townland. I've noted two cases of this one of which was Ballyphilip CP on the Ards peninsula in County Down. 
• Islands. Some offshore islands appear to be missing from the OSNI data (see The Skerries N of Portrush)

We've already caught a few examples in each of these classes through this analysis, and no doubt will find a few more. I have not yet investigated the very apparent discrepancy along the borders.

To conclude, townland boundaries show exactly the kind of positional accuracy we expected (or perhaps hoped). Perhaps 1% of the total data (90-100 townlands from about 9000) may need some form of correction. I'm biased, but this seems pretty good, for a project principally relying on rectified photo-reduced maps from 1939! It's also worth remembering, that unlike road comparisons, there is no widely available sensor data (ie GPS tracks/point) to help boundary alignments.

When time permits I'll extend this to include OSI Open Data too. A big thanks to both organisations for releasing their Open Data. OSNI staff have been contributors to OSM for a while: they host Missing Maps lunchtime sessions in their offices.

Urban Areas 4 : Derivation from OpenStreetMap using road density

Another variation on the theme from the last post: this time looking for some measure of road density.

Comparison of Urban Areas derived using "block method" and gridded road density.
Only grid squares with over 500 m of road included.
The area shown is around Butler, Butler Co, PA

The easiest way is to sum road lengths in individual cells. The cells have to be quite small (say 250 metre square) to achieve the resolution desired. I've excluded link roads and motorway & trunk highway classed in this calculation.

Urban Areas 3 : derivation from OSM using residential blocks

View S from the Cathedral of Learning in Oakland, Pittsburgh,
showing some urban areas used as tests in this post.

The incised valley of the Monogahela in the background contained railways and steel works. The plateau beyond has residential suburbs of Pittsburgh. To the left foreground are the woods and ravines of Schenley Park, with a residential area beyond. Source: Zack Weinberg via Wikimedia Commons CC-BY-SA

One of the obvious features of the highway network for the USA on OpenStreetMap is that road density is much higher in built-up areas. I started looking at how to measure this, when I recalled a method for identifying city blocks introduced to me by a Brazilian user of OpenStreetMap data.

Residential Areas for Butler Co, Pennsylvania, identified with the block method
from OpenStreetMap data. Orange line outlines Butler County.

My idea was simple, a greater road density implies smaller areas for the polygons enclosed by a set of roads. By choosing some maximum polygon size, one should be able to pick out urban areas.

The method itself is also really quite simple:

• Take the main road network for some area and make a union of it (which will be a MULTILINESTRING).
• Polygonize this data, and decompose to individual polygons.

In Lucas' implementation the first step is done by municipal areas. I wanted to try the approach for a whole state without using administrative area data. I therefore once again turned to my trusty standby of using a gridded method.

Urban Areas 2 : Derivation from OpenStreetMap using Residential Roads

Street corner, Retiro, Buenos Aires
(Libertad/ Juncal)
CC-BY-SA, the author

Following on from my last post I have now been looking in more detail at how one might start using OpenStreetMap (OSM) to create a global dataset of Urban Areas. As OSM does not have any widely used notation for urban areas I have been looking at several ways in which other OSM data can be used to identify such areas prospectively. In this post I look at the use of residential roads (and I'm not the first to do so). Later posts will look at other techniques.

Buenos Aires and hinterland, showing comparison between urban polygons
derived from OSM (green) and the Natural Earth data (light brown).

I have chosen the following places as suitable test areas for these investigations:

• East Midlands of England. Not only my home turf, but also a well-mapped area with extensive use of landuse tags, and in excess of 99% of all residential roads. In addition Ordnance Survey Meridian 2 Open Data contains a layer corresponding to urban areas which provides an excellent control for checking results from this area.
• Pakistan. Not only one of the most populous countries in the world, but one of the least well mapped in OpenStreetMap. Pakistan is a likely candidate for cities which are barely mapped. I would also expect other very populous Asian countries (notably China, India and Bangladesh) which are poorly mapped to be similar to Pakistan.
• Nigeria. Similar criteria to Pakistan: the most populous country in Africa. The .pbf file for Nigeria is approximately 50% larger than that for Pakistan, but both are smaller than that for Lesotho with a population of 2 million compared to 180 million (Nigeria) and 200 million (Pakistan).
• Côte d'Ivoire. Close to Nigeria, but a place which I know has an active OSM community. Quite a number of mapping activities. (Note to Geofabrik, it's not called the Ivory Coast any more).
• Argentina. Latin American cities are often laid out in a grid, nowhere more so than in Argentina. The prevalence of the grid system, and my believe that the urban road system is largely complete were reasons for choosing this as a Latin American example. My own experience of travelling in Argentina after SotM-14 suggests that, for the most part, urban road systems are mapped. One known gap, the newer western suburbs of Ushuaia has recently been rectified by the kind provision of aerial imagery from the Argentine National mapping agency.
• Pennsylvania. It was essential to include some US data  because of the TIGER import problem: all rural roads being tagged residential. Since I spent part of my childhood in Pennsylvania it is also a place I know and which I have edited (sporadically) to improve the rural road network.

Briefly I expected the following: good urban areas for the East Midlands and Argentina (i.e., better than Natural Earth (NE)); middling to poor for the three developing nations (gaps relative to NE, but in some cases better precision); hopeless for Pennsylvania.

Urban Areas: a meditation on why simple global geographical datasets are so poor

Puerto Vallarta, an aerial view of an urban area missing many roads on OpenStreetMap.
The area in the middle distance away from the sea was particularly lacking.
Fortunately the centre of the hurricane didn't pass over this area.
Source: Wikimedia Commons, (c) CC-BY-SA

The other night, as Hurricane 'Patricia' bore down on the Pacific coast of Mexico, I had a twitter conversation with Bill Morris and others regarding how well mapped Puerto Vallarta was on OpenStreetMap. (BTW: I'm sure it's much better mapped now).

Latin America is a weak spot for @openstreetmap. A town in the #Patricia track: https://t.co/pk4UYHHan3 pic.twitter.com/3t7ZQkWkWz

— Bill Morris (@vtcraghead) October 23, 2015

Of course, OSM is about fixing things, so I carried out the conversation in between adding around a hundred streets to the city. However the really interesting question was this one:

@vtcraghead @hotosm @openstreetmap Would there be some sort of algorithm that can be run to detect built up areas from imagery not mapped?

— Antonio Locandro (@antoniolocandro) October 23, 2015

Whilst at breakfast I thought a little more about this. I decided it ought to be possible to do something fairly simple with data which already exists.

Irish Vice Counties : the creation of a specific dataset on OpenStreetMap

I've been meaning to write about OpenStreetMap Ireland's townland mapping project for some time.

It's a wonderful example of how historical maps are of significant value in creating really useful data on OpenStreetMap which is just as relevant as it today as was in the past.

The immediate reason for writing about them is that I have been creating vice county boundaries for Ireland. In doing so I have not just been using the data, but the fantastic range of resources made available through the activities of the townlands project.

The Vice Counties of Ireland
My first complete draft of the boundaries on OpenStreetMap

Plantations : woods, forests or something else?

Poplars at Lakenheath
CC-BY-SA 2.0   © Copyright Alison Rawson and licensed for reuse under this Creative Commons Licence.

One type of woodland area I have alluded to a couple of times in the past are plantations (see here and here). I've always been frustrated at not having found good illustrations, but in the past couple of weeks I've noted a few which either already have good open images available or I've been able to snap a picture myself.

Plantations run the gamut from small areas to fully-fledged forests. In general what connects them is that the trees are planted in orderly rows, and the plantation has an expected lifetime, after which the trees will be harvested or replanted. Photographs enable some of the variety to be shown. In turn this should highlight the sorts of information we might want to capture by OpenStreetMap tags.

Why does The Guardian think that OpenStreetMap is owned by Google

My attention was drawn by a tweet to an article in The Guardian online showing some nice examples of cartography:

Some great #map and #cartography art here http://t.co/QesV8DLeGO

— Ordnance Survey (@OrdnanceSurvey) September 10, 2015

To my amazement it contained this:

To quote "Google's OpenStreetMap"!

Now, The Guardian has generally covered OpenStreetMap well over the years, and has made use of appropriately credited OSM data from time-to-time over the years. Last year it re-published Serge Wroclawski's influential blog post "Why the world needs OpenStreetMap". It's former Technology editor, Charles Arthur, was very familiar with OpenStreetMap, and in turn played a big part in the campaign to get open data from the Ordnance Survey.

This error is symptomatic of two things:

• A widespread assumption that anything to do with on-line maps must come from Google (most often seen in the belief that the images are taken with Google's own statellites).
• An absence of care in fact-checking when taking information from other webpages. 

There is not a great deal that the OpenStreetMap community can do about the former. After all commercial players such as DigitalGlobe have to put up with the same sort of thing. However, we can do something about the latter: but in general probably don't put enough effort into such things. (Perhaps unsurprising, its more fun to survey & add data than try and get people to get their facts right).

A quick google search reveals more or less the identical string in numerous webpages referring to Luis Dilger's 3-D city visualisations:

• Full Scream
• Design Collector credited to Arseny Verishin
• Trogue not credited
•  Fubiz not credited
• and many more

Clearly most of these are just straight copying from a single source. However it is not the original statement by Dilger (see link above). So it looks like The Guardian really has not checked it's facts and has the text is probably unoriginal too.

Thankfully I got a speedy response from them (thanks to twitter):

@SK53onOSM thanks for pointing that out - the caption's been amended now

— Gdn Readers' editor (@GdnReadersEd) September 10, 2015

All-in-all the episode highlights that as a group we in OpenStreetMap have a long way to go in communicating who we are and what we do to mainstream media types.

Shops in Coalville

I had not planned to do much mapping on the Bank Holiday Sunday: the day was set aside for a meeting of the British Plant Gall Society at Ryton Woods, Warwickshire.

Oak with Hazel coppice stools, Ryton Wood.
The ground layer changes here with bramble (in foreground) absent deeper into the wood.
Just to the left of the foreground hazel a bank & ditch can just be discerned. This is probably the remnants of a woodland compartment dating back to the Middle Ages.

The only thing I expected to map were paths in the wood, which surprisingly are almost all unmapped (hint to Mappa Mercia folks). This is one of the best areas of ancient woodland in the county and only a short distance from Coventry and Warwick. In fact, if anything, I would have expected to write about this wood which is mainly Oak with Hazel coppice.

Coalville : Hotel Street in 1988.
The Railway Hotel is now a day nursery, and the buildings on the left beyond have recently been demolished.
The Railway was one of a cluster of pubs adjacent to the level crossing and station: the others continue as pubs.
Source Ben Brookshank, Geograph via Wikimedia Commons

An Early Map Grid

In my post on Woodland Cartography I noted with regret that I had not consulted the Shorter Science and Civilisation in China regarding cartography.

I've now read the relevant chapter, and was rather disappointed. Although there was a fair amount of information about surveying, there was very little about cadastral surveys. Elsewhere in the world they seemed to be a major driver for collecting information about trees and woodland in the past. (Subsequently, I've also had a peek in the relevant complete volume of Science & Civilisation in China, and it suffers from the same deficiency).

What was interesting was a map from the 12th century which shows China with a rectangular grid overlay.

The Yǔjī tú (禹迹图 following the footsteps of Yu).
A detailed map of China carved in the Song dynasty on a stele now in the Stele Forest Museum, Xi'an. The original image in the Library of Congress is a scan of a rubbing of the stele. This version has been converted to monochrome, inverted and normalized to enhance contrast.

My interest was piqued, because I had not given over much thought to the historical development of gridded map systems: other than to assume most kilometric grids originated from military needs. In fact a recent discussion revealed that the Irish Grid was created around the time of the Second World War, with an antecedent in the grid used on the GSGS sheets of out-of-copyright maps shown on the Irish OSM server.

One early and prominent grid is the township grid of 6 by 6 mile squares which overlays much of the Western USA. This was created in the early days of the Republic, but like many such cadastral systems was more a series of grids with different origins than a single grid (see the wikipedia article for detail). I've made a crude attempt to map one such township in Oregon, close to the ranch founded by my great-great uncle.

Similarly in Ireland, the initial survey for the 6 inch maps produced sheets on county lines; as did the equivalent, but much later survey in Great Britain. Probably the same happened in other places: quite local grids initially, with consolidation occurring later.

Coincidentally Mike Dobson discusses location grids in a recent post about what3words. His discussion not only places their use in a modern context, but also has more information on their origins.

Retail Outlets on OpenStreetMap: Cartograms, and Patchwork Quilts

I enjoyed the process of creating a cartogram from OpenStreetMap data a couple of years back, even if it was somewhat tedious. However two things stopped me from taking it further: the QGIS plugin I was using does not work with later editions, and I really wanted something a little more refined.

Cartogram of Local Authority areas in Great Britain based on numbers of pubs on OpenStreetMap
Created using ScapeToad, this is a simple, and naive, cartogram.

Documenting Footpaths with Mapillary

I have long been a believer in the need to document OpenStreetMap survey data as thoroughly as possible.

I have a large archive of audio files, GPS traces, and tens of thousands of photographs. These span back to late 2008 when I started contributing to OSM. From time to time these prove useful, for instance, I had very precise documentation for my evidence at a Public Enquiry.

However, sharing such archival information with other mappers is difficult. It's not even straightforward for me to locate stuff. I have used OpenStreetView (OSV) since it was announced at SotM Girona. It is difficult to share photos using OSV, and the interface has not developed since 2010.



I was therefore very interested to learn about Mapillary, but was initially put off by the licensing. When they changed the licensing I was more interested. At SotM-EU Karlsruhe I was able to chat with Yubin after hearing his talk, which convinced me to give it a go. As I've said before, I regret I did not do this the following morning when full documentation of our walk at the Weingartnermoor would have been very useful, not just for mapping this particular place, but for discussing how to map woodland.

I don't have an Android phone which is compatible with Mapillary so I have had to do things manually. This is a little tedious, so I tend to keep the creation of sequences for things which are either simple or of particular value.

Interviewed by OpenCageData

I was recently interviewed by Ed Freyfogle of OpenCageData.

Ed asked some questions about this blog which I had to think about a bit. I'm not sure if I've explained myself very well, but, in case you missed it, the interview is here.

At some stage when I've cogitated on these answers even more I might expand them directly on the blog.

Bat Bridges, or why deleting lonely tags is a bad idea

The other day I was idly browsing the blog of Mark Avery, the former conservation director of the British bird protection society, the RSPB. One item caught my attention: it was about 'bat bridges'. Although I hadn't heard of them before it was pretty obvious what they might be.

A bat bridge on the A590 in Cumbria

Bats tend to follow linear features in the landscape when foraging at night, at least in part because they provide protection from predators. Bats tend to avoid flying over open spaces. Hedgerows, edges of woods, and so on, form commuting routes between roosting and feeding sites for bats. When these are damaged or destroyed, for instance by road building, bats either lose feeding locations or have to cross the open space. Usually they do this by flying low: effective against their age-old predators, but not much help when confronted by a car.

New Year footpath mapping with Mappa Mercia near the Fauld Crater

My first countryside excursion of 2014 was to investigate a man-made hole. For 2015 I choose a different bigger hole which I've meant to visit for a long time: the Fauld Crater. What was different this year is that we made it an OpenStreetMap mapping and social event!

Mapping footpaths for OpenStreetMap near the Fauld Crater, East Staffordshire

I'd mentioned at our last pub meeting of the year that I fancied doing some footpath mapping between Christmas & the New Year. Coincidentally Rob Nickerson of Mappa Mercia asked if we were organising anything after Christmas. So the idea of 2 or 3 of us getting together grew to the notion of linking up with Mappa Mercia. So in the end the meeting had quite a diverse set of goals:

• Do some mapping together
• Walk and map unmapped footpaths
• An excuse for a post-Christmas walk
• Link up socially with Mappa Mercia
• Initiate another type of OSM activity in the (East) Midlands