Thursday, 18 September 2014

OpenStreetMap at the UK Open Addresses Sympoosium

I attended the Open Addresses Symposium organised by Jeni Tennison of the Open Data Institute last month. This brought together a host of people and organisations interested in having an open alternative to the Postcode Address File (PAF).

Somewhat foolishly I'd suggested to Harry Wood that I might speak about addresses on OpenStreetMap.

Addresses mapped on OpenStreetMap in Britain
Density of address mapping in (southern) Britain on OpenStreetMap by local authority
(Northern Scotland not shown because little data, full map)
See text on map for full explanation.
I was glad to see that my talk was relatively late on in the day: the audience were unfamiliar and many of them came from large organisations., so I appreciated the chance to get an impression of them.

Later on I began to think this was a bit of a poisoned chalice. I was scheduled immediately after Bob Barr. Anyone who has heard Bob speak (sadly his great talk at SotM-13 was not successfully videoed) knows that he's a very accomplished and passionate speaker. A hard act to follow! Making sure that I managed to keep the momentum up after Bob's talk meant that I didn't pay a huge amount of attention to the talks immediately preceding mine.

You can check out my slides below or download them from Slideshare.

The summary message was:

In particular I wanted to make sure that the audience understood that a lot of OpenStreetMap data is created by a small number of people, often people who have become highly skilled at collecting the data they do. To that end I created the map at the head of the blog.

This is mainly intended to make at polemic point. Not every significant contributor to the bigger bubbles is named (at least in part because I don't know them all). This should be apparent by the names I chose to represent London (Tom, Derick and Harry). The key point is that it's not much of a crowd when I can more or less name the contributor directly.

The same phenomenon occurs with other 'crowd-sourced' data sets. If I look at a map of records for any species of fly (Diptera) in Britain there will always be a nice concentration of records round Sheffield, which will mainly be contributed by Derek Whiteley (who does mammals too). The Welsh Borders have an amazing number of rarely recorded microfungi: Bruce Ing lives in the vicinity. Many of these 'crowd-sourced' data sets do not really display what they purport to: they are most usually maps showing where the enthusiasts live. I'm afraid that addresses are the same (at least in the UK) on OpenStreetMap, and using 'crowd-sourced' as a description of the process is rather misleading.

There were lots of other good things at the symposium, not least Jeni Tennison's clever wrap-ups of each session which suggested an almost clairvoyant anticipation of what speakers were going to say. The two messages which I thought most important came in the morning:
  • Addresses as objects, not as attributes. This came in Morten Lind's talk about the Danish address register (see his slides at SotM-FR this year here). I've actually come across this long ago in Data Warehouse schema design (for instance in a very old version of IBM's Insurance Industry Architecture). Insurance has good reasons to treat addresses separately, because addresses are associated with policies, policy holders, risks, claims, and so on. Once addresses are treated as objects, not attributes, the meme about 80% of organisational information being geo-related can be discarded.
  • Open Addresses cannot just be a PAF replacement. I was pleased to find that lots of people were in agreement with me that the Royal Mail's view of what constitutes an address is far too limited for many use-cases. In the first instance a PAF-lookalike might be the aim, but it's clear that more is needed if one wants to respond to the sense of place which ordinary people use on a daily basis. (We can be sure that the folk of Kinlochbervie don't think they live "by Lairg")
In the pub I also discovered I had a lot to learn about the British Standard for addressing.

Initiatives elsewhere in the OSM community, notably BANO and, already suggest the way forward for OSM and open addresses. We need to collate and make available any open address data outside of the OSM database per se: not least because ODbL may be too restrictive a license for many use-cases, but also because a lot of the data will not be really suitable for direct incorporation into OSM.

In conclusion: I see OpenStreetMap as a likely heavy consumer of Open Address databases rather than a major contributor to them; I see the OpenStreetMap community as a significant facilitator in the creation, maintenance and management of such open data.

Tuesday, 2 September 2014

Woodland Cartography

This is an expanded version of my talk at sotm-eu:

I start by seeking inspiration from the many ways in which woods and trees have been shown on maps in the past, and then consider what elements we may want for OSM data, and how we might depict such elements.

Monday, 1 September 2014

Contributing to the Lesotho Mapathon

At the start of August I appeared in the OpenStreetMap stats for users adding most data in a day. This was the first time in ages that I've made enough edits to appear. The reason: I've been contributing this past week to a mapathon to map as much of Lesotho as possible. This has been co-ordinated by Irish OSM contributors, some of whom will travel to Lesotho early next year.

View from Lesotho village (5297237744)
A village in Mokhotlong District.
This is S of the area I have mapped, but looks similar on aerial photos.
Source: Wikimedia Commons.
The co-ordination makes use of the HOT Task Manager: a piece of software which has distant origins in something, long gone, called QualityStreetMap.

I've use the Task Manager fairly rarely, but development over the past year has added one feature which for makes it much easier to use: the creation of a bounding box in the JOSM editor. It is now much simpler to see the area one has undertaken to map. This in turn is important in reducing editing conflicts and redundant work.

In practice after completing 4 tasks, and realising I was unlikely to clash with other edits I started roving more widely. The reasons behind this were four fold:
  • Mapping lots of a single feature can get very boring. It helps to maintain interest by varying the range of tasks. As I'd started in a very mountainous area I was more-or-less only mapping cliffs and streams.
  • A basic context for mapping can often help a great deal in teasing out further detail from aerial photos: particularly for places where one has no on-the-ground experience.
  •  I was not picking up any 'culture' (human habitations, roads, tracks) etc. in my initial squares and although natural features are fine this was not the first priority of the mapathon.
  • My usual editor is Potlatch and I was having problems picking out some features from the aerial imagery in JOSM
What I did was follow a river down from the border ridge until it met another river. I then mapped villages and hamlets within the watershed using landuse=residential. For the last village along an obvious road I also mapped the distinct areas of farmland and individual buildings, most of which are round huts (according to wikipedia called mokhoro in SeSotho), and thus very, very easy to map in JOSM using the duplicate feature. I deliberately left paths and tracks because I was very unsure what their status was. I felt that as I mapped more detail I'd probably get a better feel for this over time.

Lesotho Watershed overview
Overview of the main area in which I mapped.
Mainly OSM data accessed via Overpass-Turbo
with contours from SRTM and hillshading from ASTER.
copyright OpenStreetMap contributors CC-BY-SA

Producing a simple overall visualisation of the area (such as that above was really useful for assessing how much of the area I had mapped. Using aerial imagery meant that I was tending to work at high zoom levels (z16 and above) and this often meant that it was quite difficult to be aware of the how a given edit area fitted into the overall context. It struck me that a small context window in OSM editing tools might help in this type of situation.

I've tried to add as much detail as is feasible, but the various visualisations still look rather bare. I think that what is missing and would give the map more life are the names of villages, hamlets, streams, and rivers. I did check the US Military map, but am not at all sure whether the names on this are current: they certainly don't appear to correspond to anything on the list of places from the 2011 Lesotho census.

View Larger Map

Mapped Features

With this sort of aerial mapping one can only work with a fairly small set of features to map. The ones I used frequently were:
  • waterways: I mainly mapped these as streams. Many look to be intermittent, although that is very difficult to judge from aerial imagery and some clearly supported a different vegetation even if not apparently flowing.
  • residential landuse: most villages and hamlets were nucleated. Only one place had buildings spread over a more diffuse area. This made adding residential landuse pretty easy. In turn this made finding places to map buildings easier later on.
  • farmland landuse: the places I mapped as landuse=farmland are presumably entirely given over to growing crops. Places used for pasturage, either around villages or in the mountains are not clear. In the highland area that I was mapping areas of cropland stand out quite clearly, and are often highly discrete: presumably reflecting details of the local topography (soils, insolation, water availability and so on).
  • places: I categorised settlements either as villages (roughly those with over 100 buildings), hamlets (usually around 25 (10-50) buildings) and localities (1-3 buildings). The latter I presume to mainly be huts occupied by herdsmen for some time during the summer months. These were nearly always accompanied by 2 or 3 walled enclosures, which I assume are used for cattle.
  • walls: the only walls I mapped were those associated with the highland isolated localities. In most villages there were also quite a few walls, some may represent old buildings. I chose not to map any of these.
  • buildings: the bulk of buildings were small round structures which I interpret to be mokhoro. The rest were mainly rectangular buildings which were somewhat larger than the traditional huts. After I mapped the basic buildings I realised that both types lend themselves readily to some simple 3Dshapes tags, so I added these too.
  • roads and paths: I ended up using relatively few tag values for highway. Most things I tagged with highway=path, with the intention of implying its most basic meaning: a path which might be used by foot or horse or bicycle or motorbike. I spent quite a bit of time before choosing this over track, but careful scrutiny failed to reveal many routes with evidence of passage of 4-wheeled vehicles. Those that did clearly fulfilled more important purposes than those of a track, so I accorded them unclassified or tertiary road status. Other mappers may well have used track for both things I've tagged as a path and for things I've tagged as a road.
  • cliffs: because the Drakensberg is made up from multiple layered basalt lava flows it has lots of cliffs! Choosing which to map was slightly arbitrary, as some are little more than a surface rock exposure. In many cases I was guided by the size of shadows in choosing what to map.
  • ridgelines: initially I mapped these to help delineate the area I wanted to map. When I discovered that OpenMapSurfer renders them I added more and refined the ones I had already mapped.
Interpreting aerial photos can be greatly assisted by geolocated images taken on the ground. Even the sparse number on Wikimedia Commons were of help. Photos taken locally (whether by local OSM contributors, in country AID workers, or folk visiting such as the Irish mappers who kicked this activity off) can be a really valuable aid. Whilst writing this up I came across what OSM contributors are doing in Niger. It would be great if we could turn things about so that those of us mapping remotely were really not much more than mechanical turks doing the grunt work for local mappers who already know lots of the things which often have to be guessed at.

Using the HOT Tasking Manager

I'm a very occasional user of the tasking manager, the last time I used it was in validating a square after Typhoon Haiyan/Yolanda. These are a few observations about my experience of its use:

Rectangular Task Grid: The use of existing square tiles has many advantages, not least that an grid can be laid down automatically in the aftermath of a disaster, and it lends itself to automatic subdivision.

However, I personally would find the ability to define arbitrary polygons (aka cake slices) for tasks very helpful. In validating mapping of buildings in the Philippines I found I was using roads to delineate search areas within my square. A scanning search strategy (e.g, from W to E and N to S) is often quite difficult to pursue steadily for validation because it is often important to frequently zoom in and out, and it is quite easy to lose ones place in the scan. When mapping linear features such as roads will also tend to pull the mapper away from a systematic examination of the imagery.

Completion: Completion of a square must be considered against the given task. For the Lesotho Mapathon the task was quite diffuse: "map all settlements, roads and major natural features, particularly rivers & streams within your allotted area". Even in squares which I marked as completed I have subsequently added quite a bit of data: most significantly isolated dwellings and paths connecting them. These were not obvious at the zoom levels used for initial mapping (z15-z16) and mainly were picked up by detecting paths at high zoom levels and following them. In general nothing mapped on OSM is ever done, completeness is only approached asymptotically. It should always be possible to find new things to map or refine existing data: even in well mapped places.

Patchiness: grid (or polygon) based tasking favours a depth-first not breadth-first mapping approach. When available mappers outweigh tasks this isn't a problem, and the tasking avoids duplication of work and edit conflicts. However, in many HOT tasks a surplus of mappers is not the issue (at least in part because new tasks are arriving all the time).

Two images show this for Lesotho:

Butha-Buthe, a town in Lesotho, showing severe mapping disparities between task grid squares.
Distribution of places tagged as villages in Lesotho
and surrounding parts of S Africa.
Captured 3 August 2014 using Overpass-Turbo.
Just creating the map above, prompted me to break off from writing this piece and go off and try and quickly add more villages, as is shown below.
An additional 52 village nodes added whilst writing up this point.
Note: some had been partially mapped as residential landuse.
If most villages had already been present, and (another big if) non-rectangular grids were possible, use of Voronoi triangulation might produce work units which were more consistent.

Now I suspect that during the development of Tasking Manager many of these things were thought about, but got dropped on the KISS principle!

A little bit about the data

One of the great things with OSM data is that once its been created one can start using it to ask questions. To assist in this I downloaded SRTM and ASTER data to create a set of contours for Lesotho. In principle one can then ask questions like:
  • What height are villages & hamlets in the area mapped?
  • What is the maximum elevation of farmland?
  • What is the relationship between cultivated areas and numbers of dwellings?
  • Most settlements appear to be fairly nucleated. Is this true?
A tad more processing might be needed to ask really sophisticated questions such as:
  • What is the average level of insolation of farm and non-farm areas close to villages?
Sadly I ran out of time to work on answering these questions.

There are others which relate to what we have mapped:
  • When buildings have thatched roofs: what plants are used for the thatch? what timber is used for the roof frame? Can we map the location of these resources?


I just mapped one tiny corner of Lesotho, and am perfectly aware that there's still more to map in that area. Lesotho Mapathon activities continue. If you fancy mapping something a little bit different, or even starting out mapping, there are plenty of opportunities in the coming weeks.

For myself I'm likely to contribute mainly by continuing trying to identify settlements so that we get a broadly based map of the country as well as fantastic detail in places.

Friday, 22 August 2014

WWII Bombs in Nottingham : discovering local history whilst mapping

Last Saturday I showed 2 visitors how I mapped addresses (more on this later).

Infill housing, St Cuthbert's Road, Nottingham NG3

One little vignette stood out.

Sunday, 10 August 2014

Perhaps I was too cryptic

maprebus_2 maprebus_3 maprebus_1 maprebus_4

Each of the above maps represents objects mapped on OpenStreetMap which have a tag which corresponds to a single word. Together they spell out a topical phrase:
  • Happy. Actually name=Happy's as for some reason I had not spotted lots of name=Happy in Europe.
  • 10th. A minor fudge, using tiger:name_base=10th and restricting it to Pennsylvania to reduce the amount of data returned.
  • Birthday. A couple of retail places in Japan have name=Birthday.
  • OSM, A real fudge, I used source:name=OSM (an interesting recursive defintition in itself) 
Happy 10th Birthday OSM!

Sunday, 13 July 2014

Upland woods in the Schwarzwald

I wasn't done with looking at woodlands from a mapping perspective in Baden-Wurttemburg. A couple of days after SotM I walked from the top of the gondola at Feldberg to the Feldberg summit and then back to Hinterzarten.

The first part of the walk was somewhat marred by a fierce hailstorm which left me fairly damp. However I recuperated by drying out a bit over Kaffee and Kuchen at the Baldenweger Huette below the summit.

Forest path, part of the Emil-Thoma-Weg.

Thursday, 3 July 2014

Weingartnermoor Woodland Walk : SotM-EU Mapping Workshop

At State of the Map Europe in Karlsruhe I used the opportunity to develop some of the themes I have already outlined for woodland mapping. Essentially I have three lines of attack:
  • extending the cartography of woodlands in OSM ;
  • finding richer ways of tagging woodlands in OSM ;
  • looking at how we can collect data about woodlands (mapping).
I didn't do anything about the second, but gave a lightning talk outlining just some ideas about woodland cartography (I got a few more over the course of the conference). For the latter I thought the best approach would be to get some OSMers in some real woods because real things are much easier to discuss than abstract ideas on the wiki.

We spend much of the conference listening and discussing abstract ideas, and (too) little time using the fact that we come from many countries to share our knowledge of tagging. (A little Guerilla Mapping is not out of place too). So this was a small innovation for an SotM conference too.

SotM-EU Woodland Workshop Participants
Participants on Woodland Mapping Workshop, Karlsruhe June 2014
(minus the author who can be seen in this photo)
I was fortunate in the first place that the Karlsruhe Stammtisch offered some good ideas and advice and the conference organisers chose to add the event to the programme. Secondly, I was fortunate in being supported by more OSMers than I expected: and I know I lost a few due to the early start (necessary to enable not missing the whole Hackday).