Augmenting Residential Landuse : building density

Andy Allan (gravitystorm) asked a very pertinent question after my talk at SotMEU: Why had I not used building outlines to automatically identify different classes of residential landuse?

Briefly, my answer was: "I had tried, but the results weren't satisfactory". This is a much longer answer than I could give at Vienna, and fills in some detail about the technical approach I used. I hope that readers may have some good suggestions to improve this approach.

It is clear that Urban Atlas data is a good model for studying landuse mapping through replication/simulation in OSM. After working with this data for six months I realise that one of the main reasons for this is that UA's main classification is simpler than the current OSM consensus for landuse mapping. Thus most UA landuse classes have a 1:M mapping to OSM tags. Other landuse and landcover schemes, such as CORINE, EUNIS, have many more classes and in thus in many cases the classes have N:1 or N:M mappings to OSM tags. Most of these relate to agriculture, woodland or natural vegetation: for the most part where OSM tag usage is only fitfully consistent, or the same tag has multiple meanings.

There are, however, 5 UA classes (Continuous and Discontinuous Urban Cover — 11100, 11210,11220, 11230 and 11240) which map on to a single tag in OSM : landuse=residential. The main criterion for separating this classes in UA is the degree of surface sealing, with continuous urban cover meaning that over 80% of the land surface is effectively sealed (buildings, asphalt, concrete etc.). As a significant proportion of surface sealing will be buildings it seems plausible to use the proportion of residential landuse covered by buildings as a proxy for surface sealing.

Buildings generated from OS OpenData using mapseg (orange) compared with those mapped in OSM using Bing imagery (light green) for OS grid square SK5038.

Firstly, I needed a decent source for building outlines. My test data area, Nottingham, does have a large number of mapped buildings (47000+), but is nothing like complete. Partially mapped building sets are no use at all for this purpose. I therefore made use of Mapseg a set of python routines written by Tim Sheeman-Chase to extract building outlines from the Ordnance Survey OpenData StreetView (OSSV) tiles. I had originally created this data to make address mapping easier, but was dis-satisfied with artefacts (spiky buildings, failed orthogonalisation, etc.) so I had only used a tiny proportion of the data directly in OSM. My original data set covered the whole of the county of Nottingham and adjacent areas and included over 600000 building outlines: processing and manipulating this was somewhat tedious as each OSSV tile or sub-tile generated a single OSM XML file and I therefore had around 200 files with duplicate node and way identifiers. In the end I loaded each file into PostGIS and merged the data there. I then clipped it to the bounds of the target test area to end up with a data set of 189, 420 buildings. Some geometries needed to be cleaned to remove self-intersections.

Buildings were then clipped to match the residential landuse parcels, and then assigned to individual parcels. Landuse parcels which had originally been created as multipolygons on a 1 km grid were decomposed to individual polygons (using ST_DUMP) before this step. Polygons meeting on grid boundaries were not re-assembled. At this point it is simple to sum the building area for each polygon and determine the ratio of the total polygon area covered by buildings. The results are graphed below with the data divided into 20 buckets.

For comparison here is the relative distribution of the EEA UA data for the Nottingham area:

UA Code	Area (ha)	% Total Residential
11100	137	1.0%
11210	7132	52.0%
11220	5298	38.7%
11230	846	6.2%
11240	18	0.1%
11300	272	2.0%

The vast majority of residential landuse falls in two classes with surface sealing in the range 30 to 80%, with more than half of the landuse with greater than 50% surface sealing. A cursory comparison between the two sets of figures suggests that, at best building cover may only account for half factors used to assign surface sealing in Urban Atlas. Fortunately because I created a datasets of differences and similarities of data from UA and OSM it's possible to compare individual landuse polygons: a bit more work as these may be slightly different shapes, and therefore the whole clipping of buildings and area calculation must be repeated.

The box-plot above amply confirms the initial impression. No doubt if and when I learn more about R, a proper statistical comparison could be done. Building density from OSM does not explain enough of the UA surface sealing value to be reliable for classification. At least for now. There is enough in the plot to hint that enrichment of the data with other variables may facilitate separation, but that's a different project again.

So that's the (very) long answer to Andy's question.

What does it mean for OSM:

• Using buildings to derive sub-classes of residential landuse is not generally useful.
  
• The requirement for completeness in building outlines on its own means that such an approach would at the best of times only work in certain places (good external sources of building data, excellent imagery). It may also require fanatical devotion to mapping buildings.
• The basic classes of landuse we are interested can probably be derived from local knowledge, ground survey, aerial imagery etc. To make this more generally useful this requires some guidelines and consensus on tagging.
  
• For now we could use tags similar to the Urban Atlas codes for urban areas in Europe and North America. Significant extension of these values might be needed to adequately cope with other areas in the world. (I'll be blogging in depth on this subject soon).
• Looking at extending this type of data with other OSM derived variables may be interesting. Alexander Zipf mentioned that his group at Heidelberg had used a data mining approach to identify residential areas from OSM data.

If you've got any ideas of how to extend this type of classification drop me an email or comment below.

An Exploration of Bad Polygons

I'd hoped to get further with my Urban Atlas simulation, but have been distracted by badly formed polygons. In using osm2pgsql, I relied on it for conversion of OSM data to polygons in PostGIS. The polygons get created and load fine, but once I started clipping sets of data I noticed that I was losing some landuse polygons. Specifically I noticed that the residential landuse for the two large villages (or small towns) of Bingham and Radcliffe-on-Trent were missing. The image above shows the Harlequin area of Radcliffe with residential landcover (red), but the polygon for the rest of the village has gone.

I was not at all sure where the problem lay. I reimported the data with a different version of osm2pgsql; I used an older data set; I even rendered the area using a modified version of the OSM mapnik stylesheet (see image to right with the missing polygons highlighted) . In all cases the Bingham and Radcliffe polygons could be retrieved and displayed in QGIS but disappeared on clipping. They were successfully rendered by mapnik. When I tried to perform the clipping in PostGIS the error messages were much more explicit. A bit of delving in the PostGIS manual led me to the ST_IsValid and ST_IsValidReason functions. Even better a quick search found a nifty function called CleanGeometry (link to code here) which I have now installed in my template OSM database on Postgres. Running this on the landuse polygons got rid of the intersections, so problem solved.

Only partially. It's really much better to find the problem at source and resolve it there. I had tried the JOSM validator on the data but it did not report any problems, so I was still uncertain if it was a hidden bug in osm2pgsql or a data problem.

I left the issue for a couple of days, until, whilst checking some address data, I noticed Geofabrik's OSM Inspector had a set of Geometry validation tools. I'd never found a use for these in the past. Of course, Jochen Topf and Frederik Ramm thought about this sort of problem long ago and I could instantly see the exact location of the problems, and even click on an icon to start-up Potlatch in the right location. Just another illustration of the rich endowment of the OSM ecosystem.


Great Britain has around 1500 badly formed polygons (based on data from Jan 22) or about 0.2% of the total data. Of these about 80% are self-intersecting and the rest are mainly self-intersecting rings. Many are buildings (400 or so, as seen in the screen-shot above), with the rest more or less evenly distributed between landuse, woodland (natural=wood) and water (natural=water). Overall the error rate is extraordinarily low given that most OSM contributors are, like me, probably don't do formal checks on the geometry of their data.

So I can go back to my simulation, having once more found that OSM provides the tools I need.

Simulating Urban Atlas using OSM

Urban Atlas simulated with OSM data

A few days ago the EEA Urban Atlas data came to my notice (thanks mackerski). I blogged my preliminary impressions, noting that it should be possible to use OSM as a starting baseline for such data. Today I show that this is not only possible, but straightforward and quick.

The main picture shows OSM data for the city of Nottingham and surrounding district councils of Ashfield, Broxtowe, Erewash, Gedling and Rushcliffe (area UK029L in Urban Atlas). Data have been plotted for each land cover class in Urban Atlas, except that all residential areas have been plotted as class 11220.

The basic methodology was as follows:

• Download OSM extract for England from Geofabrik.


• Extract a bounding box using osmosis


• Load the bounding box into PostGIS using osm2pqsql


• Extract a shape file for each landcover class using PostGIS


• Layer shape files and change symbology in PostGIS

Output was generated so as to resemble the PDF file distributed by the Urban Atlas project for the Nottingham area (included in this zip file). A legend was not generated for technical reasons concerning QGIS.

A few remarks about the processing:

• Roads and railways were buffered to generate an approximate landcover with buffers of 30m for motorways, 15m for primary and trunk roads, 10m for railways, 7.5m for secondary roads, 5 m for tertiary and 3.75m for others.
• Residential areas were not broken down as comprehensive building outlines are not available in OSM. Furthermore, residential areas would require breaking down using intersection operations before this could be done.
• Nature reserves should have been processed with agriculture. No doubt other groups of polygons were probably missed accidentally.
  
• Spherical mercator projection was used throughout because I could not easily get the projection option of osm2pgsql to work under windows.

The next step is to repeat this within PostGIS and do a direct comparison with the official data. However, there are some obvious edits which I can do now to make the OSM data even better.

Oh, and how long did it take me: just over 3 hours, and it would have been faster if I hadn't clipped the bounding box a little too tightly on the first pass.

The Rogue Polygon and Urban Atlas

OSM's bard was having problems yesterday with Corine data for Ireland. It turned out that the Corine class 231 (permanent pasture) ran more or less continuously through Ireland from Malin Head to Kinsale Head. This created a single relation with around 17,000 members, the "rogue polygon" (see image). Needless to say required a lot of processing with osm2pgsql for rendering with mapnik. At least it might be accurate rather than the large healthcare facility that Google thinks exists in Letterkenny.

I learnt about the related EEA (European Environment Agency) project called the Urban Atlas. This uses a similar classification of landuse to Corine, but a much smaller minimum size for survey units. The maps produced have a lot of detail and look very interesting. Compared to some other countries the UK has only got coverage of a limited number of cities. But, Nottingham is included. In fact the Nottingham Urban Atlas covers a huge area apparently contiguous with the boundaries of the Districts of Erewash, Broxtowe, Rushcliffe, Ashfield, Mansfield and Gedling, and Nottingham Unitary Authority.

In some parts of Europe, such as Poland, Urban Atlas data is being prepared for import to OSM. Although not interested in the data for import I expected it data to be useful for two reasons in the first instance: completeness of coverage, and as a control for OSM tagging. The Corine division of urban fabric (roughly equivalent to OSM landuse=residential) makes useful distinctions with regards to density and continuity of urban areas, which, to date, are under represented in OSM. This type of distinction has a long history in landuse mapping. In earlier days it was concerned with urban areas which could be productive for fruit and vegetables. Nowadays, water run-off and nature conservation issues are probably more significant. As a mapper one is always aware of the difference, but I, for one, don't try and capture it explicitly. The Urban Atlas methodology has criteria which it should not be too hard to adapt for OSM, but we do need to think about appropriate tagging.

Of course the first area I looked at in the Nottingham Urban Atlas data was the two slices I've mapped this year. Part of the point of combining survey and Bing data is to capture quantitative information about landuse. So here is the area from the Urban Atlas, using the standard colour scheme:


The first thing which strikes me is that the Edwards Lane Estate is not given a uniform classification. This seems very odd: the estate is clearly a single uniform design including road widths, garden sizes, house types. I assume that it was planned with a specific population density in mind, and specific guidelines in terms of percentage of the area devoted to houses, gardens, open space and road infrastructure. Here's the same data with houses I've recently mapped in OSM : it's very clear that areas with identical numbers of identical houses have been assigned to different categories:


So note the darker red areas with code 11210 : on the ground this is indistinguishable from the rest of the estate (coded 11220) . All I can conclude is that the data were not subjected to ground validation: an overview of the methodology is shown in this presentation. Detailed inspection of other well-mapped areas reveals similar minor anomalies: industrial and commercial areas spilling over to residential blocks; areas in campuses (colleges, hospitals and universities) being incorrectly assigned to residential categories; and soon and so forth. These nuances probably don't affect the reliability of the data for its purpose, but they do reflect how the data was sourced. One issue might be how parcel boundaries are selected, which may have the effect of putting all of some shared resource in one parcel (such as a kids' playground). I'd hoped to find time to measure housing density based on OSM data for the shapes shown above, and thus have more confirmation. It will have to wait for another time.

The bottom line: this sort of data set can be created as a side effect of areas which have already been mapped in OSM. A small amount of additional tagging is needed on residential areas, but otherwise a comparable dataset can already be produced for much of the Nottingham area. The EEA should start considering OSM as a primary source for any extensions of this dataset in places like Germany and many parts of the UK. OSM data is likely to be more accurate, reflect better understanding of the locale, and be more up-to-date.

Postscript. I find it slightly boggling that I'm disparaging a data set which two years ago I would have thought was fantastic. It's great to find how powerful OSM is becoming.