A little surprise hidden in OS Terrain50 open elevation data

The reporting and discussion of a mountain rescue on the the minor Lakeland peak of Barf continues to generate a mass of things to investigate and follow-up. I've already written two OSM diary entries on assigning slope angle to paths because of it: here and here.

To recap, recently the Keswick Mountain Rescue were called out to help 3 people who were stuck ("cragfast") on steep ground on Barf. They didn't feel confident to retrace their steps back down very steep (30 degrees or over) scree and had reached a small crag with no obvious visible route through it. They had been using one of the mobile phone outdoor hiking apps which uses OpenStreetMap data for suggesting walks.

The event was reported in the national press (The Guardian) and, of most relevance, by a specialist magazine, The Great Outdoors. The latter did a great job of thoroughly researching a news article, getting comments from the OpenStreetMap Foundation. Their specialist advisor Alex Roddie decided to publish the content of his initial response to Carey Davies, the author of the article. Both articles are well worth reading for an in-depth initial response to the situation and the issues arising. There have also been lively exchanges both on Twitter and Mastodon, again containing useful comments from people with a lot of experience of the outdoor scene in the UK.

I won't comment here on the whys and wherefores of what kinds of paths and hiking routes should get mapped on OSM. The subject is complicated enough in Great Britain as all this discussion shows. One of the side distractions this did prompt for me was looking at how we could show contour data for OSM in the British Isles (and specifically on Andy Townsend's rural hiking map).

A toot by Nigel Parish did highlight that typically maps based on OpenStreetMap have contours which are greatly smoothed compared with the actual terrain. His example was from Andy Allan's Thunderforest Outdoor style which is incorporated into a number of hiking apps. Like most regular OSM contributors, I'm used to OSM-based maps using SRTM data to create contours: both because it's free and because it has more-or-less worldwide coverage. However, we are also aware of the deficiencies of this data: it has a relatively low resolution of around 1 second of arc (so roughly 30 metres resolution) which smooths contour lines and hillshade overlays.

I had recently downloaded the Ordnance Survey's own open data elevation model Terrain50, which I used for the first of my diary entries. This has a grid interval of 50 m, so although undoubtedly more accurate than SRTM, it is of somewhat lower resolution. I therefore expected contours generated from the DTM to be comparable with those of SRTM.

I'd forgotten that I'd used the vector dataset, so my first step was to generate contours from the raster tiles provided by Terrain50.

 

Barf with contours derived from the raster DTM of Terrain50.

After I'd done this, I remembered that the Geopackage was easy to use, so I added that. To my surprise the contours had rather more detail than those derived from ostensibly the same dataset.

With contours from Terrain50 vector data

Lastly, I created 10 m contours from the 1m Lidar data available from the Environment Agency for NW22NW. This, as expected shown both high resolution and accuracy.

With 10m contours derived from Environment Agency 1m Lidar.

To compare the three types of 10 m contours I changed colours and their background. Here are all three at a scale of 1:1000 on the steep SW slope of Barf between the Bishop of Barf and the crag which balks some walkers. In the image below it can be seen that the vector Terrain50 data is of much higher resolution than the raster data from the same source. In most places vector Terrain50 data is much closer in alignment with the contours derived from 1m Lidar data. As Nigel pointed out the more detailed contours give a significantly different impression of the shape of the hillside.

All three types of contours shown together. Red derived from OS Terrain50 raster; Orange: OS Terrain50 vector; and Green derived from Environment Agency 1m Lidar DTM. Scale 1:1000.

Nigel also pointed out that lots of relevant terrain features, shown on printed maps (Harvey & OS), and available as online tiles, were missing. This type of detail is often difficult to generate automatically using software, and even then is unlikely to match the output of skilled draughtsman working to detailed composition rules. I tried adding a hillshade layer from EA data, but it didn't really pick out this detail. Then I remembered that Luke Smith of Grough had used a layer provided with OS VectorMap District called "Ornament" when he prototyped maps based on open data. I therefore added this layer. It works quite well at low zooms, but is rather ugly at anything higher than 1:5000.

Added hillshade from Environment Agency data and Ornament from OS VectorMap District open data.
Hillshade is less effective than I expected at high zooms, and the ornaments only really work in a range 1:5k (as here) to 1;25k or thereabouts.

Lastly, and more for fun than anything else, I regenerated contours at 15 m intervals and clipped these by scree layers mapped on OpenStreetMap and changed the colour of the contours to a shade of grey to replicate how this information is shown on the Harvey Map. The end result is a good illustration of these differences: the Harvey Map generalises features such as cliffs and places them to enhance the map users' impression of the terrain, whereas the OS ornament looks nice, but is nothing like as good a guide for the walker.

With 15 m contours (from EA 1m DTM) and with contours on scree coloured grey rather than brown (as done by Harvey Maps), plus OS VMD ornament.

The bottom line from this is quite simple: don't bother with Environment Agency Lidar data if you just want contours: Terrain50 vector data is much easier to get up and running. The data is also packaged as mbtiles, but I couldn't find a way to style this in QGIS.

Lastly a big thank you to Nigel Parrish who caused to to look at the data, and thus discover the difference between the vector & raster versions.

10 years of East Midlands OSM Meet-ups

After a hiatus of a year I held an on-line meeting for East Midland mappers back in March, with another in April. March was an important anniversary as we held our first meeting in March 2011, so it was the group's 10th anniversary. This is a little bit about recent meetings, but mainly a chance to share what works for us in the hope it may help others contemplating trying to get a local group together.

2021 Meetings

Somewhat daringly we agreed to meet in person in May, albeit with a number of changes designed to mitigate risk from covid: a quiet location, an afternoon meeting time, location reachable quickly by public transport by participants, possibility of being outdoors. Fortunately the venue we use in Derby, The Brunswick, meets all these criteria. 

 

I was a little shocked on viewing OSMCal that we were in the vanguard of OSM local meetings returning to in-person gatherings. My appetite for risk is perhaps higher than I thought, given that I have spend much of the past 15 months shielding as a CEV (clinically-extremely vulnerable) person. Even by May I had only made 1 shopping trip into the city centre, and had not been on public transport or visited a supermarket since March 2020. This was also my first meeting with friends rather than family or neighbours.

It all felt remarkably normal. The pub was quiet, service discrete and speedy. Despite my initial misgivings I felt comfortable indoors (it was pretty cold at the end of May & I was wearing winter-levels of clothing). I think it helps that all of us have been vaccinated, most with 2 doses, and, as discussed below, as a group most of us have a low number of day-to-day contacts. In fact I felt able to take my father there a week later for his birthday.

I can't remember exactly which subject we touched on in May, but topics we have talked about this year:

• Locked gates. This was a particular desire of John Stanworth who has been mapping them fairly systematically around Sheffield. SomeoneElse now shows them on his hiking map style for the UK & Ireland (see changelog entry for March 29).
  

• Static railway carriages mapped as building=railway_carriage

• Trackless walking & hiking routes. These are very common in Scotland where very well known ways up mountains have very few traces on the ground, and even when they exist they are often hard to tell from paths made by deer or sheep. Although walkers will not follow exactly the same route, they are relatively easy to verify from experience and knowledge. Often a path does appear high-up the hill as route options coalesce. This is an example from my own surveying in North Wales. Similar issues occur for backcountry skiing, alpine and ski mountaineering routes and bushwhacking in the Adirondacks. I've never found a tagging combination which meets this need, other than using a highway tag seems wrong, and potentially dangerous. (An side aspect are parts of regular trails which are for practical purposes invisible, and therefore should probably not be mapped as highway=path or highway=footway).
  
  
• Simple 3D buildings and F4map. Paul the Archivist has explained his approach in a diary entry.
  
  
• Gate types. I learnt something reading the wiki, in that wicket gates originally referred to small gates in larger doors or gates. Despite having passed through one many times a day for 3 years I never heard it referred to by that term, and in general my image is much more related to a gate which looks like part of a wicket fence (presumably influenced by cricket wickets). There are numerous tags which are duplicated across barrier and gate[_:]type: bump gate, kissing gate, lych gate etc. From a walker's perspective often the most important thing is whether the gate is meant for pedestrians or vehicles.
  
  
• Electric Vehicle chargers. Rovastar gave a detailed breakdown of the complexity of mapping electric vehicle chargers, and how it will also be problematic to show the data once they become ubiquitous. I'm hoping he might write this up as a diary entry: he's thought about the issues in depth.
  

Whether we meet next month really depends on the state of the Covid pandemic. Current doubling times make me pessimistic.

East Midlands Local Group 

After 10 years it's worth taking stock of where we are, mainly because there may be things of value for other local group organisers. Some of these notes were written originally for Kyle Pullicino, who asked for information on the talk mailing list, although they were always written with this purpose in mind.

I co-organised the original Nottingham OSM get-together nearly 10 years ago, and I have more-or-less organised, or, perhaps better, facilitated, these activities ever since.

This is a long post as I'm taking the opportunity to provide some detailed thoughts on these 10 years. First, I describe our group, what we do & things I've thought of doing. Second, I try & make some concrete recommendations. 

We meet regularly (or did until Covid-19) once a month. We get anywhere from 3-8 people on a regular basis with a regular core of 5-6. We have a number of very active mappers (at one stage 3-4 in top 100 worldwide), and people active in the wider OSM community (OSM-Carto, weeklyOSM, DWG, SotM volunteers, etc.). A couple of members run or are major contributors to OSM-related websites or resources (OSM-Nottingham, Evesham Mapped and switch2osm.org). We have a steady, but small, number of people who pop-in to ask specific questions or just to meet people, these might be mappers, developers or academics.

Location

Our meetings are held in a pub, partly because OSM in the UK features a lot of folk who like pubs, but also for mundane practical reasons. People who work can't meet in the day time, relatively few coffee places are open later in the evening, it's free (except for drinks & in a larger group one can join without having to buy a drink & most pubs do a good range of non-alcoholic drinks these days), relatively easy to find a central location. From my point of view, if no-one turns up I can still sit quietly with a drink for a while, and I'm not risking a not insignificant part of my monthly income on venue hire. I also try & choose one which has some food options (some people may come straight from work) & caters for a range of dietary choices (gluten free & vegetarian I always check). In London suitable adjacent fast-food options are more common, but this breaks up the meeting as people leave to buy food.

For location we chose somewhere central convenient for public transport (within 10 minutes walk of different options). This has been very useful as we now get people coming from further afield. For the past few years we've used other towns as well & this brings in different people.

Time & Date

For personal reasons weekend meetings were not possible for me: I do other things at weekends, caring responsibilities etc.. I suspect the same is true of most of our other attendees too. I do organise an Saturday meeting once or twice a year with the objective of mapping footpaths, where we share lunch together. The Norwich group have been successful with a Saturday morning meeting, followed by coffee or lunch.

I think it's generally true that people who map in OpenStreetMap may have a lot of other shared interests. Many of our group are keen walkers, cyclists, followers of folk music, owners of allotments, real ale enthusiasts, interested in historic buildings and so on. These shared interests do help a group to gel, but perhaps make it harder for newcomers to feel included. 

Who we are

We are, for the most part, men with an IT/STEM background over 40 with perhaps more over 60 than under. The age range is not untypical of the OSM mappers in the UK, but in other countries typical contributors may be younger (France, Germany, Spain, much of Latin America to my knowledge). Our youngest member, turned 30 last year. She is a professional geographer, coming to our first meeting when she was a student, but pretty much shares the interests I list above. Our oldest members are in their late '60s or early '70s. 

Also, fairly typically for OSM, the vast majority of us have degrees with perhaps half having a masters or doctorate. A very high proportion either run their own business or work for small companies, and quite a few of us are retired. Many worked from home long before the Covid pandemic. This may reflect the STEM-bias found across the board in OSM contributors, but may also represent a pattern of working which is more compatible with contributing to OSM.

From a personal viewpoint this has all been very rewarding, these people are good friends (in fact the monthly OSM meeting has become my most regular social gathering, because other ones fell by the wayside when people's caring responsibilities, including my own, grew too much. I think this also sustains the group. 

I should add that a perhaps under-appreciated aspect of the group is how many have had some caring responsibilities: largely for elderly parents, but also grandchildren and one diabetic cat. Another related factor is that in the 10-year time-frame about half of us have lost one parent; and shockingly we lost a member, David Evans, when he was in his early 40s. This means some useful conversations have been about night-sitting care-workers or getting probate. The downside is, as I say, it may be harder for newcomers to feel able to join or contribute to: a) an existing group with its own dynamic; or b) a group which is different from oneself. I have no good answers to these issues, but perhaps having different forms of meeting may attract different groups.

Mapping Activities

During the Summer months (April-October) we provide for an hour of mapping activity before meeting (largely determined by how long I can make available beforehand). At the outset we did this together as it was a vehicle for explaining OSM to people. More recently we meet & go off & do our own thing, although I am always available to show someone the ropes if a newcomer turns up. Our pub locations are deliberately chosen to maximise the range of different urban landscapes within a short distance. For the past few years we have concentrated on updating shops in the city centre (which is not just good for explaining OSM to people, but also fits with a constrained mapping time, and for exploring what continuous maintenance of data entails). This is a good opportunity to show different editors (GoMap, Vespucci, Street Complete etc).

What we talk about

Once in the pub we probably focus the first 45-60 minutes to discussing mapping problems: particularly as we have someone who comes with a long list of such questions (see above for this month's issues). General gossip about events in the broader OSM community may also happen. After that our discussions become much more social. The format works well for someone wanting some specific answers as they can leave after the first hour. 

Alternative types of Events

Things I haven't done, which might be worth considering:

• Formal talks (like Geomob in London), this could work well, but requires a suitable space, speakers & much more preparation. Other tech groups such as Linux User Groups use this approach.
• Half or one day workshops. Again more work (& possibly more than one person). Topics from mapping through to configuring a render server or using QGIS
• Missing Maps sessions or similar humanitarian mapping. These sessions appear to be very popular and can be held in small or large settings. They attract a different group of people (in London, more young people, more women & more GIS professionals). On the other hand I think there is a poor transference from mapping for humanitarian purposes to mapping locally (not so true the other way), so it might not build a local mapping community
• Interacting with local student groups. I've done a bit, but have meant to do more.
• Linking up with other like-minded orgs: Wikimedia, tech user groups etc.

One thing I would caution is that creating a group may not actually increase the number of mappers you have. We are pretty much the same set of active mappers as 10 years ago, although we have acquired one new active mapper in the past 3 years.

Starting out: recommendations for a local group

This is very much based on what I did. No doubt there are other ways of doing it.

• Decide on an initial meeting format, location & time well in advance. In the current circumstances that may well be a Zoom online meeting or similar. You need to use your judgement (& personal circumstances) to choose these. You may want to consider if these are suitable for a repeating event, as it's much easier to schedule follow-ups on that basis.

• Contact as many active mappers as you can (via OSM messaging, email). Contact local interest groups: walkers, cyclists, tech, wikimedia etc). You can canvas people for themes at this point which can set a structure for the meeting.

• Once you have basic interest & a time & place for a meeting, advertise it more widely (OSM mail, twitter, facebook .....). It's difficult to underestimate just how many different communication methods are used by those interested in OSM. Add it to OSMCal and then it will be included in weeklyOSM. Write an OSM Diary entry (& then we can write a news item about it in weeklyOSM).
• Have the meeting! Importantly, check at the end if people want to do it again, what frequency they feel is suitable (monthly, every 6 weeks & quarterly are good choices) & what things they would like to do (social/mapping etc).
• Write up what you did. I did this for the first few meetings, but it can me more of a burden later on.
• If all goes well, schedule meetings for the rest of the calendar year. It makes organising easier & helps diary management; and also helps attendees.

Personal Motivation

Lastly, think a bit about your own goals of trying to organise the community. I know we have woefully failed at my original one to get 10 highly active mappers in the city of Nottingham, on the other hand we do provide a continuous point-of-contact for local people to find out about OSM. Getting to know each other, what we are interested in, what & how we map have been immensely beneficial in terms of creating a more communal approach to mapping. It's often much easier to discuss how to map something face-to-face than by email & even better if you stand in front of it.

Whatever you do, only take on that which you feel comfortable with. It's quite a lot of work (at least initially) and there are lots of things it would be nice to do if one had the time, skills etc. OSM is built on 'good enough' not being perfect. Ideally, you might find people with complementary skills who might be interested in doing some of the other things.

People do run out of steam doing these things, and its worth recognising it. The one thing I did find extremely demotivating was the "Craftmappers" post by Michal Migurski back in 2016. For a few days I felt like throwing in the towel, and leaving the group to its own devices. I think for many of us it is important to avoid much of the mud-slinging which purports to be discussion in many OSM channels (for this reason I do not subscribe to many mailing lists), and focus on what works for us at a local level. 

Local groups are not so common that they do not need a little bit of nurturing. I'd like the OSM Foundation and local chapters to consider this more seriously.

Housing Terraces in Wales : a minor addressing conundrum

Two terraces of housing off the Holyhead Road (former A5) in Llanfair PG.
Penucheldre and (to the right Britannia) Terrrace. A typical type of housing throughout Wales, they provoke some addressing conundrums.

In May 2015 I attended the funeral of a relative in the Anglesey village of Llanfairpwllgywngyll (usually referred to as Llanfairpwll or Llanfair PG, but perhaps better known for having an extremely silly long form of the name). This is the ancestral village of my maternal great-grandfather: the funeral marked the end of around 150 years of members of the family living in the same house that he grew up in.

I know the house well. As a small child I visited frequently. It was only a short excursion by bus from Bangor, where we lived. My mother knew it much better: in the early part of WWII she and her grandmother lived here with my great-grandfather's sister. Even when I was small the village was not very large. When my mother was a girl there was much open land between the old part of the village (Pentre Uchaf) and the newer part (Pentre Isaf) next to the railway and main road to Holyhead. By the late 1970s the village had grown immeasurably with lots of overspill suburban housing for Bangor.

The big change between 1939 and the end of the '70s was that streets started to be named and houses numbered along the street. Prior to that building development had been piecemeal: most usually a mix of individual houses and most typical of many parts of Wales: named terraces. By this I mean short terraces of houses where the terrace rather than the closest street provides the name used in the address. Elsewhere there are plenty of terraced houses where individual terraces have names (often shown on a carved stone set into the brickwork of the terrace), but the numbering of houses solely relates to the street.

2019 New Year Footpath Mapping : Lees Derbyshire

Since 2015, East Midland mappers (and one or two from further afield) have met close to New Year to fill gaps in footpath mapping on OpenStreetMap. This year we met at Lees, a small village, only a few miles west of Derby, but distinctly rural nonetheless.

SomeoneElse & will_p joining trigpoint & me near Grange Hill Farm

The area roughly between Lees and Longford and south of Long Lane (which follows the line of a Roman Road) has long been a hotspot of unmapped paths on OSM.

Hotspots of public rights of way missing from OSM in the East Midlands
Lincolnshire is too far for our gathering so for the past two years we have met at each of the areas to the W & SW of Derby. The other area to the NE of Derby has been largely filled in during the year.

Lees is just about the only place which can truly be called a village: even the places with churches — Long Lane village, Trusley and Thurvaston — do not have much more than a church, a few houses and a couple of farms. Other hamlets, such as Osleston, are even smaller, but have field signs that in medieval times they were much more extensive. I haven't been able to find much on the historical development of the area which might explain this rather unusual settlement pattern.

Solar panels by the stream just outside Lees.
 This gives a good feel for the countryside: sheep grazing (on ridge and furrow), many hedges with scattered standard trees (usually ash), farm ponds surrounded by trees (willows & ash), streams with a narrow band of riparian woodland (willows, poplars, hawthorn & blackthorn scrub). 

The countryside is unremarkable: low rolling hills on Mercia Mudstones, but without the deeply incised streams, as found, for instance, in the Trent Valley. The farmland still seems to be predominantly pasture, with relatively little subdivision of fields for horses as is now common near towns. Farmsteads are large, and either heavily modernised with massive industrial sheds or apparently mouldering with decaying brick outbuildings.

Farm out-building at Osleston

The formula for these gatherings is now fairly set: meet at 10:30, walk, usually separately or perhaps as a pair for a couple of hours; gather at a pub for lunch from 12:30 onwards; repeat the walking in the afternoon. Although this limits the actual mapping time it provides a proper opportunity for everyone to get together, allows scope for people with other commitments, and allows a bit more co-ordination for the afternoon. It is also not unknown for the weather to be unfriendly in Britain, so the provision of the chance of a bit of dry shelter is always a good idea.

Early results of the mapping, new (or changed) paths in blue, hedges & fences in red

We had a reasonable idea of suitable targets when we met.: I had taken the chance of a trip to Shrewsbury in October to make a preliminary recce of the area and of the pub; SomeoneElse had mapped one longer distance route (the Bonnie Prince Charlie Walk through the area too); and Dudley passes through Kirk Langley on his way to work.

Lees, detailed mapping by will_p
(from SomeoneElse's map)

Will, as he did last year, combined mapping villages (Long Lane, Lees) with footpaths. I went off to explore the path system to the west of Osleston; SomeoneElse headed south to Trusley; Trigpoint arriving from the west, looked at paths around Sutton-on-the-Hill, and Dudley worked on an intricate network of short paths around Kirk Langley.

Detailed mapping of Long Lane village by will_p

During my mapping I made an effort to capture panoramas of the countryside from suitable vantage points to double check against aerial imagery when mapping hedges, farmyards, buildings, and the type of farmland. I hope to be able to use this to continue to add detail.

Tight corner at Russel's Old Place, Osleston.
Only the verge on the left is actually a verge, the grass on the right being part of a garden.

One particular problem in this area is that the roads are narrow (one car wide), and often little in the way of verges. The lack of verges is a hazard for walking along those sections of road, but also restricts viable places to park. SomeoneElse's custom map render therefore shows verge key values for minor roads as can be seen here.

Trigpoint trimming back blackthorn on a fairly typical footbridge with a stile at one end.

We all went our different ways for the morning session, and through a quirk of our choices did not touch the area immediately to the west of Lees. In the afternoon we therefore set off as a group, divided into two parties, re-met again (see main picture) and split off again. This allowed us to cover most of the target paths in the area with the exception of one loop in the SW.

Violet Bramble Rust (I didn't entirely neglect looking at the natural history).

A WWII pill box near Broad Chase

These joint footpath mapping sessions are undoubtedly useful, but its worth putting this in perspective: we mapped around 25 km of paths. Derbyshire as a whole has over 5,200 km. So 5 of us spending about 12 hours walking paths only touched a half of one percent of the total. The real value is we can add much more detail to the area than the single mapper, and that detail can be enriched on return visits.

An experimental rendering of OSM data for the area at 1:20k using QGIS.
The aim is to show similar features to the OS 1:25k Explorer maps.

We are not done with this area there's perhaps twice as much more still to be surveyed, and the same is true of the National Forest area, S of Swadlincote, which we visited last year. However, it will not be long before the remaining hotspots are all in much less accessible parts of Lincolnshire. In a couple of years I'll need a different theme for these gatherings.

Social Housing polygons for England : generalisation from point data

A likely Addison Street candidate, Cefn Fforest, Blackwood
cc-by-sa/2.0 - © Jaggery - geograph.org.uk/p/2

John Boughton (Municipal Dreams) was recently looking for streets named after Christopher Addison a pioneer ofpost-WWI housing legislation in Britain. It was easy to find all the roads with Addison in the name from OpenStreetMap, but much less easy to spot those which were likely to be named after him rather than other Addisons.

Merseyside, NW Cheshire & SW Lancs, showing areas of social housing.
These are concave hull polygons derived from clusters of NROSH postcodes.

In order to reduce the number of roads to be searched  one would ideally have information about when the buildings were built, and whether they were built to provide social (council) housing or not. There is limited open data on the overall age of British housing stock, but no direct information on the original developer of housing. Both are things which may ultimately be of interest to add to OSM, but it will be many years before such information has any utility on a national scale. Furthermore both are hard to check on the ground: at least for the typical mapper.

It occurred to me that one national open data set, that of the National Register of Social Housing (hereafter NROSH), could be useful. This stopped being maintained in 2013, but provides addresses for millions of houses (approx 4 million in 350k postcodes) as of that time. Given that, since then, very few new homes have been added to social housing stock, and many have been removed, this can identify likely areas of social housing.

The NROSH data therefore seemed a good place to get to grips with clustering in PostGIS, particularly as I had a specific objective in mind.

Clustering NROSH Data

Normally one sees clustering as a means of reducing clutter on webmaps, but it's only relatively recently that I realised that these techniques have great potential for performing various generalisations on detailed geographic data (particularly OSM, which tends to the detail rather than the general).

NROSH data is only geocoded at the postcode level. There may be tens of addresses at an individual postcode or just one. At the outset I treated all postcodes equally ignoring the number of addresses. I was mainly concerned to aggregate them into coherent clusters. I grabbed some code from a GIS StackOverflow question & tweaked it very lightly:

SELECT row_number() over () AS id,
  ST_NumGeometries(gc),
  gc AS geom_collection,
  ST_Centroid(gc) AS centroid,
  ST_MinimumBoundingCircle(gc) AS circle,
  sqrt(ST_Area(ST_MinimumBoundingCircle(gc)) / pi()) AS radius
FROM (
  SELECT unnest(ST_ClusterWithin(geom, 100)) gc
  FROM nrosh_pc_geo
) f

To my mind ST_ClusterWithin is still rather like magic. It groups individual postcodes which are within (in the example) 100 metres of each other. It returns all the clusters in an array, so this needs to be unnested to get each cluster. It is an aggregate function so other columns can be used for clustering (for instance local authority might have been a useful one if I'd included it in the imported data).

I initially experimented with NG8 postcodes: this area of Nottingham (see my last post) has many council estates built between the early 1920s to the 1970s (see Municipal Dreams blog for details). Trying with various distances for clustering I found 150 m worked pretty well. In London, and possibly other large cities with many postcodes on a road, this was too high.

The cluster itself is a geometry collection of the original points. It is therefore trivial to calculate a hull for the collection. Fortunately these days ST_ConcaveHull does not break with target percents of less than 99%, and it produced sensible results.

Odd-shaped polygons for Irby on the Wirral. Individual postcodes only have a few NROSH entries. Presumably both roads around the school were built as council housing, but most have now become privately owned.

I extended the code to the entire data set. I soon realised that it was excluding areas of social housing sharing a single postcode. As there are some interesting examples of rural council housing I wanted  them in the overall data set too.

One of the more uunusal social housing forms, Stoford
cc-by-sa/2.0 - © Nigel Mykura - geograph.org.uk/p/4314325

My solution was simple: instead of using points I buffered them by 10 metres. This simply ensures that no data gets thrown away in subsequent steps. It does not mean that one gets a very accurate polygon when there are a very small number of postcodes in the cluster (less than 5 perhaps). If actual geocoded addresses are available then it will be possible to produce more accurate polygons. I haven't tested this, but this should be possible for any local authority where a decent number of addresses are mapped on OSM. In my local path there are several areas of Nottingham, Gedling, Broxtowe and Erewash which meet this requirement.

Overview of the Resultant Data

See full screen

Throughout the Midlands, North-west England and parts of East Anglia the data looks pretty sensible. In general I've looked at places I know and checked that the edges of the polygons accord with what I know of housing patterns in those areas. For now I've tried to put a sample area for Notts, Derbys up on umap.

Hampton, Hanworth & Teddington area of SW London.
In general this is a pretty prosperous part of London. It does pick out some areas of social housing (e.g., near Apex Corner (A312/A junction), but the notion that most of Teddington is social housing is absurd. The 2015 Index of Multiple Deprivation gives a better picture of this area. See additional notes at the foot of the blog.

NW1 postcode. 150 m clusters with 100 m clusters overlaid.
Reducing the cluster distance to 100 m greatly improves the elimination of false positives. Places like the Ossulston Estate between Euston & the British Library show clearly. I'm less convinced that this does not result in many false negatives

I have not scrutinised everywhere, but a few obvious oddities I've noticed:

• Sheffield & Redditch seem to be data deficient.
• Areas in London are far too large to be usable, and even reducing the clustering distance does not make a massive difference (see images & commentary in the captions above).

The data is quite large so I havent yet been able to publish it somewhere readily accessible. In the meantime I can share it in various geoformats if you are interested. There's also scope to use IMD & the housing age stats to separate things out a bit more, but I'm just as interested in places which are now predominantly privately owned, but were built as social housing.

I hope this data can be used for various things. In particular, I have long been interested in the possibility of finding Radburn layouts using OSM data. (Ian Waites has more on these estates). A reduction in the total areas to search is always valuable. I'm sure other uses will occur to both social historians, and mappers. On the technical side I hope this might also provoke others to explore the potential for clustering in PostGIS: there's lot to learn.

Further Notes on Hanworth, Hampton & Teddington

I looked at this area because I lived in three places here during the 1980s and 1990s.

Hanworth, the area in the London Borough of Hounslow had a lot of social housing, especially north of the Great Chertsey Road. S of the road housing was extremely mixed with small private speculative developments, older properties, infill, fields with horses grazing and so on. I bought a house in this area in 1986 which was built in the 1960s. Today this property appears in the NROSH data, so it has moved from the private to the social sector in the past 25 years. We bought it because the location was convenient (I used public transport and caught the bus at Apex Corner) and the house had been extended and was larger than equivalent properties we had looked at. It was sold in the early-1990s to an family of South Asian heritage, who probably bought it for similar reasons.

Further south is the Hampton Nurserylands Estate. In the 1980s this was full of young professionals, many with young children. However, it changed demographically rather quickly. Many of the original buyers moved out to bigger houses within 2-3 years, and were replaced by older less-prosperous families. I remember looking at a flat here in 1993 and being staggered how much the area had changed in 3-4 years. The houses on the W side of Oak Avenue were social housing in the 1980s. Clearly these changes have continued.

I can't really explain how Teddington has so many social housing postcodes. It is really one of the most prosperous places in Britain.

Mapping roof-top Solar Panels

We've all noticed that solar panels have become increasingly frequent on the roofs of residential buildings. It's one of the things I have taken note of ever since I started contributing to OpenStreetMap. However, I had never tried to add any to OSM. Until now!

Roof mounted solar PV panels on a semi-detached house.
There are 18 panel modules (in 2 rows of 7 and one of 4). Each module consists of a 12 by 6 array of solar cells (see below for further discussion).
Photo by Phil Sangwell on Flickr via Wikimedia Commons. CC-BY-SA

A couple of days ago Jack Kelly suggested that perhaps we could use OSM to capture the presence of solar panels across the UK.  A lively twitter discussion ensued.

How crazy would it be for national electricity grids to use @OpenStreetMap as *the* authoritative database of all solar photovoltaic systems?@mclduk @alex_rogers_cs @dekstop @EnergySRE

— jack_kelly (@jack_kelly) March 6, 2019

It seemed sensible to have a go at scoping what was involved. Over the past few months I've been improving the mapping of inter-war housing estates in Nottingham, with the current focus on the Aspley Estate which contains perhaps 2,400 houses. In the course of visits and scrutinising aerial imagery I already knew that there were a fair number of roof-top photovoltaic (PV) panels already installed. Unfortunately on my last visit to get representative photos of the buildings I only caught one house in the background.

Housing in the Aspley Estate: note the solar PV panel on house in background (this one on OSM).

Particularly useful is that the best quality imagery layer available for Aspley is Bing, and it shows the panels clearly. In fact sufficiently clearly that I decided to map them as areas.

In a relatively short time (15-20 minutes) I had found just over 200. Unfortunately I was also reminded that there were quite a few houses in the NW sector of the estate which I had not mapped, so I then spent a while adding houses and addresses, followed by fixing a lot of QA issues pointed out by the JOSM validator tool. Only then was I able to align the houses & solar panels, which took another hour.

This was a little long winded for first-cut mapping, so the following morning I gave myself 30 minutes and searched through adjacent housing estates which I suspected would have a similar density of panels as I had found in Aspley.

Solar Panels on new-build replacement housing at Rutland Close, The Meadows.
A recent example of housing I've surveyed without paying particular attention to solar.

One reason for this is that social housing often has a much higher density of solar panel installation than private housing. Firstly the housing stock is often of similar or identical buildings under one ownership enabling economies of scale. Secondly, owners of social housing are much exercised by fuel poverty of their tenants: reducing fuel costs through providing electricity from solar power therefore has much to commend it. Thirdly, Nottingham City Homes, the at-length housing provider for Nottingham City Council, has a great deal of expertise in applying greener energy polices to their housing. Through an odd coincidence I saw a tweet about what they have achieved as I was doing my second scoping task. This set a target of around 4,500 panels to find in the city.

With my second run just mapping each panel as a node I found 320, or just over 10 a minute. This was partially targeted because I pretty much restricted myself to examining areas of social housing. It therefore represents a rather efficient data acquisition rate.

Jack extrapolated this to the whole country mapped in 5 days by 33 people working full-time. This is of course highly optimistic because I was mapping areas I know well from having being mapping them for 10 years on OSM. However, the OSM community is full of people with detailed knowledge of their local areas, so this ought to apply or many parts of the country. even if it was 5-10 times as much effort, say 1 a minute, 100 people mapping for a couple of hours a week for a quarter might find 150k. This suggests solar panels may be a good subject for a Quarterly Project.

Solar PV panels mapped in Nottingham, via Overpass Turbo

Now, 48 hours after I started, I have added 1760 solar panels to OSM in Nottingham. It's time to summarise what I have learnt. In no particular order:

• All available imagery layers need to be searched. New installations are occurring all the time and it's unlikely that the better quality imagery layers will be recent enough to enable adequate coverage.
• Newer imagery, such as the Digital Globe layers can be quite grainy & hard to interpret. However once panels have been spotted it is usually possible to then find many more.
• The huge variability in available imagery is likely to make any attempt to use machine learning to identify targets is likely to be fraught. I would also expect things like glass roofed extensions would generate many false positives.
• Knowing where panels are likely to be installed helps a great deal: both at the neighbourhood and building level. Christian Quest's OpenSolarMap used some crowd-sourced information from aerial photos to train  system to identify buildings in France with potential for installation of photovoltaic panels. Such information for the UK could reduce the total number of buildings needing to be inspected.
• Larger detached houses with solar panels are very difficult (impossible?) to pick out from aerial imagery. Shadows from chimneys, and changing roof lines obscure the presence of the panels.
• Mapping panels as nodes is the best approach initially. I used ID which has a suitable preset (and checked what others had already done, for instance brianboru around Birmingham). Thereafter I just copied the original node.
• Adding a tag to show that they are roof-mounted is useful (particularly if the building has not been mapped yet). I've used generator:location=roof. Indicating domestic use might also be helpful. The basic tags I used are also used for complete solar farm installations & clearly it is important to distinguish them. (I've subsequently learnt that generator:place=roof is the established tag).
• Many installations are sufficiently clear on aerial imagery to allow estimation of the number of panel modules involved. Virtually all the ones at Aspley are 2 rows of 5 modules. Unfortunately I don't know the exact module size, but they are probably 10 or 12 by 6 cells. Tagging the module array explicitly is probably better than guesstimating the area (as I have done).
• If module size is known (see top photo) the array area can be calculated directly. Each solar cell is likely to be 156 mm square, so a 10 by 6 array will be 1.56 x 0.96 m (1.46 sq m).
• Cell size, module size and number of modules allow optimal power rating to be estimated. I think these arrays of roughly 15 sq m are around 3500-3700 W.
• Adding compass orientation of the array in degrees, and angle from the horizontal would also be helpful for using the data for estimating likely power output. (Both could be derived from simple 3D building tags, but adding these is much more complex).
• The last few items (no. of modules, module size, area, power rating, orientation & angle) represent data which can be added iteratively.
• It's worthwhile surveying at least some of those added from aerial imagery to capture other information.
• Even if the panel is mapped as an area, most of these tags are still useful as it is unlikely that enough information will be present on the underlying buildings to derive them.
• Surprisingly few public buildings, education establishments or industrial buildings have solar panel installations. I've only noticed a few on buildings of Derby Hall, a hall of residence at the University of Nottingham, and a couple of warehouses in Bulwell.

Other than quickly looking for whether anyone had mapped roof-mounted solar panels in the UK, I haven't looked at activity in other countries. There may be places with a more developed approach to mapping and tagging.

A couple of caveats:

• solar panel distribution is likely to be very patchy; 
• aerial imagery may not be good enough to pick out panels, or recent enough (many of the Nottingham panels have been installed since 2014). 

To help judge what the latter point may mean I provide below a selection of available aerial imagery of various locations in Nottingham, and Basingstoke (Hampshire Council Open Data). The latter includes false colour infra-red (FCIR). 

Aspley Estate (Bing Imagery), roughly here.

Aspley Estate (Bing Imagery) roughly here.

Aspley Estate (Digital Globe Standard Imagery) location as above

Aspley Estate (Digital Globe Premium and ESRI World Imagery) location as above

Broxtowe Lane (Bing Imagery), about here.

Broxtowe Lane (Digital Globe Premium Imagery), same location as above. Obviously newer as a solar panel can be made out on the terrace in the centre. Note on the next terrace down a dark area on the roof. This does not appear to have the same visual appearance as other solar panels, so may be a solar hot water system. 

Broxtowe Lane, as above (Digital Globe Standard imagery)

Deptford Crescent area, Highbury Vale (Bing imagery). Area bottom right is Highbury Hospital.
No solar panels visible

Deptford Crescent (ESRI World Imagery)

Deptford Crescent (Digital Globe Standard Imagery)

Astrid Gardens, Bestwood Estate (ESRI Imagery).
Just occasionally panels have very strong reflections as here.

Astrid Gardens, Bestwood Estate (Digital Globe Standard Imagery). 

Astrid Gardens, Bestwood Estate (Bing Imagery). 

Bilborough Estate (Digital Globe Standard Imagery)
Note the panel lower right which is pretty hard to pick out.

Britten Road, Basingstoke (Hampshire false-colour Infra-red imagery)
This estate has quite a few solar hot water installations but only a few solar PV. The hot water ones are smaller and less obviously modular.

Britten Road, Basingstoke (Hampshire visible spectrum RGB imagery)

So if there are any other takers I suggest this is a suitable topic for a future UK Quarterly Project, perhaps in association with Jez Nicholson's current interest in solar & wind farms.

Lastly I'd like to thank Jack Kelly & Dan Stowell for comments & ideas whilst mapping & writing this up.