tag:blogger.com,1999:blog-20824227836926310782024-03-18T09:48:09.873+00:00Maps MatterSK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.comBlogger129125tag:blogger.com,1999:blog-2082422783692631078.post-90937686640339427722023-01-15T20:06:00.023+00:002023-01-15T20:20:27.884+00:00A little surprise hidden in OS Terrain50 open elevation data<p>The reporting and discussion of a mountain rescue on the the minor Lakeland peak of <a href="https://en.wikipedia.org/wiki/Barf_(Lake_District)">Barf</a> continues to generate a mass of things to investigate and follow-up. I've already written two OSM diary entries on assigning slope angle to paths because of it: <a href="https://www.openstreetmap.org/user/SK53/diary/400702">here</a> and <a href="https://www.openstreetmap.org/user/SK53/diary/400712">here</a>.</p><p style="margin-left: 40px; text-align: left;"><span style="font-size: x-small;">To recap, recently the Keswick Mountain Rescue were called out to help 3 people who were stuck ("cragfast") on steep ground on Barf. They didn't feel confident to retrace their steps back down very steep (30 degrees or over) scree and had reached a small crag with no obvious visible route through it. They had been using one of the mobile phone outdoor hiking apps which uses OpenStreetMap data for suggesting walks.</span></p><p style="margin-left: 40px; text-align: left;"><span style="font-size: x-small;">The event was reported in the national press (<a href="https://www.theguardian.com/uk-news/2023/jan/06/hiking-app-alltrails-changes-route-rescue-three-walkers-lake-district">The Guardian</a>) and, of most relevance, by a specialist magazine, The Great Outdoors. The latter did a great job of thoroughly researching a <a href="https://www.thegreatoutdoorsmag.com/news/concerns-raised-over-crowdsourced-maps-used-by-popular-hiking-apps/?fbclid=IwAR06UnMiPKkHQmQjbfz5AZ6o5AHEJQ6ndQUEU3EWle0hLRKJSubNqKJ4yjk">news article</a>, getting comments from the OpenStreetMap Foundation. Their specialist advisor Alex Roddie decided to publish <a href="https://www.theguardian.com/uk-news/2023/jan/06/hiking-app-alltrails-changes-route-rescue-three-walkers-lake-district">the content </a>of his initial response to Carey Davies, the author of the article. Both articles are well worth reading for an in-depth initial response to the situation and the issues arising. There have also been lively exchanges both on Twitter and Mastodon, again containing useful comments from people with a lot of experience of the outdoor scene in the UK.</span></p><p>I won't comment here on the whys and wherefores of what kinds of paths and hiking routes should get mapped on OSM. The subject is complicated enough in Great Britain as all this discussion shows. One of the side distractions this did prompt for me was looking at how we could show contour data for OSM in the British Isles (and specifically on Andy Townsend's rural hiking map).</p><p>A <a href="https://osdatahub.os.uk/downloads/open/Terrain50">toot</a> by Nigel Parish did highlight that typically maps based on OpenStreetMap have contours which are greatly smoothed compared with the actual terrain. His example was from Andy Allan's <a href="https://www.thunderforest.com/maps/outdoors/">Thunderforest Outdoor</a> style which is incorporated into a number of hiking apps. Like most regular OSM contributors, I'm used to OSM-based maps using <a href="https://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission" target="_blank">SRTM</a> data to create contours: both because it's free and because it has more-or-less worldwide coverage. However, we are also aware of the deficiencies of this data: it has a relatively low resolution of around 1 second of arc (so roughly 30 metres resolution) which smooths contour lines and hillshade overlays.</p><p>I had recently downloaded the Ordnance Survey's own open data elevation model <a href="https://osdatahub.os.uk/downloads/open/Terrain50">Terrain50</a>, which I used for the first of my diary entries. This has a grid interval of 50 m, so although undoubtedly more accurate than SRTM, it is of somewhat lower resolution. I therefore expected contours generated from the DTM to be comparable with those of SRTM.</p><p>I'd forgotten that I'd used the vector dataset, so my first step was to generate contours from the raster tiles provided by Terrain50. <br /></p><p> </p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-7oJWClN01wzjCud5imWEKX_gaN_KRo0qjIYv8-N1qeTcjrasZzzBWkkpp0L5cvv79nY40zK8QQrlXzjsoiHUG2DwVMk-s9_ETFdkzVZJaKTEX3OXVRjIFuPPp67K4rmT2N5YJTxqUdK3A1R-xLMCV_Wan_T_bjnOEH89shTuK60KmxmWpfgg0-QEwQ/s1516/barf_contours.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1516" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-7oJWClN01wzjCud5imWEKX_gaN_KRo0qjIYv8-N1qeTcjrasZzzBWkkpp0L5cvv79nY40zK8QQrlXzjsoiHUG2DwVMk-s9_ETFdkzVZJaKTEX3OXVRjIFuPPp67K4rmT2N5YJTxqUdK3A1R-xLMCV_Wan_T_bjnOEH89shTuK60KmxmWpfgg0-QEwQ/w400-h231/barf_contours.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Barf with contours derived from the raster DTM of Terrain50.</span><br /></td></tr></tbody></table><br /><p>After I'd done this, I remembered that the Geopackage was easy to use, so I added that. To my surprise the contours had rather more detail than those derived from ostensibly the same dataset.<br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd2kHq8lqO_0QbmmZ6ran-Q7RwxtZRuMw7HM-LyIRbc3X3DP009rTL1iARxVZ_Kx5yNqSG512igkRy2KdTH-7aQa87miM1uhWRViGwoWl_g7ysKqxjbmJ8_3NvgHkQx6RsjVj6-23oFHxqmu0tJgYIYgZnzk_RNp8UB5NJevBk373dv3vuJkokppkawA/s1516/barf_contours_vector.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1516" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd2kHq8lqO_0QbmmZ6ran-Q7RwxtZRuMw7HM-LyIRbc3X3DP009rTL1iARxVZ_Kx5yNqSG512igkRy2KdTH-7aQa87miM1uhWRViGwoWl_g7ysKqxjbmJ8_3NvgHkQx6RsjVj6-23oFHxqmu0tJgYIYgZnzk_RNp8UB5NJevBk373dv3vuJkokppkawA/w400-h231/barf_contours_vector.png" width="400" /></a><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">With contours from Terrain50 vector data</span><br /></td></tr></tbody></table><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd2kHq8lqO_0QbmmZ6ran-Q7RwxtZRuMw7HM-LyIRbc3X3DP009rTL1iARxVZ_Kx5yNqSG512igkRy2KdTH-7aQa87miM1uhWRViGwoWl_g7ysKqxjbmJ8_3NvgHkQx6RsjVj6-23oFHxqmu0tJgYIYgZnzk_RNp8UB5NJevBk373dv3vuJkokppkawA/s1516/barf_contours_vector.png" style="margin-left: auto; margin-right: auto;"></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><br /></td></tr></tbody></table><p></p><p>Lastly, I created 10 m contours from the 1m Lidar data available from the Environment Agency for NW22NW. This, as expected shown both high resolution and accuracy.<br /></p><p><br /></p><p></p><p></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJRCaC_ffyWPsREr3moRdFKpOxVvKYoQDhF4wh30yIep5sz_IZL5U57Th_06zwXc1ED_Xh-2DZmtxOI5Ux05mYaZkABVKklR5hMVG8esehXX8nAytgKrZAv7RSBdG7wfYN2DH3Or8rntvFHNcphWcNcMOTtRphvjPitRx61ly-M0FFFIhOugzP6COR9Q/s1038/barf_contours_defra.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1038" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJRCaC_ffyWPsREr3moRdFKpOxVvKYoQDhF4wh30yIep5sz_IZL5U57Th_06zwXc1ED_Xh-2DZmtxOI5Ux05mYaZkABVKklR5hMVG8esehXX8nAytgKrZAv7RSBdG7wfYN2DH3Or8rntvFHNcphWcNcMOTtRphvjPitRx61ly-M0FFFIhOugzP6COR9Q/w400-h338/barf_contours_defra.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">With 10m contours derived from Environment Agency 1m Lidar.</span><br /></td></tr></tbody></table><p>To compare the three types of 10 m contours I changed colours and their background. Here are all three at a scale of 1:1000 on the steep SW slope of Barf between the Bishop of Barf and the crag which balks some walkers. In the image below it can be seen that the vector Terrain50 data is of much higher resolution than the raster data from the same source. In most places vector Terrain50 data is much closer in alignment with the contours derived from 1m Lidar data. As Nigel pointed out the more detailed contours give a significantly different impression of the shape of the hillside. <br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPQr_tGD1CGsTFbO7nR17-c0M19GKsvf0ZTF_KatrFIi7uYU-rJpCFxXAJ-utfkf3VR6e-K5HbduDuLFnCWbVJOgdQiZReq2iVh83w08z7aj12Xr9kw9DcZ3i6iBylw24jjnscKFriZBLd_CeQDkNjL3ekXIgU6-kyq9TQll_XUZfBWut_4WbFcXDZ4w/s1038/barf_contour_comparison.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1038" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPQr_tGD1CGsTFbO7nR17-c0M19GKsvf0ZTF_KatrFIi7uYU-rJpCFxXAJ-utfkf3VR6e-K5HbduDuLFnCWbVJOgdQiZReq2iVh83w08z7aj12Xr9kw9DcZ3i6iBylw24jjnscKFriZBLd_CeQDkNjL3ekXIgU6-kyq9TQll_XUZfBWut_4WbFcXDZ4w/w400-h338/barf_contour_comparison.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">All three types of contours shown together. Red derived from OS Terrain50 raster; Orange: OS Terrain50 vector; and Green derived from Environment Agency 1m Lidar DTM. Scale 1:1000.</span><br /></td></tr></tbody></table><p></p><p>Nigel also <a href="https://en.osm.town/@nigelparrish@mastodonapp.uk/109688440936706340">pointed out</a> that lots of relevant terrain features, shown on printed maps (Harvey & OS), and available as online tiles, were missing. This type of detail is often difficult to generate automatically using software, and even then is unlikely to match the output of skilled draughtsman working to detailed composition rules. I tried adding a hillshade layer from EA data, but it didn't really pick out this detail. Then I remembered that Luke Smith of Grough <a href="https://lists.openstreetmap.org/pipermail/talk-gb/2016-April/018740.html">had used</a> a layer provided with OS VectorMap District called "Ornament" when he prototyped maps based on open data. I therefore added this layer. It works quite well at low zooms, but is rather ugly at anything higher than 1:5000.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4BS5PrbSk2k5tDVOWd0paMajLqUCzpPDhha6vFgk5LLlopOQfLcDh3JW-fmHeVYADon4nOaLao2xgdjJUcmaNJKjOBl3Zud62VBEFKPlbkZWfF12O3LQzV4AjWlz_ailjg9oZew8gxHUGPkG9cWg1fGQFyG2EdVB1f9WJjefhQjjT4_0hHjyTCXe6qQ/s1038/barf_contours_vmd_ornament.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1038" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4BS5PrbSk2k5tDVOWd0paMajLqUCzpPDhha6vFgk5LLlopOQfLcDh3JW-fmHeVYADon4nOaLao2xgdjJUcmaNJKjOBl3Zud62VBEFKPlbkZWfF12O3LQzV4AjWlz_ailjg9oZew8gxHUGPkG9cWg1fGQFyG2EdVB1f9WJjefhQjjT4_0hHjyTCXe6qQ/w400-h338/barf_contours_vmd_ornament.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Added hillshade from Environment Agency data and Ornament from OS VectorMap District open data.<br />Hillshade
is less effective than I expected at high zooms, and the ornaments only
really work in a range 1:5k (as here) to 1;25k or thereabouts.</span><br /></td></tr></tbody></table><p></p><p>Lastly, and more for fun than anything else, I regenerated contours at 15 m intervals and clipped these by scree layers mapped on OpenStreetMap and changed the colour of the contours to a shade of grey to replicate how this information is shown on the Harvey Map. The end result is a good illustration of these differences: the Harvey Map generalises features such as cliffs and places them to enhance the map users' impression of the terrain, whereas the OS ornament looks nice, but is nothing like as good a guide for the walker.<br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6hvtPjwuniLUw4RpM-KuLqk3oy0A3ckCiVbJGTucTPNxvdiA3hTqk_rWU_X3z7M_zIOE-m-qELGSzja9a6VL9ZQWolrKjuipdXBKU6yPdNiGgIPxsrLOIePbJB98NdWKK-d-Bte202QhtZfG1i-MPgXvdh3nU6iVRcHaJ618tv5aMclqWaAcKqXmThA/s1038/barf_contour_defra_15m_vmd_orn.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="875" data-original-width="1038" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6hvtPjwuniLUw4RpM-KuLqk3oy0A3ckCiVbJGTucTPNxvdiA3hTqk_rWU_X3z7M_zIOE-m-qELGSzja9a6VL9ZQWolrKjuipdXBKU6yPdNiGgIPxsrLOIePbJB98NdWKK-d-Bte202QhtZfG1i-MPgXvdh3nU6iVRcHaJ618tv5aMclqWaAcKqXmThA/w400-h338/barf_contour_defra_15m_vmd_orn.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">With 15 m contours (from EA 1m DTM) and with contours on scree coloured grey rather than brown (as done by Harvey Maps), plus OS VMD ornament.<br /></td></tr></tbody></table><p><br /><br />The bottom line from this is quite simple: don't bother with Environment Agency Lidar data if you just want contours: Terrain50 vector data is much easier to get up and running. The data is also packaged as mbtiles, but I couldn't find a way to style this in QGIS.</p><p>Lastly a big thank you to Nigel Parrish who caused to to look at the data, and thus discover the difference between the vector & raster versions.<br /><br /></p><p><br /></p>SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.comtag:blogger.com,1999:blog-2082422783692631078.post-14167303285515139512021-06-24T10:55:00.001+01:002021-06-24T10:55:34.981+01:0010 years of East Midlands OSM Meet-ups<p>After a hiatus of a year I held an on-line meeting for East Midland mappers back in March, with another in April. March was an important anniversary as we held our first meeting in <a href="http://sk53-osm.blogspot.com/2011/03/nottingham-osm-pub-meet-march-8th.html">March 2011</a>, so it was the group's 10th anniversary. This is a little bit about recent meetings, but mainly a chance to share what works for us in the hope it may help others contemplating trying to get a local group together.</p><h2 style="text-align: left;">2021 Meetings<br /></h2><p>Somewhat daringly we agreed to meet in person in May, albeit with a number of changes designed to mitigate risk from covid: a quiet location, an afternoon meeting time, location reachable quickly by public transport by participants, possibility of being outdoors. Fortunately the venue we use in Derby, The Brunswick, meets all these criteria. </p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxZNRxZj4WP7PjUWT2dlB7b_bObRi27GvyOqSITixnWflgo-4QF2i29E4fJq2EPr-y-mrKwDYElrohpUgnuOkOLIpN4AohF24cW4Z4wsVG_mq4HV3ajoc1X-Fpqhh-a8KYt5dbT9e9fvQK/s3855/20210525_164125%255B1%255D.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Mappers in the snug at The Brunswick" border="0" data-original-height="816" data-original-width="3855" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxZNRxZj4WP7PjUWT2dlB7b_bObRi27GvyOqSITixnWflgo-4QF2i29E4fJq2EPr-y-mrKwDYElrohpUgnuOkOLIpN4AohF24cW4Z4wsVG_mq4HV3ajoc1X-Fpqhh-a8KYt5dbT9e9fvQK/w640-h136/20210525_164125%255B1%255D.jpg" title="May 2021 East Midlands Pub Meet-up" width="640" /></a></div><br /> <p></p><p>I was a little shocked on viewing <a href="https://osmcal.org/">OSMCal</a> that we were in the vanguard of OSM local meetings returning to in-person gatherings. My appetite for risk is perhaps higher than I thought, given that I have spend much of the past 15 months shielding as a CEV (clinically-extremely vulnerable) person. Even by May I had only made 1 shopping trip into the city centre, and had not been on public transport or visited a supermarket since March 2020. This was also my first meeting with friends rather than family or neighbours.</p><p>It all felt remarkably normal. The pub was quiet, service discrete and speedy. Despite my initial misgivings I felt comfortable indoors (it was pretty cold at the end of May & I was wearing winter-levels of clothing). I think it helps that all of us have been vaccinated, most with 2 doses, and, as discussed below, as a group most of us have a low number of day-to-day contacts. In fact I felt able to take my father there a week later for his birthday.</p><p>I can't remember exactly which subject we touched on in May, but topics we have talked about this year:</p><ul style="text-align: left;"><li><b>Locked gates</b>. This was a particular desire of John Stanworth who has been mapping them fairly systematically around Sheffield. SomeoneElse now shows them on his <a href="https://map.atownsend.org.uk/maps/map/map.html#zoom=15&lat=54.209&lon=-1.23322">hiking map</a> style for the UK & Ireland (see <a href="https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/changelog.md">changelog</a> entry for March 29).<br /></li></ul><ul style="text-align: left;"><li><b>Static railway carriage</b>s mapped as building=railway_carriage</li></ul><ul style="text-align: left;"><li><b>Trackless walking & hiking routes</b>. These are very common in Scotland where very well known ways up mountains have very few traces on the ground, and even when they exist they are often hard to tell from paths made by deer or sheep. Although walkers will not follow exactly the same route, they are relatively easy to verify from experience and knowledge. Often a path does appear high-up the hill as route options coalesce. This is an example from my own surveying in North Wales. Similar issues occur for <a href="http://sk53-osm.blogspot.com/2013/12/mapping-off-piste-ski-routes.html">backcountry skiing</a>, alpine and ski mountaineering routes and <a href="https://dftscript.blogspot.com/2020/11https://dftscript.blogspot.com/2020/11/2020-11-14-delhi-trails.html/2020-11-14-delhi-trails.html">bushwhacking</a> in the Adirondacks. I've never found a tagging combination which meets this need, other than using a highway tag seems wrong, and potentially dangerous. (An side aspect are parts of regular trails which are for practical purposes invisible, and therefore should probably not be mapped as highway=path or highway=footway). <br /><br /></li><li><b>Simple 3D buildings and F4map</b>. Paul the Archivist has explained his approach in a diary entry.<br /><br /></li><li><b>Gate types</b>. I learnt something reading the wiki, in that wicket gates originally referred to small gates in larger doors or gates. Despite having passed through one many times a day for 3 years I never heard it referred to by that term, and in general my image is much more related to a gate which looks like part of a wicket fence (presumably influenced by cricket wickets). There are numerous tags which are duplicated across barrier and gate[_:]type: bump gate, kissing gate, lych gate etc. From a walker's perspective often the most important thing is whether the gate is meant for pedestrians or vehicles.<br /><br /></li><li><b>Electric Vehicle chargers</b>. Rovastar gave a detailed breakdown of the complexity of mapping electric vehicle chargers, and how it will also be problematic to show the data once they become ubiquitous. I'm hoping he might write this up as a diary entry: he's thought about the issues in depth.<br /></li></ul><p>Whether we meet next month really depends on the state of the Covid pandemic. Current doubling times make me pessimistic.<br /></p><h2 style="text-align: left;">East Midlands Local Group </h2><p style="text-align: left;">After 10 years it's worth taking stock of where we are, mainly because there may be things of value for other local group organisers. Some of these notes were written originally for Kyle Pullicino, who <a href="https://lists.openstreetmap.org/pipermail/talk/2020-December/085843.html">asked</a> for information on the talk mailing list, although they were always written with this purpose in mind.<br /></p><p>I co-organised the original Nottingham OSM get-together nearly 10 years
ago, and I have more-or-less organised, or, perhaps better, facilitated,
these activities ever since.<br /><br />This is a long post as I'm taking
the opportunity to provide some detailed thoughts on these 10 years.
First, I describe our group, what we do & things I've thought of
doing. Second, I try & make some concrete recommendations. </p><p>We meet regularly (or did until Covid-19) once a month. We get anywhere from 3-8 people on a regular basis with a regular core of 5-6. We have a number of very active mappers (at one stage 3-4 in top 100 worldwide), and people active in the wider OSM community (<a href="https://github.com/gravitystorm/openstreetmap-carto">OSM-Carto</a>, <a href="https://weeklyosm.eu/">weeklyOSM</a>, <a href="https://wiki.openstreetmap.org/wiki/Data_working_group">DWG</a>, <a href="http://web.archive.org/web/20150210001912/http://2013.stateofthemap.org/about/credits/">SotM volunteers</a>, etc.). A couple of members run or are major contributors to OSM-related websites or resources (<a href="https://osm-nottingham.org.uk/">OSM-Nottingham</a>, <a href="http://www.evesham-mapped.org.uk/">Evesham Mapped</a> and <a href="http://switch2osm.org">switch2osm.org</a>). We have a steady, but small, number of people who pop-in to ask specific questions or just to meet people, these might be mappers, developers or academics.</p><h3 style="text-align: left;">Location<br /></h3><p>Our meetings are held in a pub, partly because OSM in the UK features a lot of folk who like pubs, but also for mundane practical reasons. People who work can't meet in the day time, relatively few coffee places are open later in the evening, it's free (except for drinks & in a larger group one can join without having to buy a drink & most pubs do a good range of non-alcoholic drinks these days), relatively easy to find a central location. From my point of view, if no-one turns up I can still sit quietly with a drink for a while, and I'm not risking a not insignificant part of my monthly income on venue hire. I also try & choose one which has some food options (some people may come straight from work) & caters for a range of dietary choices (gluten free & vegetarian I always check). In London suitable adjacent fast-food options are more common, but this breaks up the meeting as people leave to buy food.<br /><br />For location we chose somewhere central convenient for public transport (within 10 minutes walk of different options). This has been very useful as we now get people coming from further afield. For the past few years we've used other towns as well & this brings in different people.</p><h3 style="text-align: left;">Time & Date<br /></h3><p>For personal reasons weekend meetings were not possible for me: I do other things at weekends, caring responsibilities etc.. I suspect the same is true of most of our other attendees too. I do organise an Saturday meeting once or twice a year with the objective of mapping footpaths, where we share lunch together. The Norwich group have been successful with a Saturday morning meeting, followed by coffee or lunch.<br /><br />I think it's generally true that people who map in OpenStreetMap may have a lot of other shared interests. Many of our group are keen walkers, cyclists, followers of folk music, owners of allotments, real ale enthusiasts, interested in historic buildings and so on. These shared interests do help a group to gel, but perhaps make it harder for newcomers to feel included. </p><h3 style="text-align: left;">Who we are<br /></h3><p>We are, for the most part, men with an IT/STEM background over 40 with perhaps more over 60 than under. The age range is not untypical of the OSM mappers in the UK, but in other countries typical contributors may be younger (France, Germany, Spain, much of Latin America to my knowledge). Our youngest member, turned 30 last year. She is a professional geographer, coming to our first meeting when she was a student, but pretty much shares the interests I list above. Our oldest members are in their late '60s or early '70s. </p><p>Also, fairly typically for OSM, the vast majority of us have degrees with perhaps half having a masters or doctorate. A very high proportion either run their own business or work for small companies, and quite a few of us are retired. Many worked from home long before the Covid pandemic. This may reflect the STEM-bias found across the board in OSM contributors, but may also represent a pattern of working which is more compatible with contributing to OSM.<br /></p><p>From a personal viewpoint this has all been very rewarding, these people are good friends (in fact the monthly OSM meeting has become my most regular social gathering, because other ones fell by the wayside when people's caring responsibilities, including my own, grew too much. I think this also sustains the group. </p><p>I should add that a perhaps under-appreciated aspect of the group is how many have had some caring responsibilities: largely for elderly parents, but also grandchildren and one diabetic cat. Another related factor is that in the 10-year time-frame about half of us have lost one parent; and shockingly we lost a member, <a href="https://vimeo.com/78424813">David Evans</a>, when he was in his early 40s. This means some useful conversations have been about night-sitting care-workers or getting probate. The downside is, as I say, it may be harder for newcomers to feel able to join or contribute to: a) an existing group with its own dynamic; or b) a group which is different from oneself. I have no good answers to these issues, but perhaps having different forms of meeting may attract different groups.</p><h3 style="text-align: left;">Mapping Activities<br /></h3><p>During the Summer months (April-October) we provide for an hour of mapping activity before meeting (largely determined by how long I can make available beforehand). At the outset we did this together as it was a vehicle for explaining OSM to people. More recently we meet & go off & do our own thing, although I am always available to show someone the ropes if a newcomer turns up. Our pub locations are deliberately chosen to maximise the range of different urban landscapes within a short distance. For the past few years we have concentrated on updating shops in the city centre (which is not just good for explaining OSM to people, but also fits with a constrained mapping time, and for exploring what continuous maintenance of data entails). This is a good opportunity to show different editors (GoMap, Vespucci, Street Complete etc).</p><h3 style="text-align: left;">What we talk about<br /></h3><p>Once in the pub we probably focus the first 45-60 minutes to discussing mapping problems: particularly as we have someone who comes with a long list of such questions (see above for this month's issues). General gossip about events in the broader OSM community may also happen. After that our discussions become much more social. The format works well for someone wanting some specific answers as they can leave after the first hour. </p><h3 style="text-align: left;">Alternative types of Events<br /></h3><p>Things I haven't done, which might be worth considering:<br /></p><ul style="text-align: left;"><li><b>Formal talks</b> (like Geomob in London), this could work well, but requires a suitable space, speakers & much more preparation. Other tech groups such as Linux User Groups use this approach.</li><li>Half or one day <b>workshops</b>. Again more work (& possibly more than one person). Topics from mapping through to configuring a render server or using QGIS</li><li><b>Missing Maps</b> sessions or similar humanitarian mapping. These sessions appear to be very popular and can be held in small or large settings. They attract a different group of people (in London, more young people, more women & more GIS professionals). On the other hand I think there is a poor transference from mapping for humanitarian purposes to mapping locally (not so true the other way), so it might not build a local mapping community</li><li>Interacting with <b>local student groups</b>. I've done a bit, but have meant to do more.</li><li>Linking up with other like-minded orgs: Wikimedia, tech user groups etc.</li></ul><p><br />One thing I would caution is that creating a group may not actually increase the number of mappers you have. We are pretty much the same set of active mappers as 10 years ago, although we have acquired one new active mapper in the past 3 years.<br /><br /></p><h3 style="text-align: left;">Starting out: recommendations for a local group</h3><p style="text-align: left;">This is very much based on what I did. No doubt there are other ways of doing it.<br /></p><ul style="text-align: left;"><li>Decide on an initial meeting format, location & time well in advance. In the current circumstances that may well be a Zoom online meeting or similar. You need to use your judgement (& personal circumstances) to choose these. You may want to consider if these are suitable for a repeating event, as it's much easier to schedule follow-ups on that basis.</li></ul><ul style="text-align: left;"><li>Contact as many active mappers as you can (via OSM messaging, email). Contact local interest groups: walkers, cyclists, tech, wikimedia etc). You can canvas people for themes at this point which can set a structure for the meeting.</li></ul><ul style="text-align: left;"><li>Once you have basic interest & a time & place for a meeting, advertise it more widely (OSM mail, twitter, facebook .....). It's difficult to underestimate just how many different communication methods are used by those interested in OSM. Add it to OSMCal and then it will be included in weeklyOSM. Write an OSM Diary entry (& then we can write a news item about it in weeklyOSM).</li><li>Have the meeting! Importantly, check at the end if people want to do it again, what frequency they feel is suitable (monthly, every 6 weeks & quarterly are good choices) & what things they would like to do (social/mapping etc).</li><li>Write up what you did. I did this for the first few meetings, but it can me more of a burden later on.</li><li>If all goes well, schedule meetings for the rest of the calendar year. It makes organising easier & helps diary management; and also helps attendees.</li></ul><h3 style="text-align: left;">Personal Motivation <br /></h3><p>Lastly, think a bit about your own goals of trying to organise the community. I know we have woefully failed at my original one to get 10 highly active mappers in the city of Nottingham, on the other hand we do provide a continuous point-of-contact for local people to find out about OSM. Getting to know each other, what we are interested in, what & how we map have been immensely beneficial in terms of creating a more communal approach to mapping. It's often much easier to discuss how to map something face-to-face than by email & even better if you stand in front of it.<br /><br />Whatever you do, only take on that which you feel comfortable with. It's quite a lot of work (at least initially) and there are lots of things it would be nice to do if one had the time, skills etc. OSM is built on 'good enough' not being perfect. Ideally, you might find people with complementary skills who might be interested in doing some of the other things.</p><p>People do run out of steam doing these things, and its worth recognising it. The one thing I did find extremely demotivating was the <a href="http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.html">"Craftmappers"</a> post by Michal Migurski back in 2016. For a few days I felt like throwing in the towel, and leaving the group to its own devices. I think for many of us it is important to avoid much of the mud-slinging which purports to be discussion in many OSM channels (for this reason I do not subscribe to many mailing lists), and focus on what works for us at a local level. </p><p>Local groups are not so common that they do not need a little bit of nurturing. I'd like the OSM Foundation and local chapters to consider this more seriously.<br /><br /><br /></p>SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-554802175231992702020-06-05T12:22:00.002+01:002020-06-05T12:22:46.537+01:00Housing Terraces in Wales : a minor addressing conundrum<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7F8nrtsyr-KYm5NUxe5_9V2PxnKxxCXR9WreEfMSap2Nf4R_qTR8gA6fOajFHJc1F30roiab3cGX2GGg8JxqVuaNBBRXYiZ1qhkFAjX8G2nb3VHuv3uHKfaCccdIvlYfGbaiXLaV_faaA/s1600/IMG_7493a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7F8nrtsyr-KYm5NUxe5_9V2PxnKxxCXR9WreEfMSap2Nf4R_qTR8gA6fOajFHJc1F30roiab3cGX2GGg8JxqVuaNBBRXYiZ1qhkFAjX8G2nb3VHuv3uHKfaCccdIvlYfGbaiXLaV_faaA/s320/IMG_7493a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Two terraces of housing off the Holyhead Road (former A5) in Llanfair PG.<br />
Penucheldre and (to the right Britannia) Terrrace. A typical type of housing throughout Wales, they provoke some addressing conundrums.</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx5xNKQFBIP852R7DTxifUqt5mwWVmTkTy1UVbIDg9xKwJVtcUbYvH6DILxI3_w91qve0JOEHMxvv8hTfW5V2s5yGrb4Fupu9EPlz_ioPuzp2zyQ6ZwTEyYlaPZJ8ulCBY2n76fBYHyTfR/s1600/llanfair_terraces.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
In May 2015 I attended the funeral of a relative in the Anglesey village of <a href="https://en.wikipedia.org/wiki/Llanfairpwllgwyngyll">Llanfairpwllgywngyll </a>(usually referred to as Llanfairpwll or Llanfair PG, but perhaps better known for having an extremely silly long form of the name). This is the ancestral village of my maternal great-grandfather: the funeral marked the end of around 150 years of members of the family living in the same house that he grew up in.<br />
<br />
I know <a href="https://www.peoplescollection.wales/items/24372">the house</a> well. As a small child I visited frequently. It was only a short excursion by bus from Bangor, where we lived. My mother knew it much better: in the early part of WWII she and her grandmother lived here with my great-grandfather's sister. Even when I was small the village was not very large. When my mother was a girl there was much open land between the old part of the village (Pentre Uchaf) and the newer part (Pentre Isaf) next to the railway and main road to Holyhead. By the late 1970s the village had grown immeasurably with lots of overspill suburban housing for Bangor.<br />
<br />
The big change between 1939 and the end of the '70s was that streets started to be named and houses numbered along the street. Prior to that building development had been piecemeal: most usually a mix of individual houses and most typical of many parts of Wales: <b>named terraces</b>. By this I mean short terraces of houses where the terrace rather than the closest street provides the name used in the address. Elsewhere there are plenty of terraced houses where individual terraces have names (often shown on a carved stone set into the brickwork of the terrace), but the numbering of houses solely relates to the street.<br />
<br />
<a name='more'></a><br />
<h4>
Terraces in Llanfair PG</h4>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AThe_junction_of_Lon_Penmynydd_and_Holyhead_Road_(A5)_-_geograph.org.uk_-_1431551.jpg" style="margin-left: auto; margin-right: auto;" title="Eric Jones [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="The junction of Lon Penmynydd and Holyhead Road (A5) - geograph.org.uk - 1431551" src="https://upload.wikimedia.org/wikipedia/commons/c/ca/The_junction_of_Lon_Penmynydd_and_Holyhead_Road_%28A5%29_-_geograph.org.uk_-_1431551.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Looking towards the main road (Ffordd Caergybi or Holyhead Road). Two terraces are visible: one on the L angled at around 60 degrees to the street, and the one on the right, Williams Terrace (I think) facing onto the street. There is another terrace behind Williams Terrace, and two more off the main road on the right.</td></tr>
</tbody></table>
There are still plenty of these terraces in the cores of the original settlements. In Llanfair they usually consist of between 3 and 6 small houses, presumably built by a single builder of fairly limited resources. The terraces are often not placed in any consistent fashion with respect to the road layout. For instance along the main Holyhead Road there are terraces fronting directly onto the road; at 90 degrees to the road accessed by a footway, and at various other angles. Given that there was plenty of land I speculate that the orientation of the terrace was probably most usually determined by size and shape of the parcel of land acquired by the builder.<br />
<br />
In practice it is quite easy to pick out candidate terraces from aerial photos. I did a couple of passes over the buildings which had already been mapped on OSM and then cross-checked with old 6 inch maps at the National Libraries of <a href="http://maps.nls.uk/geo/explore/#zoom=15&lat=53.2257&lon=-4.2067&layers=6">Scotland </a>and <a href="http://www.peoplescollection.wales/locate">Wales </a>(the Peoples Collection has more maps but no permalinks).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBcd5WDW2IU7Ons2NOkXAIPJm14QCcnfYYOE4UQ8gUluLrUf97JA_tAOIyYnF6DWqckYNbIdXV0gC4suJHttnaEp3UVxMUWpw4yBI40whEjPFH2eNSgECv3aT-3zvKTdaEbSUU3neIM_Qb/s1600/llanfair_terraces.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" height="329" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiH89Qc6KIJg8TKVQ5hyphenhyphen4Iovt0nxWu7siE-mtFgG71MRlP5ePF8BBm75dDkBAhLmsQRUxV023S5GRlw8BB_fd8_9qjeaTROzkdPvbWY2jZMRLyGuMc90jSGBHs8o2zKCoMk951xlGm8AR69/s1600/llanfair_terraces.jpg" title="Housing Terraces in Llanfair PG" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Modern day Llanfair PG shown on Bing Imagery. buildings identified as terraces are from OpenStreetMap and are shown in orange with a halo (to aid visibility).</td></tr>
</tbody></table>
Today Llanfair PG is a densely <a href="https://en.wikipedia.org/wiki/Nucleated_village">nucleated settlement</a>. enclosed between the new and old Holyhead Roads, but certainly up until WWII, and perhaps as late as the 1960s it was a dispersed rural settlement. The location of housing terraces, which for the most part date from the late 19th century, highlights this dispersal. There are two main clusters at the two historical centres of the settlement, Pentre-Isaf and Pentre-Uchaf.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ACofeb_ryfel_Llanfairpwll_War_memorial_(geograph_3172615).jpg" style="margin-left: auto; margin-right: auto;" title="Alan Fryer [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Cofeb ryfel Llanfairpwll War memorial (geograph 3172615)" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/44/Cofeb_ryfel_Llanfairpwll_War_memorial_%28geograph_3172615%29.jpg/512px-Cofeb_ryfel_Llanfairpwll_War_memorial_%28geograph_3172615%29.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Memorial to villagers of Llanfair PG who lost their lives in the First World War. There is much of historical interest here, and personal one for me as my Grandmother's cousin is one of those listed, but it is the addresses which are germane to this post.</td></tr>
</tbody></table>
The <a href="http://www.anglesey.info/llanfair-pg-war-memorials.htm">War Memorial</a> in the village is a useful source of historical addresses: virtually all are either individual house names or names of terraces. Amongst the terraces mentioned are:<br />
<ul>
<li><a href="http://www.openstreetmap.org/way/23547735">Crossing Terrace</a>. next to the railway line and a named residential road today.</li>
<li><a href="http://www.openstreetmap.org/way/35271897">Williams Terrace</a> : off Ffordd Penymynnydd and behind Britannia Terrace.</li>
<li>Bryn Goleu. Somewhere in Pentre-Uchaf, perhaps <a href="http://www.openstreetmap.org/way/371362156">this building</a>.</li>
<li><a href="http://www.openstreetmap.org/way/372600814">Britannia Terrace</a>. Fronting on to the Holyhead Road.</li>
<li><a href="http://www.openstreetmap.org/way/343540059">Penucheldre Terrace</a>. At right-angles to Holyhead Road and immediately to the W of Britannia Terrace.</li>
<li><a href="http://www.openstreetmap.org/way/371638021">Min-y-ffrwd</a>. Behind the Calvinist Methodist Chapel (and marked on the 6 inch map</li>
</ul>
Other places are easily recognised. Ty Capel & Ty Twr are both houses associated with, respectively the Chapel and the Tower (the Marquess of Anglesey's Column). Other locations are the local estates of large landowners (<a href="http://www.openstreetmap.org/way/458779199">Plas Llanfair</a> & <a href="http://www.openstreetmap.org/way/39065434">Plas Newydd</a>), and an island ([Ynys] <a href="http://www.openstreetmap.org/way/23890593">Gorad Goch</a>). My impression is that many men still worked on the large estates in a range of roles (farmhands, ostlers, servants, gamekeepers, etc., but 4 of the 26 had careers at sea. Others would have been fishermen (on Gorad Goch), dairymen or engaged in other agricultural trades. The silly long name on the station indicates that there was reasonable tourist trade which probably sustained shops and the family selling tickets for the Tower. Many younger people from the village would have moved to larger cities such as Liverpool or London to work in the building trades, in service or retail (all examples from my own family).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AThe_eastern_section_of_the_Stryd_Fawr_(High_Street)_-_geograph.org.uk_-_522556.jpg" style="margin-left: auto; margin-right: auto;" title="Eric Jones [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="The eastern section of the Stryd Fawr (High Street) - geograph.org.uk - 522556" src="https://upload.wikimedia.org/wikipedia/commons/5/54/The_eastern_section_of_the_Stryd_Fawr_%28High_Street%29_-_geograph.org.uk_-_522556.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">E end of Stryd Fawr (High Street) in Brynsiencyn. Two terraces can be seen in the middle distance on the right-hand side.<br />
Almost all the houses on this street date from late-1800s.</td></tr>
</tbody></table>
I don't believe that there were any quarrymen in Llanfair PG. The adjacent village of <a href="https://cy.wikipedia.org/wiki/Brynsiencyn">Brynsiencyn </a>did have a significant number of men who worked away during the week in slate quarries near Llanberis. Interestingly, Brynsiencyn is much more nucleated than Llanfair P.G. which may relate to having a development more greatly affected by industry. However, the main street there still consists of individual terraces.<br />
<h4>
Terraces elsewhere in North Wales</h4>
A similar pattern occurred in the larger quarrying towns such as <a href="https://en.wikipedia.org/wiki/Bethesda,_Gwynedd">Bethesda </a>and <a href="https://en.wikipedia.org/wiki/Blaenau_Ffestiniog">Blaenau Ffestiniog</a>. In Bethesda all along the A5, the main road through the middle of the town, are long terraces with up to 20 houses, such as <a href="http://www.openstreetmap.org/way/45742721">Rhes Douglas</a> (Douglas Terrace). Again each has an individual name and numbers belong to the terrace not the main road. In the <a href="http://www.openstreetmap.org/#map=17/53.17687/-4.04654">suburb of Gerlan</a> the quarrymen's cottages are clearly arranged in groups of terraces: some, such as Gwyernydd Terrace, being individually named on the <a href="http://maps.nls.uk/geo/explore/#zoom=17&lat=53.1769&lon=-4.0486&layers=6">6 inch map</a>. The same applies to the other great slate town Ffestiniog (some possibly developed by another great-grandfather of mine). Here many terraces are <a href="http://maps.nls.uk/geo/explore/#zoom=17&lat=52.9953&lon=-3.9368&layers=6">also named</a> and it is clear that a fair number were only accessible by footpaths.<br />
<br />
It wasn't only villages and quarry settlements where this held true. I first really appreciated this as a distinct phenomenon in 2010 when I carried out a brief survey around the suburb of Garth in Bangor, Gwynedd. Two roads connect the town to what was originally a fishing settlement. The upper road (Ffordd Garth Uchaf or Upper Garth Road) was largely developed in the 1930s with semi-detached houses, but the lower road, Ffordd Garth was built-up earlier. Largely this was as a series of terraces with 8 3-storey houses in each terrace. Again house numbering was within the terrace. Much more recently there has been an attempt to impose regular house numbers on both roads. House names have been supplemented by regular house numbers on Ffordd Garth Uchaf, and a single numbering scheme has also been adopted for Ffordd Garth. One can judge how successful it is by the fact that dustbins have the terrace name & number in the terrace painted on them.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsH3IssyQg25MtKNjIeM957K9XptkR9zX30ABu7WB1_BFg3I1PMVBnOcW9A8HGKDLrDkqmNu9yeGkVCY3MD0iYxUXhs2plhD0asJNSGKUo79nMwVI78wKShdOEto4OLWov0gI4SA73C78V/s1600/wales_terraces.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="247" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsH3IssyQg25MtKNjIeM957K9XptkR9zX30ABu7WB1_BFg3I1PMVBnOcW9A8HGKDLrDkqmNu9yeGkVCY3MD0iYxUXhs2plhD0asJNSGKUo79nMwVI78wKShdOEto4OLWov0gI4SA73C78V/s320/wales_terraces.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Highways with "Terrace" in their name within Wales on OpenStreetMap<br />
We can be confident that many have been missed as much of the detail mapping of Wales has used Ordnance Survey resources (either Out-of-copyright or Open Data)</td></tr>
</tbody></table>
At present OSM has at least 1500 highways (residential, footway, service etc) with Terrace in the name for Wales. An equivalent region of England, the East Midlands has less than 400, and the latter area has received more on-the-ground surveys than Wales. <br />
<br />
<h4>
Terraces Further Afield </h4>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:The_Woodcutters%27_Arms_and_Woodcutters_Row,_Foxt.jpg" style="margin-left: auto; margin-right: auto;" title="Ian Calderwood / The Woodcutters' Arms and Woodcutters Row"><img alt="The Woodcutters' Arms and Woodcutters Row, Foxt" src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/00/The_Woodcutters%27_Arms_and_Woodcutters_Row%2C_Foxt.jpg/512px-The_Woodcutters%27_Arms_and_Woodcutters_Row%2C_Foxt.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A small terrace at Foxt, Staffordshire. Unusual in that one house was a pub.<br />
Ian Calderwood on Geograph via <a href="https://commons.wikimedia.org/wiki/File:The_Woodcutters%27_Arms_and_Woodcutters_Row,_Foxt.jpg">Wikimedia Commons</a>, <span style="font-size: xx-small;">CC-BY-SA</span>.</td></tr>
</tbody></table>
Of course small discrete terraces are not restricted to Wales. A couple of examples come to mind: <a href="http://www.openstreetmap.org/way/394953915">a terrace</a> for brickyard workers on the edge of Maidenhead, and <a href="http://www.openstreetmap.org/way/39821724">terraces</a> in the Staffordshire village of Foxt close to the Churnet Valley which used to contain much water-powered industry. In addition in the densely developed urban areas of industrial cities another type of terrace was quite common. For the most part these have been demolished, but a few survive. In <a href="http://osm-nottingham.org.uk/?z=14&lon=-1.14824&lat=52.96027&bgl=OSM,1,18&s=terrace&st=AddressSearchJson">Nottingham </a>these are usually two rows of terraced housing with a common walkway: typically they have 12 or 16 houses. <br />
<h4>
The Addressing Problem</h4>
Having provided some background on the history and distribution of this form of housing in England & Wales, it's time to look at why they present some issues for tagging addresses on OSM.<br />
<br />
Loosely we can divide these terraces into 4 categories:<br />
<ol>
<li>Those where the terrace name has been transferred to a standard residential road.</li>
<li>Those where some distinct vehicle access is provided, for instance by an unsigned service road.</li>
<li>Those accessed solely by a footway</li>
<li>Those fronting onto a street which has a different name from the terrace, and house numbering is only continuous in the terrace.</li>
</ol>
For the first three categories we can use the fairly standard Karlsruhe scheme with the name of the terrace held in addr:street. To ensure that Nominatim can retrieve such addresses in cases 2 & 3 I usually add the name of the terrace to the service road or footway (even when the service road only provides access to the rear of the properties. In some senses this is a fudge, but I think a justifiable one.<br />
<br />
It is the 4th case which provides problems in the context of the Karlsruhe scheme. Essentially I know of two different strategies for resolving the conundrum:<br />
<ul>
<li>Store the name of the terrace in <span style="font-size: x-small;"><span style="font-family: "Courier New", Courier, monospace;">addr:housenam</span></span>e and the name of the road in <span style="font-size: x-small;"><span style="font-family: "Courier New", Courier, monospace;">addr:street</span></span>. (This is actually what Royal Mail do in their address file). </li>
<li>Add the name of the terrace as <span style="font-size: x-small;"><span style="font-family: "Courier New", Courier, monospace;">addr:street</span></span> and make the houses members of an <span style="font-size: x-small;"><span style="font-family: "Courier New", Courier, monospace;">associatedStree</span></span>t relation for the road.</li>
</ul>
Currently I prefer the second: at least it stores all the relevant relationships, even if they are not readily retrievable. My problem with the first approach is that it takes a common type of object and stores data associated with it in two different ways. We also cannot distinguish between a house which has both a name & a number and one which has a number and is part of a terrace. (Or a house in a named terrace with both an individual name and a number)<br />
<br />
In general <span style="font-size: x-small;"><span style="font-family: "Courier New", Courier, monospace;">addr:flats</span></span> is used for subsidiary numbers which share the same primary address. This distinction is needed because often a block of flats will both have a name and a number. It also has the advantage of not shoehorning different types of address objects into a single tag.<br />
<br />
I'm sure that Frederik Ramm, the creator of the Karlsruhe Schema would say that the former approach should be used. The original intention was to store pure postal addresses after all. However, postal addresses are not the be-all-and-end-all of addressing. I've written before about how a <a href="https://en.wikipedia.org/wiki/Procrustes#Cultural_references">Procustean approach</a> to addresses, however convenient for the postal authority or company, is often unsatisfactory for a number of reasons. As the Karlsruhe Schema is what we have had for a long time on OSM, it is this which tends to be co-opted for other addressing needs.<br />
<br />
<h4>
The Mapping Problem </h4>
This post started out from the viewpoint of how to provide accurate addresses for this distinctive housing pattern. In the course of writing it I also realise that they also present a mapping challenge.<br />
<br />
Land Registry Prices Paid ([not at all open<strike> <a href="http://mapgubbins.tumblr.com/post/160076080760/ico-land-registrys-price-paid-data-is-personal">less-and-less</a></strike>] <strike>openish</strike>) data suggests at least 2600 different street names containing "Terrace" in Wales compared with around 1500 on OSM. These might be missed because they are small groups of houses which aren't noticeable until a ground survey is undertaken; or they might be terraces aligned along main roads which actually require that the additional street name be collected (see above regarding terraces in Bethesda).<br />
<br />
In Nottingham we know that many of the residual terraces were not named in OS Open Data products, and it was only around 2013 and 2014 that we identified a number in inner city areas (largely developed before WWI). Most of these were found in surveys conducted before our monthly pub meetings. Our usual meeting place, The Lincolnshire Poacher, was chosen in part because of transport links (it's close for buses to Derby, Mansfield and the Ashfields), but also because there is a very broad mix of different urban development within 10-15 minutes walk. Many of the terraces within this area are <a href="http://www.openstreetmap.org/way/225337890">not even obvious</a> from residential streets, but come to light because there are odd gaps between streets or buildings which are not obviously accessible from the street, Field papers were invaluable for spotting many of these.<br />
<br />
In Wales we don't have as that many mappers so they are less likely to be spotted as part of in-fill surveys. Also terraces are fairly common in small rural towns and villages as well as in the post-industrial parts of South Wales. We have additional resources in terms of open data and imagery now, so perhap sthis should be another mapping project for these times!SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-903765926186209092019-12-20T13:01:00.000+00:002019-12-20T13:01:28.110+00:002019 New Year Footpath Mapping : Lees DerbyshireSince 2015, East Midland mappers (and one or two from further afield) have met close to New Year to fill gaps in footpath mapping on OpenStreetMap. This year we met at Lees, a small village, only a few miles west of Derby, but distinctly rural nonetheless.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTeojiYYDy1sfJsj6gpkwrEIJfLS1-teisFnhKZfi_9GqPY1nfuRHh4v1Yv410UcpevfHijPTJVrPMlYO241FYVTeiJtfMG1hN63AJDaSA6ZlqNQxtn3eo91Kn2d68wfPlJdH7wKuEsSwQ/s1600/P1230708-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTeojiYYDy1sfJsj6gpkwrEIJfLS1-teisFnhKZfi_9GqPY1nfuRHh4v1Yv410UcpevfHijPTJVrPMlYO241FYVTeiJtfMG1hN63AJDaSA6ZlqNQxtn3eo91Kn2d68wfPlJdH7wKuEsSwQ/s320/P1230708-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">SomeoneElse & will_p joining trigpoint & me <a href="https://www.openstreetmap.org/node/5075475257">near</a> Grange Hill Farm</td></tr>
</tbody></table>
<br />
The area roughly between <a href="https://en.wikipedia.org/wiki/Dalbury_Lees">Lees</a> and <a href="https://en.wikipedia.org/wiki/Longford,_Derbyshire">Longford</a> and south of <a href="https://en.wikipedia.org/wiki/The_Long_Lane_(Derbyshire)">Long Lane</a> (which follows the line of a Roman Road) has long been a hotspot of unmapped paths on OSM.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgn8mo2Ee-wjHBL6Es1-V9ymOhIQIeRGjsf70E9niPWun_yt9moepWueoDq-fgkjsHwRpGyr2FZP699Pw2M7XxRmbBrcPUpEhh7B2qTwomgxsurZV0bJ-BKGvDtmHbqGSN_QPy-a4xWJlj/s1600/emid_tetrad_d1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1132" data-original-width="1600" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgn8mo2Ee-wjHBL6Es1-V9ymOhIQIeRGjsf70E9niPWun_yt9moepWueoDq-fgkjsHwRpGyr2FZP699Pw2M7XxRmbBrcPUpEhh7B2qTwomgxsurZV0bJ-BKGvDtmHbqGSN_QPy-a4xWJlj/s320/emid_tetrad_d1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Hotspots of public rights of way missing from OSM in the East Midlands<br />
Lincolnshire is too far for our gathering so for the past two years we have met at each of the areas to the W & SW of Derby. The other area to the NE of Derby has been largely filled in during the year.</td></tr>
</tbody></table>
Lees is just about the only place which can truly be called a village: even the places with churches — Long Lane village, Trusley and <a href="https://en.wikipedia.org/wiki/Thurvaston">Thurvaston</a> — do not have much more than a church, a few houses and a couple of farms. Other hamlets, such as Osleston, are even smaller, but have field signs that in medieval times they were much more extensive. I haven't been able to find much on the historical development of the area which might explain this rather unusual settlement pattern.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv-pLdkFLgUPGNpxHFbSaZtGCbBnSfKomVz-QgazpUz3cgmweZTOKl3pW3epf9NGfJ83UK3fjCoLtGZbTOyRUSdaqKvfBa5d1RUB4sCDKsdIJVNFPUprjrTN2yWHMe8hOlCDD337goFT7G/s1600/P1230679-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv-pLdkFLgUPGNpxHFbSaZtGCbBnSfKomVz-QgazpUz3cgmweZTOKl3pW3epf9NGfJ83UK3fjCoLtGZbTOyRUSdaqKvfBa5d1RUB4sCDKsdIJVNFPUprjrTN2yWHMe8hOlCDD337goFT7G/s320/P1230679-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Solar panels by the stream just outside Lees.<br />
This gives a good feel for the countryside: sheep grazing (on <a href="https://en.wikipedia.org/wiki/Ridge_and_furrow">ridge and furrow</a>), many hedges with scattered standard trees (usually ash), farm ponds surrounded by trees (willows & ash), streams with a narrow band of riparian woodland (willows, poplars, hawthorn & blackthorn scrub). </td></tr>
</tbody></table>
The countryside is unremarkable: low rolling hills on Mercia Mudstones, but without the deeply incised streams, as found, for instance, in the Trent Valley. The farmland still seems to be predominantly pasture, with relatively little subdivision of fields for horses as is now common near towns. Farmsteads are large, and either heavily modernised with massive industrial sheds or apparently mouldering with decaying brick outbuildings.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQx7lUU-nSHAw5S8l537uBejVUwDA7BcOlVDFp3oCOiT5aeknq4oA44Wn0Y87hlfF66lxsNONvSlLAlZZtuhi27czxQAa0cjhD4nJc8at5gh78xiKJAduwMII5MToL_7936y1irDXDL3RG/s1600/P1230400-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="685" data-original-width="1024" height="321" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQx7lUU-nSHAw5S8l537uBejVUwDA7BcOlVDFp3oCOiT5aeknq4oA44Wn0Y87hlfF66lxsNONvSlLAlZZtuhi27czxQAa0cjhD4nJc8at5gh78xiKJAduwMII5MToL_7936y1irDXDL3RG/s320/P1230400-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Farm out-building at Osleston</td></tr>
</tbody></table>
The formula for these gatherings is now fairly set: meet at 10:30, walk, usually separately or perhaps as a pair for a couple of hours; gather at a pub for lunch from 12:30 onwards; repeat the walking in the afternoon. Although this limits the actual mapping time it provides a proper opportunity for everyone to get together, allows scope for people with other commitments, and allows a bit more co-ordination for the afternoon. It is also not unknown for the weather to be unfriendly in Britain, so the provision of the chance of a bit of dry shelter is always a good idea.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX4HDXAxLvRgEehpMdAHkFRAaiSea2-bKAhAqIlAO8AO2rLBoqnFXg_RWwWaf2rA2whHYizTIz3b9YnicIODklUOIBrTd9y1gu0EEFZoA5hugh1M5F7cT82IxMy3lntmD5LClGlFPz-V3S/s1600/lees_paths_hedges.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="631" data-original-width="1022" height="299" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiX4HDXAxLvRgEehpMdAHkFRAaiSea2-bKAhAqIlAO8AO2rLBoqnFXg_RWwWaf2rA2whHYizTIz3b9YnicIODklUOIBrTd9y1gu0EEFZoA5hugh1M5F7cT82IxMy3lntmD5LClGlFPz-V3S/s320/lees_paths_hedges.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Early results of the mapping, new (or changed) paths in blue, hedges & fences in red</td></tr>
</tbody></table>
We had a reasonable idea <span style="background-color: transparent; color: black; display: inline; float: none; font-family: "times new roman"; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">of suitable targets when we met.</span>: I had taken the chance of a trip to Shrewsbury in October to make a preliminary recce of the area and of the pub; SomeoneElse had mapped one longer distance route (the <a href="https://hiking.waymarkedtrails.org/#route?id=1574096&map=12!52.9554!-1.6262">Bonnie Prince Charlie Walk</a> through the area too); and Dudley passes through <a href="https://en.wikipedia.org/wiki/Kirk_Langley">Kirk Langley</a> on his way to work.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl1xY5acRTnCfVcHJUyoGktFVm9NFU7nUS0KpuTNAqYMqqSeU11y2gWG9mahdz2LLPCX8zohV5bM41HzvKUblJWrdAUYjEgJLDxQShn3kJcHWhJ18tAs5-aN03nWJ1x4AagsgBV5F_o4Za/s1600/lees_village_mapped.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1132" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl1xY5acRTnCfVcHJUyoGktFVm9NFU7nUS0KpuTNAqYMqqSeU11y2gWG9mahdz2LLPCX8zohV5bM41HzvKUblJWrdAUYjEgJLDxQShn3kJcHWhJ18tAs5-aN03nWJ1x4AagsgBV5F_o4Za/s320/lees_village_mapped.png" width="452" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lees, detailed mapping by will_p<br />
(from SomeoneElse's map)</td></tr>
</tbody></table>
Will, as he did last year, combined mapping villages (Long Lane, Lees) with footpaths. I went off to explore the path system to the west of <a href="https://en.wikipedia.org/wiki/Osleston">Osleston;</a> SomeoneElse headed south to <a href="https://en.wikipedia.org/wiki/Trusley">Trusley;</a> Trigpoint arriving from the west, looked at paths around <a href="https://en.wikipedia.org/wiki/Sutton_on_the_Hill">Sutton-on-the-Hill</a>, and Dudley worked on an intricate network of short paths around Kirk Langley.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://wiki.openstreetmap.org/wiki/File:Long_Lane_Village_mapped.png" style="margin-left: auto; margin-right: auto;" title="Long Lane Village detailed mapping"><img alt="Long Lane Village" src="https://wiki.openstreetmap.org/w/images/b/bd/Long_Lane_Village_mapped.png" width="359" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Detailed mapping of Long Lane village by will_p</td></tr>
</tbody></table>
During my mapping I made an effort to capture panoramas of the countryside from suitable vantage points to double check against aerial imagery when mapping hedges, farmyards, buildings, and the type of farmland. I hope to be able to use this to continue to add detail.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg87RhMMZvtUWRwouIH1EKxZnvXUDX4h_nDEXEKH9HmxLo33Iy7Mg1u9WoJ7LXFI8G9OcvhJYqh8CrxXBzxcheemZSq69C9DW4ZbH_WXPyzT-lD-iCeQTcx5n4qbNFxs2xqPHC5AL_5-iBh/s1600/P1230455-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg87RhMMZvtUWRwouIH1EKxZnvXUDX4h_nDEXEKH9HmxLo33Iy7Mg1u9WoJ7LXFI8G9OcvhJYqh8CrxXBzxcheemZSq69C9DW4ZbH_WXPyzT-lD-iCeQTcx5n4qbNFxs2xqPHC5AL_5-iBh/s320/P1230455-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Tight corner at Russel's Old Place, Osleston.<br />
Only the verge on the left is actually a verge, the grass on the right being part of a garden.</td></tr>
</tbody></table>
<br />
One particular problem in this area is that the roads are narrow (one car wide), and often little in the way of verges. The lack of verges is a hazard for walking along those sections of road, but also restricts viable places to park. SomeoneElse's custom map render therefore shows <a href="https://wiki.openstreetmap.org/wiki/Key:verge">verge key</a> values for minor roads as can be <a href="https://map.atownsend.org.uk/maps/map/map.html#zoom=16&lat=52.92689&lon=-1.60725">seen here</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjv035gwcTVF_1HICYrhxZR4Jp5CrsIG2PuvJeFIgvVpqoMZRqBiJl9GCAF2YV1HK0hnC_S8MuqbU61KqGvu3MgCRIPujK-K6MoIyq_OxorwFVt6rdIgvsncA8VoObaHtnBcsruEboUJb78/s1600/P1230695-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1024" data-original-width="768" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjv035gwcTVF_1HICYrhxZR4Jp5CrsIG2PuvJeFIgvVpqoMZRqBiJl9GCAF2YV1HK0hnC_S8MuqbU61KqGvu3MgCRIPujK-K6MoIyq_OxorwFVt6rdIgvsncA8VoObaHtnBcsruEboUJb78/s320/P1230695-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Trigpoint trimming back blackthorn on a fairly typical footbridge with a stile at one end.</td></tr>
</tbody></table>
<br />
We all went our different ways for the morning session, and through a quirk of our choices did not touch the area immediately to the west of Lees. In the afternoon we therefore set off as a group, divided into two parties, re-met again (see main picture) and split off again. This allowed us to cover most of the target paths in the area with the exception of one loop in the SW.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzPcJIdCEF3KQahbv-CHF-WaYyE12MNAAJRYrEBBVxamDf9iD7M4oXX8dYAFcHOt5bEzS3CgWYlX0bJsS9UJZxf7UnQY56Gyvob34MK1Bs7LO-rxmgj_Ftu1HROJjG_wlTmsSvfa-5G2MY/s1600/P1230467-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzPcJIdCEF3KQahbv-CHF-WaYyE12MNAAJRYrEBBVxamDf9iD7M4oXX8dYAFcHOt5bEzS3CgWYlX0bJsS9UJZxf7UnQY56Gyvob34MK1Bs7LO-rxmgj_Ftu1HROJjG_wlTmsSvfa-5G2MY/s320/P1230467-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Violet Bramble Rust (I didn't entirely neglect looking at the natural history).</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIPdkup4de8U-u3wvgG4xnVAxRuxapLNZT2XJ9gchkPUdpQgfF0ymwjPp-FBJsJZoN3FaAU7Nquwp05M_IEg6yNdtaOJ2nC6UqYTXwXCAsQJWhwwtrcfZcFeAbwvqBrjM7m5R3ME8gNLHV/s1600/P1230826-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIPdkup4de8U-u3wvgG4xnVAxRuxapLNZT2XJ9gchkPUdpQgfF0ymwjPp-FBJsJZoN3FaAU7Nquwp05M_IEg6yNdtaOJ2nC6UqYTXwXCAsQJWhwwtrcfZcFeAbwvqBrjM7m5R3ME8gNLHV/s320/P1230826-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A WWII pill box near Broad Chase</td></tr>
</tbody></table>
These joint footpath mapping sessions are undoubtedly useful, but its worth putting this in perspective: we mapped around 25 km of paths. Derbyshire as a whole has over <a href="https://osm.mathmos.net/prow/progress/derbs/">5,200 km</a>. So 5 of us spending about 12 hours walking paths only touched a half of one percent of the total. The real value is we can add much more detail to the area than the single mapper, and that detail can be enriched on return visits.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5t9u_NEBBHUkN9sh0FJCYjx4_6tJoXrHtRUPabjRY1chaaUOXY9qz42Lv8-MqSTAPE-xcFDe3VNClXF2ldAEocCP4pZNQdcPX6Vqi-lQ_sm2AkTv6Z1g11wrm6MFjvQfzB9BXEFrs2nDw/s1600/test_render4.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1132" data-original-width="1600" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5t9u_NEBBHUkN9sh0FJCYjx4_6tJoXrHtRUPabjRY1chaaUOXY9qz42Lv8-MqSTAPE-xcFDe3VNClXF2ldAEocCP4pZNQdcPX6Vqi-lQ_sm2AkTv6Z1g11wrm6MFjvQfzB9BXEFrs2nDw/s320/test_render4.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">An experimental rendering of OSM data for the area at 1:20k using QGIS.<br />
The aim is to show similar features to the OS 1:25k Explorer maps.</td></tr>
</tbody></table>
<br />
We are not done with this area there's perhaps twice as much more still to be surveyed, and the same is true of the National Forest area, S of Swadlincote, which we visited last year. However, it will not be long before the remaining hotspots are all in much less accessible parts of Lincolnshire. In a couple of years I'll need a different theme for these gatherings.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-57953883858247374912019-03-19T16:44:00.000+00:002019-03-19T16:44:48.559+00:00Social Housing polygons for England : generalisation from point data<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB6-kG4TWTegn4vEUkTG0JUBpPd9dNhsuxZapDitFZpxVQ__yJpDVg8hBz5s4nqIIkP-siJu_ErwX8b3a_H5JNMHsOrCH3Og1UmxhzGjWDB4tS8x7pD29HETEpqN50aoXazy4uWF7X_cf2/s1600/geograph-2253720-by-Jaggery.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="603" data-original-width="640" height="470" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB6-kG4TWTegn4vEUkTG0JUBpPd9dNhsuxZapDitFZpxVQ__yJpDVg8hBz5s4nqIIkP-siJu_ErwX8b3a_H5JNMHsOrCH3Og1UmxhzGjWDB4tS8x7pD29HETEpqN50aoXazy4uWF7X_cf2/s400/geograph-2253720-by-Jaggery.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-family: inherit;">A likely Addison Street candidate, Cefn Fforest, Blackwood</span><br />
<a href="http://creativecommons.org/licenses/by-sa/2.0/">cc-by-sa/2.0</a> -
© <a href="https://www.geograph.org.uk/profile/39302" title="View profile">Jaggery</a> - <a href="https://www.geograph.org.uk/photo/2253720">geograph.org.uk/p/2</a></td></tr>
</tbody></table>
John Boughton (<a href="https://twitter.com/MunicipalDreams">Municipal Dreams</a>) was recently <a href="https://twitter.com/MunicipalDreams/status/1104791725179052032">looking</a> for streets named after <a href="https://en.wikipedia.org/wiki/Christopher_Addison,_1st_Viscount_Addison">Christopher Addison</a> a pioneer ofpost-WWI <a href="https://en.wikipedia.org/wiki/Housing,_Town_Planning,_%26c._Act_1919">housing legislation</a> in Britain. It was easy to find all the roads with Addison in the name from OpenStreetMap, but much less easy to spot those which were likely to be named after him rather than other Addisons.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBT0C3EJxXMkRVtAi7WQlkokquJmWeVlMrmKa0VU0FvRBdmYEv3vfPk_pnhEpkUKKvgjV-krEt5CHQsAOnwddC2uLVh3b72S_58BWNRsEoEzG_GFem0yuaAHC8H20gmK0NmWv_x40YzyhR/s1600/nrosh_hulls_nwengland.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1132" data-original-width="1600" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBT0C3EJxXMkRVtAi7WQlkokquJmWeVlMrmKa0VU0FvRBdmYEv3vfPk_pnhEpkUKKvgjV-krEt5CHQsAOnwddC2uLVh3b72S_58BWNRsEoEzG_GFem0yuaAHC8H20gmK0NmWv_x40YzyhR/s400/nrosh_hulls_nwengland.png" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Merseyside, NW Cheshire & SW Lancs, showing areas of social housing.<br />
These are concave hull polygons derived from clusters of NROSH postcodes.</td></tr>
</tbody></table>
<br />
In order to reduce the number of roads to be searched one would ideally have information about when the buildings were built, and whether they were built to provide social (council) housing or not. There is <a href="https://www.owenboswarva.com/blog/post-hou1.htm">limited open data</a> on the overall age of British housing stock, but no direct information on the original developer of housing. Both are things which may ultimately be of interest to add to OSM, but it will be many years before such information has any utility on a national scale. Furthermore both are hard to check on the ground: at least for the typical mapper.<br />
<br />
It occurred to me that one national open data set, that of the National Register of Social Housing (hereafter <a href="https://data.gov.uk/dataset/cf3aac3e-f3f3-4564-9df4-099346246251/national-register-of-social-housing-nrosh">NROSH</a>), could be useful. This stopped being maintained in 2013, but provides addresses for millions of houses (approx 4 million in 350k postcodes) as of that time. Given that, since then, very few new homes have been added to social housing stock, and many have been removed, this can identify likely areas of social housing.<br />
<br />
The NROSH data therefore seemed a good place to get to grips with clustering in PostGIS, particularly as I had a specific objective in mind.<br />
<br />
<h4>
Clustering NROSH Data</h4>
Normally one sees clustering as a means of reducing clutter on webmaps, but it's only relatively recently that I realised that these techniques have great potential for performing various generalisations on detailed geographic data (particularly OSM, which tends to the detail rather than the general). <br />
<br />
NROSH data is only geocoded at the postcode level. There may be tens of addresses at an individual postcode or just one. At the outset I treated all postcodes equally ignoring the number of addresses. I was mainly concerned to aggregate them into coherent clusters. I grabbed some code from a GIS StackOverflow question & tweaked it very lightly:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">SELECT row_number() over () AS id,<br /> ST_NumGeometries(gc),<br /> gc AS geom_collection,<br /> ST_Centroid(gc) AS centroid,<br /> ST_MinimumBoundingCircle(gc) AS circle,<br /> sqrt(ST_Area(ST_MinimumBoundingCircle(gc)) / pi()) AS radius<br />FROM (<br /> SELECT unnest(ST_ClusterWithin(geom, 100)) gc<br /> FROM nrosh_pc_geo<br />) f </span></blockquote>
To my mind <a href="https://> ST_ClusterWithipostgis.net/docs/ST_ClusterWithin.html">ST_ClusterWithin</a> is still rather like magic. It groups individual postcodes which are within (in the example) 100 metres of each other. It returns all the clusters in an array, so this needs to be unnested to get each cluster. It is an aggregate function so other columns can be used for clustering (for instance local authority might have been a useful one if I'd included it in the imported data).<br />
<br />
I initially experimented with NG8 postcodes: this area of Nottingham (see my <a href="http://sk53-osm.blogspot.com/2019/03/mapping-roof-top-solar-panels.html">last post</a>) has many council estates built between the early 1920s to the 1970s (see Municipal Dreams <a href="https://municipaldreams.wordpress.com/2018/06/19/8951/">blog</a> for details). Trying with various distances for clustering I found 150 m worked pretty well. In London, and possibly other large cities with many postcodes on a road, this was too high.<br />
<br />
The cluster itself is a geometry collection of the original points. It is therefore trivial to calculate a hull for the collection. Fortunately these days ST_ConcaveHull does not break with target percents of less than 99%, and it produced sensible results.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcCtEZPP6MwVjxWxiJ8oSFPhiVSP2EUCAnz8N92RUt95abEx0-kfDXMOZd66QBQ8aqn88WxjHsjlksrPJEhol9gzbTGsMCX0vtwKLoZpZ2M0S3_YyswAU6vGTgzmcgpGBvM-u2IfV5VGse/s1600/woodside_egs.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="630" data-original-width="723" height="434" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcCtEZPP6MwVjxWxiJ8oSFPhiVSP2EUCAnz8N92RUt95abEx0-kfDXMOZd66QBQ8aqn88WxjHsjlksrPJEhol9gzbTGsMCX0vtwKLoZpZ2M0S3_YyswAU6vGTgzmcgpGBvM-u2IfV5VGse/s400/woodside_egs.png" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Odd-shaped polygons for <a href="https://en.wikipedia.org/wiki/Irby,_Merseyside">Irby</a> on the Wirral. Individual postcodes only have a few NROSH entries. Presumably both roads around the school were built as council housing, but most have now become privately owned.</td></tr>
</tbody></table>
I extended the code to the entire data set. I soon realised that it was excluding areas of social housing sharing a single postcode. As there are <a href="https://municipaldreams.wordpress.com/2018/04/24/nissen-petren-houses-not-obnoxious-and-the-people-would-be-delighted-to-pay-an-economic-rent/">some interesting examples</a> of rural council housing I wanted them in the overall data set too.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLblZUhvOGjhlF4H3nwihu_oI0EM0RYTJL0Yktg70PNhEc0HIvDKau8ZmAnOs2LWORIAbUEsUotOskfAPRZ0-ruNUYl9WVLgxYoJcexITNgTJGNMgibH220v4mG-8vGLjZh8skMzdrh4QO/s1600/geograph-4314325-by-Nigel-Mykura.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLblZUhvOGjhlF4H3nwihu_oI0EM0RYTJL0Yktg70PNhEc0HIvDKau8ZmAnOs2LWORIAbUEsUotOskfAPRZ0-ruNUYl9WVLgxYoJcexITNgTJGNMgibH220v4mG-8vGLjZh8skMzdrh4QO/s400/geograph-4314325-by-Nigel-Mykura.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div>
One of the more uunusal social housing forms, Stoford<br />
<a href="http://creativecommons.org/licenses/by-sa/2.0/">cc-by-sa/2.0</a> -
© <a href="https://www.geograph.org.uk/profile/14584" title="View profile">Nigel Mykura</a> - <a href="https://www.geograph.org.uk/photo/4314325">geograph.org.uk/p/4314325</a></div>
</td></tr>
</tbody></table>
<br />
My solution was simple: instead of using points I buffered them by 10 metres. This simply ensures that no data gets thrown away in subsequent steps. It does not mean that one gets a very accurate polygon when there are a very small number of postcodes in the cluster (less than 5 perhaps). If actual geocoded addresses are available then it will be possible to produce more accurate polygons. I haven't tested this, but this should be possible for any local authority where a decent number of addresses are mapped on OSM. In my local path there are several areas of Nottingham, Gedling, Broxtowe and Erewash which meet this requirement.<br />
<br />
<h4>
Overview of the Resultant Data </h4>
<br />
<iframe allowfullscreen="" frameborder="0" height="300px" src="https://umap.openstreetmap.fr/en/map/nrosh-derived-social-housing-polygons_305360?scaleControl=false&miniMap=false&scrollWheelZoom=false&zoomControl=true&allowEdit=false&moreControl=true&searchControl=null&tilelayersControl=null&embedControl=null&datalayersControl=true&onLoadPanel=undefined&captionBar=false" width="100%"></iframe><br />
<a href="http://umap.openstreetmap.fr/en/map/nrosh-derived-social-housing-polygons_305360">See full screen</a><br />
<br />
Throughout the Midlands, North-west England and parts of East Anglia the data looks pretty sensible. In general I've looked at places I know and checked that the edges of the polygons accord with what I know of housing patterns in those areas. For now I've tried to put a sample area for Notts, Derbys up on <a href="http://umap.openstreetmap.fr/en/map/nrosh-derived-social-housing-polygons_305360#12/52.9869/-1.2004">umap</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvJ9g9_wQbPIlXt_QQXArbvlJEUfzjcavqiMgAbFepB58RjB3q0aUKi59Ua6Hhoz1VOwn8sFJGMrYXxX5PpV6xXqnWOhf0tRt48-YSbvbbnrSnQgs37k_BaNI-g9SmwJk8QRD3cqqKt_hK/s1600/hampton.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="581" data-original-width="859" height="337" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvJ9g9_wQbPIlXt_QQXArbvlJEUfzjcavqiMgAbFepB58RjB3q0aUKi59Ua6Hhoz1VOwn8sFJGMrYXxX5PpV6xXqnWOhf0tRt48-YSbvbbnrSnQgs37k_BaNI-g9SmwJk8QRD3cqqKt_hK/s400/hampton.png" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Hampton, Hanworth & Teddington area of SW London.<br />
In general this is a pretty prosperous part of London. It does pick out some areas of social housing (e.g., near Apex Corner (A312/A junction), but the notion that most of Teddington is social housing is absurd. The <a href="https://maps.cdrc.ac.uk/#/geodemographics/imde2015/default/BTTTFFT/13/-0.3647/51.4353/">2015 Index</a> of Multiple Deprivation gives a better picture of this area. See additional notes at the foot of the blog.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSuvxc48gDvi2ilNv8sacVoQDpXwkzjyoCql81TcxSVlhkh2IWGOS4I9KH8XrwHGaD9iNiRggG5FXAidMIrdjnuo_Wr_3vycFP4ZFbrOkdJF8sCheRUDI-Znx7dUXaeMCNCw8vOhI6pxl/s1600/nw1_cluster_100.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="581" data-original-width="859" height="337" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBSuvxc48gDvi2ilNv8sacVoQDpXwkzjyoCql81TcxSVlhkh2IWGOS4I9KH8XrwHGaD9iNiRggG5FXAidMIrdjnuo_Wr_3vycFP4ZFbrOkdJF8sCheRUDI-Znx7dUXaeMCNCw8vOhI6pxl/s400/nw1_cluster_100.png" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">NW1 postcode. 150 m clusters with 100 m clusters overlaid.<br />
Reducing the cluster distance to 100 m greatly improves the elimination of false positives. Places like the Ossulston Estate between Euston & the British Library show clearly. I'm less convinced that this does not result in many false negatives</td></tr>
</tbody></table>
<br />
I have not scrutinised everywhere, but a few obvious oddities I've noticed:<br />
<ul>
<li>Sheffield & Redditch seem to be data deficient.</li>
<li>Areas in London are far too large to be usable, and even reducing the clustering distance does not make a massive difference (see images & commentary in the captions above).</li>
</ul>
<br />
The data is quite large so I havent yet been able to publish it somewhere readily accessible. In the meantime I can share it in various geoformats if you are interested. There's also scope to use IMD & the housing age stats to separate things out a bit more, but I'm just as interested in places which are now predominantly privately owned, but were built as social housing.<br />
<br />
I hope this data can be used for various things. In particular, I have long been interested in the possibility of finding <a href="http://sk53-osm.blogspot.com/2011/04/radburn-exemplar-or-nightmare.html">Radburn layouts</a> using OSM data. (Ian Waites <a href="http://instancesofachangedsociety.blogspot.com/2018/04/the-radburn-idea-3-private-sphere-true.html">has more </a>on these estates). A reduction in the total areas to search is always valuable. I'm sure other uses will occur to both social historians, and mappers. On the technical side I hope this might also provoke others to explore the potential for clustering in PostGIS: there's lot to learn. <br />
<ul>
</ul>
<h4>
Further Notes on Hanworth, Hampton & Teddington</h4>
I looked at this area because I lived in three places here during the 1980s and 1990s.<br />
<br />
Hanworth, the area in the London Borough of Hounslow had a lot of social housing, especially north of the Great Chertsey Road. S of the road housing was extremely mixed with small private speculative developments, older properties, infill, fields with horses grazing and so on. I bought a house in this area in 1986 which was built in the 1960s. Today this property appears in the NROSH data, so it has moved from the private to the social sector in the past 25 years. We bought it because the location was convenient (I used public transport and caught the bus at Apex Corner) and the house had been extended and was larger than equivalent properties we had looked at. It was sold in the early-1990s to an family of South Asian heritage, who probably bought it for similar reasons.<br />
<br />
Further south is the Hampton Nurserylands Estate. In the 1980s this was full of young professionals, many with young children. However, it changed demographically rather quickly. Many of the original buyers moved out to bigger houses within 2-3 years, and were replaced by older less-prosperous families. I remember looking at a flat here in 1993 and being staggered how much the area had changed in 3-4 years. The houses on the W side of Oak Avenue were social housing in the 1980s. Clearly these changes have continued.<br />
<br />
I can't really explain how Teddington has so many social housing postcodes. It is really one of the most prosperous places in Britain.<br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-23897880134969540282019-03-11T15:29:00.000+00:002019-03-11T15:29:04.648+00:00Mapping roof-top Solar PanelsWe've all noticed that solar panels have become increasingly frequent on the roofs of residential buildings. It's one of the things I have taken note of ever since I started contributing to OpenStreetMap. However, I had never tried to add any to OSM. Until now!<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Solar_panels_(6925093968).jpg" style="margin-left: auto; margin-right: auto;" title="Phil Sangwell from United Kingdom [CC BY 2.0 (https://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons"><img alt="Solar panels (6925093968)" src="https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Solar_panels_%286925093968%29.jpg/512px-Solar_panels_%286925093968%29.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Roof mounted solar PV panels on a semi-detached house.<br />
There are 18 panel modules (in 2 rows of 7 and one of 4). Each module consists of a 12 by 6 array of solar cells (see below for further discussion).<br />
P<a href="https://www.flickr.com/photos/12634019@N08/6925093968">hoto</a> by Phil Sangwell on Flickr via Wikimedia Commons. CC-BY-SA</td></tr>
</tbody></table>
A couple of days ago <a href="https://twitter.com/jack_kelly">Jack Kelly</a> suggested that perhaps we could use OSM to capture the presence of solar panels across the UK. A lively twitter discussion ensued.<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
How crazy would it be for national electricity grids to use <a href="https://twitter.com/openstreetmap?ref_src=twsrc%5Etfw">@OpenStreetMap</a> as *the* authoritative database of all solar photovoltaic systems?<a href="https://twitter.com/mclduk?ref_src=twsrc%5Etfw">@mclduk</a> <a href="https://twitter.com/alex_rogers_cs?ref_src=twsrc%5Etfw">@alex_rogers_cs</a> <a href="https://twitter.com/dekstop?ref_src=twsrc%5Etfw">@dekstop</a> <a href="https://twitter.com/EnergySRE?ref_src=twsrc%5Etfw">@EnergySRE</a></div>
— jack_kelly (@jack_kelly) <a href="https://twitter.com/jack_kelly/status/1103252121334484992?ref_src=twsrc%5Etfw">March 6, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
It seemed sensible to have a go at scoping what was involved. Over the past few months I've been improving the mapping of inter-war housing estates in Nottingham, with the current focus on the <a href="https://www.openstreetmap.org/#map=16/52.9762/-1.2009&layers=N">Aspley Estate</a> which contains perhaps 2,400 houses. In the course of visits and scrutinising aerial imagery I already knew that there were a fair number of roof-top photovoltaic (PV) panels already installed. Unfortunately on my last visit to get representative photos of the buildings I only caught one house in the background.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3wZRD2HEux_R-0SBFKRs4T2XJAPQ0giUjZybcaKkQhf3D4cZwWITENs4JRtoQ-y-gtkp5DmLgQiTsQ1igN58M3gHiytSZsk-xABBYZYVnOB-NbidtYBoUJnDFpZMFjg22cGcDMPael2GY/s1600/P1220547-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3wZRD2HEux_R-0SBFKRs4T2XJAPQ0giUjZybcaKkQhf3D4cZwWITENs4JRtoQ-y-gtkp5DmLgQiTsQ1igN58M3gHiytSZsk-xABBYZYVnOB-NbidtYBoUJnDFpZMFjg22cGcDMPael2GY/s400/P1220547-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Housing in the Aspley Estate: note the solar PV panel on house in background (<a href="https://www.openstreetmap.org/way/675059370">this one</a> on OSM).</td></tr>
</tbody></table>
Particularly useful is that the best quality imagery layer available for Aspley is <a href="https://binged.it/2tZ5fFM">Bing</a>, and it shows the panels clearly. In fact sufficiently clearly that I decided to map them as areas.<br />
<br />
In a relatively short time (15-20 minutes) I had found just over 200. Unfortunately I was also reminded that there were quite a few houses in the NW sector of the estate which I had not mapped, so I then spent a while adding houses and addresses, followed by fixing a lot of QA issues pointed out by the JOSM validator tool. Only then was I able to align the houses & solar panels, which took another hour.<br />
<br />
This was a little long winded for first-cut mapping, so the following morning I gave myself 30 minutes and searched through adjacent housing estates which I suspected would have a similar density of panels as I had found in Aspley.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnvmOPUSAHIyq0kCa1R5KDGrMQYMfkZo3ZBpxxnZyJsz8j8dOmHrtbANEizR53-sBVWs0yPO8bMphkYxl_bLBrfcP6Vt1t71wQB-FXRnDzFW4HSwwTtUjFePBl9fJuKuYFg_0-4RUs7T3c/s1600/DSC_0279-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnvmOPUSAHIyq0kCa1R5KDGrMQYMfkZo3ZBpxxnZyJsz8j8dOmHrtbANEizR53-sBVWs0yPO8bMphkYxl_bLBrfcP6Vt1t71wQB-FXRnDzFW4HSwwTtUjFePBl9fJuKuYFg_0-4RUs7T3c/s400/DSC_0279-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Solar Panels on new-build replacement housing at <a href="https://www.openstreetmap.org/#map=19/52.94187/-1.15844">Rutland Close</a>, The Meadows.<br />
A recent example of housing I've surveyed without paying particular attention to solar.</td></tr>
</tbody></table>
One reason for this is that social housing often has a much higher density of solar panel installation than private housing. Firstly the housing stock is often of similar or identical buildings under one ownership enabling economies of scale. Secondly, owners of social housing are much exercised by fuel poverty of their tenants: reducing fuel costs through providing electricity from solar power therefore has much to commend it. Thirdly, Nottingham City Homes, the at-length housing provider for Nottingham City Council, has a great deal of expertise in applying greener energy polices to their housing. Through an odd coincidence I saw <a href="https://twitter.com/EoinRedahan/status/1103572087032172544">a tweet </a>about what <a href="https://www.linkedin.com/pulse/spoils-sun-nottingham-story-eoin-redahan/?published=t">they have achieved</a> as I was doing my second scoping task. This set a target of around 4,500 panels to find in the city.<br />
<br />
With my <a href="https://www.openstreetmap.org/changeset/67882445#map=13/52.9912/-1.1885&layers=N">second run</a> just mapping each panel as a node I found 320, or just over 10 a minute. This was partially targeted because I pretty much restricted myself to examining areas of social housing. It therefore represents a rather efficient data acquisition rate.<br />
<br />
Jack <a href="https://twitter.com/jack_kelly/status/1103648646120292353">extrapolated</a> this to the whole country mapped in 5 days by 33 people working full-time. This is of course highly optimistic because I was mapping areas I know well from having being mapping them for 10 years on OSM. However, the OSM community is full of people with detailed knowledge of their local areas, so this ought to apply or many parts of the country. even if it was 5-10 times as much effort, say 1 a minute, 100 people mapping for a couple of hours a week for a quarter might find 150k. This suggests solar panels may be a good subject for a Quarterly Project.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS136LogR_ZY9Z6LPmzrInDG4thQWF_ziBhfFJykCcY_PARRtEbFufocKYEO5jJBtm8EVeVS7ZECSDLV47fjSJPkH6CdgFdsmLjOGspvLjrI2bvdwWAcBvuS7uZJUM_n08kz-8fYoN43xA/s1600/Screenshot+%252829%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="714" data-original-width="748" height="305" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS136LogR_ZY9Z6LPmzrInDG4thQWF_ziBhfFJykCcY_PARRtEbFufocKYEO5jJBtm8EVeVS7ZECSDLV47fjSJPkH6CdgFdsmLjOGspvLjrI2bvdwWAcBvuS7uZJUM_n08kz-8fYoN43xA/s320/Screenshot+%252829%2529.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Solar PV panels mapped in Nottingham, via <a href="http://overpass-turbo.eu/s/GMD">Overpass Turbo</a></td></tr>
</tbody></table>
<br />
Now, 48 hours after I started, I have added 1760 solar panels to OSM in Nottingham. It's time to summarise what I have learnt. In no particular order:<br />
<ul>
<li>All available imagery layers need to be searched. <a href="https://twitter.com/cllrsamwebster/status/1104114977504481287">New installations</a> are occurring all the time and it's unlikely that the better quality imagery layers will be recent enough to enable adequate coverage.</li>
<li>Newer imagery, such as the Digital Globe layers can be quite grainy & hard to interpret. However once panels have been spotted it is usually possible to then find many more.</li>
<li>The huge variability in available imagery is likely to make any attempt to use machine learning to identify targets is likely to be fraught. I would also expect things like glass roofed extensions would generate many false positives.</li>
<li>Knowing where panels are likely to be installed helps a great deal: both at the neighbourhood and building level. Christian Quest's OpenSolarMap used some crowd-sourced information from aerial photos to train system to identify buildings in France with potential for installation of photovoltaic panels. Such information for the UK could reduce the total number of buildings needing to be inspected.</li>
<li>Larger detached houses with solar panels are very difficult (impossible?) to pick out from aerial imagery. Shadows from chimneys, and changing roof lines obscure the presence of the panels.</li>
<li>Mapping panels as nodes is the best approach initially. I used ID which has a suitable preset (and checked what others had already done, for instance <a href="https://www.openstreetmap.org/user/brianboru">brianboru</a> around Birmingham). Thereafter I just copied the original node.</li>
<li>Adding a tag to show that they are roof-mounted is useful (particularly if the building has not been mapped yet). I've used generator:location=roof. Indicating domestic use might also be helpful. The basic tags I used are also used for complete solar farm installations & clearly it is important to distinguish them. (I've subsequently learnt that generator:place=roof is the established tag).</li>
<li>Many installations are sufficiently clear on aerial imagery to allow estimation of the number of panel modules involved. Virtually all the ones at Aspley are 2 rows of 5 modules. Unfortunately I don't know the exact module size, but they are probably 10 or 12 by 6 cells. Tagging the module array explicitly is probably better than guesstimating the area (as I have done).</li>
<li>If module size is known (see top photo) the array area can be calculated directly. Each <a href="https://en.wikipedia.org/wiki/Solar_cell">solar cell</a> is likely to be 156 mm square, so a 10 by 6 array will be 1.56 x 0.96 m (1.46 sq m).</li>
<li>Cell size, module size and number of modules allow optimal power rating to be estimated. I think these arrays of roughly 15 sq m are around 3500-3700 W.</li>
<li>Adding compass orientation of the array in degrees, and angle from the horizontal would also be helpful for using the data for estimating likely power output. (Both could be derived from simple 3D building tags, but adding these is much more complex).</li>
<li>The last few items (no. of modules, module size, area, power rating, orientation & angle) represent data which can be added iteratively.</li>
<li>It's worthwhile surveying at least some of those added from aerial imagery to capture other information.</li>
<li>Even if the panel is mapped as an area, most of these tags are still useful as it is unlikely that enough information will be present on the underlying buildings to derive them.</li>
<li>Surprisingly few public buildings, education establishments or industrial buildings have solar panel installations. I've only noticed a few on buildings of Derby Hall, a hall of residence at the University of Nottingham, and a couple of warehouses in Bulwell.</li>
</ul>
Other than quickly looking for whether anyone had mapped roof-mounted solar panels in the UK, I haven't looked at activity in other countries. There may be places with a more developed approach to mapping and tagging.<br />
<br />
A couple of caveats:<br />
<ul>
<li>solar panel distribution is likely to be very patchy; </li>
<li>aerial imagery may not be good enough to pick out panels, or recent enough (many of the Nottingham panels have been installed since 2014). </li>
</ul>
<div>
To help judge what the latter point may mean I provide below a selection of available aerial imagery of various locations in Nottingham, and Basingstoke (Hampshire Council Open Data). The latter includes false colour infra-red (FCIR). </div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim7KUVhlG88rwv5HqhUXubm15YZzIZqZDzfH1FUz5VZ07rhhCKGskr5sU03Wq1ANaVrKSscca2C7XHooIzlMT-6jtAtzm32ZcPzYfYEw-1WcjU84TtgjAppyNWCahBKCDhwC0QWNUWdpvu/s1600/Screenshot+%252830%2529-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim7KUVhlG88rwv5HqhUXubm15YZzIZqZDzfH1FUz5VZ07rhhCKGskr5sU03Wq1ANaVrKSscca2C7XHooIzlMT-6jtAtzm32ZcPzYfYEw-1WcjU84TtgjAppyNWCahBKCDhwC0QWNUWdpvu/s400/Screenshot+%252830%2529-1.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Aspley Estate (Bing Imagery), roughly <a href="https://www.openstreetmap.org/#map=18/52.97875/-1.20159">here</a>.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibdlmYL_iP0p7B6ktV2V9yV8CnxZKhik8cV4cRn6nwD7106sVt0SDYLbbyke-CVdHRA-gv_A37eLOejelSs3VxN04x2q3Xj9fTgZ9JvYfiKr2cgxa5YhpZI6rrqu-SgUC060oz3wL3tH1B/s1600/Screenshot+%252831%2529-2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibdlmYL_iP0p7B6ktV2V9yV8CnxZKhik8cV4cRn6nwD7106sVt0SDYLbbyke-CVdHRA-gv_A37eLOejelSs3VxN04x2q3Xj9fTgZ9JvYfiKr2cgxa5YhpZI6rrqu-SgUC060oz3wL3tH1B/s400/Screenshot+%252831%2529-2.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Aspley Estate (Bing Imagery) roughly <a href="https://www.openstreetmap.org/#map=19/52.97721/-1.20489">here</a>.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2mXQ2vaEnidS10ZSX4qHaYvjzT6eK-6vtdQD9cpBFYytkqRCNnWgo_7zT8jCouHDgP6xOwHRg7APpRpn6q6S-ZVhGB7wbAtZkBEXah-PMylhDIGeL-HoKWzUFV-WwPWgpulh7oqNnrjYB/s1600/Screenshot+%252832%2529-3.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2mXQ2vaEnidS10ZSX4qHaYvjzT6eK-6vtdQD9cpBFYytkqRCNnWgo_7zT8jCouHDgP6xOwHRg7APpRpn6q6S-ZVhGB7wbAtZkBEXah-PMylhDIGeL-HoKWzUFV-WwPWgpulh7oqNnrjYB/s400/Screenshot+%252832%2529-3.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="background-color: transparent; color: black; display: inline; float: none; font-family: "times new roman"; font-size: 12.8px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Aspley Estate (Digital Globe Standard Imagery) location as <a href="https://www.openstreetmap.org/#map=19/52.97721/-1.20489">above</a></span></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3yxRggcIrxBM9jwGOBaWLyMzKq-wVk01XVH8pECvQsFLTCg2ZZa3TfTYBb9GarijEOTPIGDXMDd2Ogd6GUiVsZo3H8xJjsUWXSZUJR9tpIj1VGYrHkGUdyrbzGdn2YScaXJMcbmnWrTQZ/s1600/Screenshot+%252833%2529-4.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3yxRggcIrxBM9jwGOBaWLyMzKq-wVk01XVH8pECvQsFLTCg2ZZa3TfTYBb9GarijEOTPIGDXMDd2Ogd6GUiVsZo3H8xJjsUWXSZUJR9tpIj1VGYrHkGUdyrbzGdn2YScaXJMcbmnWrTQZ/s400/Screenshot+%252833%2529-4.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="background-color: transparent; color: black; display: inline; float: none; font-family: "times new roman"; font-size: 12.8px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Aspley Estate (Digital Globe Premium and ESRI World Imagery) location as </span><a href="https://www.openstreetmap.org/#map=19/52.97721/-1.20489" style="-webkit-text-stroke-width: 0px; background-color: transparent; color: #0066cc; font-family: Times New Roman; font-size: 12.8px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: center; text-decoration: underline; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">above</a></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjM-Q3UQtAo9gY2hVcTchTetGaSpQJjP6w35BtpD8712GnlKP7QnZk0dz6TglozhGNOx4HONKgc09Uf9SxVCFqpb4JcjvvX_1iqmFYCX220MacNVETy3j8QXm4NTFA75SFtDrch9v5s3YmQ/s1600/Screenshot+%252834%2529-5.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjM-Q3UQtAo9gY2hVcTchTetGaSpQJjP6w35BtpD8712GnlKP7QnZk0dz6TglozhGNOx4HONKgc09Uf9SxVCFqpb4JcjvvX_1iqmFYCX220MacNVETy3j8QXm4NTFA75SFtDrch9v5s3YmQ/s400/Screenshot+%252834%2529-5.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Broxtowe Lane (Bing Imagery), about <a href="https://www.openstreetmap.org/#map=18/52.97756/-1.20967">here</a>.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNkNuLCcGiCDUurJ2Ylf8LS7Iuhmo_9VYa5rqNnqAOKmRC7ghHtA2svnZav2tWNUuVSgv0B406Azr2Yvdt5PhOD-KZjfn_3xbXi4UbF6fGM0jP_yd7ATk51d6id_lEyMGn_u4zy-T3Hv3R/s1600/Screenshot+%252835%2529-6.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNkNuLCcGiCDUurJ2Ylf8LS7Iuhmo_9VYa5rqNnqAOKmRC7ghHtA2svnZav2tWNUuVSgv0B406Azr2Yvdt5PhOD-KZjfn_3xbXi4UbF6fGM0jP_yd7ATk51d6id_lEyMGn_u4zy-T3Hv3R/s400/Screenshot+%252835%2529-6.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Broxtowe Lane (Digital Globe Premium Imagery), same <a href="https://www.openstreetmap.org/#map=18/52.97756/-1.20967">location</a> as above. Obviously newer as a solar panel can be made out on the terrace in the centre. Note on the next terrace down a dark area on the roof. This does not appear to have the same visual appearance as other solar panels, so may be a solar hot water system. </td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEin-rqIm105PNWJdesNX1vXlyicYoLVTUeNdOdRVLl65Rb4gxbufNsIN7HLSCyV2b2OE8uRsfdkNCSGO0zn3PIughyphenhyphenxaZtswklm30pewXvAOp-B2VZFPkKyq9tsIBiqcOhGwSk6Ov2M0Iac/s1600/Screenshot+%252836%2529-7.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEin-rqIm105PNWJdesNX1vXlyicYoLVTUeNdOdRVLl65Rb4gxbufNsIN7HLSCyV2b2OE8uRsfdkNCSGO0zn3PIughyphenhyphenxaZtswklm30pewXvAOp-B2VZFPkKyq9tsIBiqcOhGwSk6Ov2M0Iac/s400/Screenshot+%252836%2529-7.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Broxtowe Lane, as above (Digital Globe Standard imagery)</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk7i_NNxzoqjfP3bKgXzTVxF1qDEwEDFmFzySKNeBJkQpLsRPJcO2pIH_CpDxZLmnHaOCzjR7ZRcItRz1SyzDqU2kkAP9vdUvQFNY6FyGch9er9BlgokhPBxYuuQ4W-AH54QC6gq7aa47l/s1600/Screenshot+%252837%2529-8.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjk7i_NNxzoqjfP3bKgXzTVxF1qDEwEDFmFzySKNeBJkQpLsRPJcO2pIH_CpDxZLmnHaOCzjR7ZRcItRz1SyzDqU2kkAP9vdUvQFNY6FyGch9er9BlgokhPBxYuuQ4W-AH54QC6gq7aa47l/s400/Screenshot+%252837%2529-8.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Deptford Crescent <a href="https://www.openstreetmap.org/#map=18/52.97756/-1.20967">area</a>, Highbury Vale (Bing imagery). Area bottom right is Highbury Hospital.<br />
No solar panels visible</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6ZYpHbpAMA69skQk1dJ4x_sxZUwGICseQHOwiyItQqwYITFZXkBDnDmdsViObLs2fxd9OZaEwLuJJg4znUk4mUKCPZaQfCxxxmtlcsROxmagLab0hTPp_pfaSmhwlOht9btVooR_wiI7U/s1600/Screenshot+%252838%2529-9.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6ZYpHbpAMA69skQk1dJ4x_sxZUwGICseQHOwiyItQqwYITFZXkBDnDmdsViObLs2fxd9OZaEwLuJJg4znUk4mUKCPZaQfCxxxmtlcsROxmagLab0hTPp_pfaSmhwlOht9btVooR_wiI7U/s400/Screenshot+%252838%2529-9.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Deptford Crescent (ESRI World Imagery)</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVGHpi1U43-EsWbyOBhPyMB9Rh8hqXAZT6MhzJJnNbtTrShuRPNyRKLHLgAIQR0XV6XvMdo7oHLUrbKkCnANINLaJyUNIJSBQrZpT1F0zuHcAF69eHTRvGHwdBgt37eoknxAH4lBcl7ANs/s1600/Screenshot+%252839%2529-10.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVGHpi1U43-EsWbyOBhPyMB9Rh8hqXAZT6MhzJJnNbtTrShuRPNyRKLHLgAIQR0XV6XvMdo7oHLUrbKkCnANINLaJyUNIJSBQrZpT1F0zuHcAF69eHTRvGHwdBgt37eoknxAH4lBcl7ANs/s400/Screenshot+%252839%2529-10.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Deptford Crescent (Digital Globe Standard Imagery)</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCXPuIGQG1Lec8B0KTu-tVurQ2u0_UdlGoKW1BRBU2CwJn6agqc5-qnrf5wPVmOmmqnyoQfRHxZ5Q53xEKwCVivmIm0haUHZPgzqjYeECHXffLXm57h7v_K3rcObtxH4R8gidh2l_gaxFK/s1600/Screenshot+%252840%2529-11.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCXPuIGQG1Lec8B0KTu-tVurQ2u0_UdlGoKW1BRBU2CwJn6agqc5-qnrf5wPVmOmmqnyoQfRHxZ5Q53xEKwCVivmIm0haUHZPgzqjYeECHXffLXm57h7v_K3rcObtxH4R8gidh2l_gaxFK/s400/Screenshot+%252840%2529-11.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.openstreetmap.org/#map=19/52.99754/-1.17034">Astrid Gardens</a>, Bestwood Estate (ESRI Imagery).<br />
Just occasionally panels have very strong reflections as here.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZVg4zYZek-ZL8PyuETFkrnK3nxkLSffDN26z_9JSr8GsLmBE3xyEMUTzEuJ_mCWTM5MKmm2FXf_7WWCd7a9QYKgoHwkLjsoiVDnbxe_ONxVMZ8kAQ59J0IlZdtl1XyQQ0KmEX6MAJ0uA2/s1600/Screenshot+%252841%2529-12.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZVg4zYZek-ZL8PyuETFkrnK3nxkLSffDN26z_9JSr8GsLmBE3xyEMUTzEuJ_mCWTM5MKmm2FXf_7WWCd7a9QYKgoHwkLjsoiVDnbxe_ONxVMZ8kAQ59J0IlZdtl1XyQQ0KmEX6MAJ0uA2/s400/Screenshot+%252841%2529-12.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="background-color: transparent; color: black; display: inline; float: none; font-family: "times new roman"; font-size: 12.8px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Astrid Gardens, Bestwood Estate (Digital Globe Standard Imagery). </span></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl4ZDkIZfOV9ZejptMgNltEz6GQ2Z3e5-_4wczvyk95GeebQyRGaCwjIMviD69ELJKTObg8f84o6dsIf_q5CUtaMCYZBrr-H0VAG_Ju7tc3jkCX3lMWWC9ra6RXYIliuK5gMhKXSeObLMb/s1600/Screenshot+%252842%2529-13.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgl4ZDkIZfOV9ZejptMgNltEz6GQ2Z3e5-_4wczvyk95GeebQyRGaCwjIMviD69ELJKTObg8f84o6dsIf_q5CUtaMCYZBrr-H0VAG_Ju7tc3jkCX3lMWWC9ra6RXYIliuK5gMhKXSeObLMb/s400/Screenshot+%252842%2529-13.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="background-color: transparent; color: black; display: inline; float: none; font-family: "times new roman"; font-size: 12.8px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; text-align: center; text-decoration: none; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Astrid Gardens, Bestwood Estate (Bing Imagery). </span></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_HalTe7QinT8P0lNGxvQioIKjk32USgostGZCF4uKwihAoo9fP5ee5lYKaa5nQwgShfOzFVxRMW7JPziDdmKmPeegJlv1-gP84QTW-5WlyPaQOtvP7PGzoZR7nKK5wJsUgc_69e3Rggya/s1600/Screenshot+%252843%2529-14.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_HalTe7QinT8P0lNGxvQioIKjk32USgostGZCF4uKwihAoo9fP5ee5lYKaa5nQwgShfOzFVxRMW7JPziDdmKmPeegJlv1-gP84QTW-5WlyPaQOtvP7PGzoZR7nKK5wJsUgc_69e3Rggya/s400/Screenshot+%252843%2529-14.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.openstreetmap.org/#map=19/52.96355/-1.23825">Bilborough Estate</a> (Digital Globe Standard Imagery)<br />
Note the panel lower right which is pretty hard to pick out.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_uQGIiUlb7MkAsBbdc1KHVMr9uFNeFQtkEjwwsYHBOwZIdbjFQWsBPcPtBZgW4Z5RSn_fy1Ai9_0gOJ5ogdBiBH0AvD4L1hUsdUTpMeb3Rzn3o9s8T36UCoScTPNXtNQa4oYsqYYpei5M/s1600/Screenshot+%252846%2529-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_uQGIiUlb7MkAsBbdc1KHVMr9uFNeFQtkEjwwsYHBOwZIdbjFQWsBPcPtBZgW4Z5RSn_fy1Ai9_0gOJ5ogdBiBH0AvD4L1hUsdUTpMeb3Rzn3o9s8T36UCoScTPNXtNQa4oYsqYYpei5M/s400/Screenshot+%252846%2529-1.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.openstreetmap.org/way/21349688#map=17/51.24588/-1.10726">Britten Road</a>, Basingstoke (Hampshire false-colour Infra-red imagery)<br />This estate has quite a few solar hot water installations but only a few solar PV. The hot water ones are smaller and less obviously modular.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTTPbCIHvHbJ01UDa4Bh4dgBj4bn7UscbDjB2N4rbbwMX8Gsq4x8XPnGq6fnIs42GUW1r1emF-CNRTYcez93AgVZpkpsY9wvCcL1JOZ_lnpv7V6v0vuHyztkigq1BH_TmrkKYbnu3xgUsJ/s1600/Screenshot+%252847%2529-2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="576" data-original-width="768" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTTPbCIHvHbJ01UDa4Bh4dgBj4bn7UscbDjB2N4rbbwMX8Gsq4x8XPnGq6fnIs42GUW1r1emF-CNRTYcez93AgVZpkpsY9wvCcL1JOZ_lnpv7V6v0vuHyztkigq1BH_TmrkKYbnu3xgUsJ/s400/Screenshot+%252847%2529-2.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Britten Road, Basingstoke (Hampshire visible spectrum RGB imagery)</td></tr>
</tbody></table>
<br />
<div>
So if there are any other takers I suggest this is a suitable topic for a future <a href="https://wiki.openstreetmap.org/wiki/UK_Quarterly_Project">UK Quarterly Project</a>, perhaps in association with <a href="https://twitter.com/jnicho02">Jez Nicholson</a>'s current interest in <a href="https://wiki.openstreetmap.org/wiki/Renewable_energy_in_the_United_Kingdom">solar & wind farms</a>.</div>
<div>
<br />
Lastly I'd like to thank Jack Kelly & Dan Stowell for comments & ideas whilst mapping & writing this up.</div>
<br />
<br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com3tag:blogger.com,1999:blog-2082422783692631078.post-79846443206554160682018-10-14T13:40:00.003+01:002018-10-21T12:06:20.477+01:00Creating MapMate Picture Files for Ireland<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/26481919032/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="H34 Base Map for Map Mate Hillshaded"><img alt="H34 Base Map for Map Mate Hillshaded" height="500" src="https://farm2.staticflickr.com/1505/26481919032_470a09bc5a.jpg" width="461" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">H34 West Donegal: Vice County raster map suitable for MapMate<br />
Source data: (c) OpenStreetMap contributors; hill shading NASA SRTM via <a href="http://viewfinderpanoramas.com/">viewfinderpanoramas.com</a></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script><br />
One of the things I'd always planned to do once I had a half decent <a href="http://sk53-osm.blogspot.co.uk/2015/10/irish-vice-counties-creation-of.html">data set</a> for the <a href="https://en.wikipedia.org/wiki/Vice-county">Irish Vice Counties </a>was to create detailed raster maps for each of them. This is mainly because I've found using such raster maps useful for my own biological recording using <a href="http://www.mapmate.co.uk/">MapMate</a> software. They make choosing the correct recording location much easier, and reduce the hassle in producing more attractive (and communicative) outputs. I've detailed the principles behind the creation of such maps <a href="http://sk53-osm.blogspot.co.uk/2013/02/mundane-cartography-mapping-vice.html">here</a> in the past.<br />
<br />
With the first such map I produced I struggled to get it to align properly with the Irish Grid displayed with MapMate. I could upload <a href="https://en.wikipedia.org/wiki/MapInfo">MapInfo</a> <span style="font-family: "courier new" , "courier" , monospace;">.MIF</span> files which would align, but these offered nothing like the degree of detail I want to show. Furthermore the import process with my copy of MapMate only seems to work with polygons. I tried a variety of ways: mainly trying to tweak the <span style="font-family: "courier new" , "courier" , monospace;">.TAB</span> file format which I'd used successfully with files for Great Britain.So I put the idea to one side.<br />
<br />
Very recently, prompted by Julia Nunn, VC recorder for <a href="https://en.wikipedia.org/wiki/MapInfo">County Down</a>, I re-looked at the problem. Once again I was getting nowhere and I was on the verge of seeking expert help from <a href="https://twitter.com/ManAboutCouch">Richard Cantwell</a> on the arcana of <a href="https://en.wikipedia.org/wiki/MapInfo_Professional">MapInfo</a>'s formats, many of which date back to the 1980s. (Richard works professionally with MapInfo data a lot and has written some very informative articles available from his firm's website.)<br />
<br />
<b>Note</b>. Much of this post will mainly be of interest to users of MapMate. However, towards the end I digress into geonerd territory when discussing the pros and cons of MapMate's technical choices about projections. Also please note that has been sitting in my draft folder for a while before being published. <br />
<br />
<br />
<a name='more'></a>I soon realised that there was another approach I had not tried.<br />
<br />
MapMate allows one to 'calibrate' a raster image by clicking on two locations and entering the grid co-ordinates. I have always found this a bit chancy and tedious to set up, and even then one is still often a pixel or two out. Even with this approach I was failing: I could get a raster map registered but it would be to the British not Irish Gird. Whilst doing this I finally remembered that MapMate will save a given map as a bitmap with an associated calibration file. This finally gave me proper insight as to how to crack the problem.<br />
<br />
What I did was create a map of County Down in MapMate using its own internal data, and then saved this as a calibrated bit map. The calibration files are rather simple:<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"> <mapmate><br /> <cal><br /> <name>test.jpg</name><br /> <timestamp><timestamp><br /> <copyright></copyright><br /> <units>m</units><br /> <datum>WGS84-modified</datum><br /> <projection>Local</projection><br /> <coodtype>EN</coodtype><br /> <rotation>0</rotation><br /> <type>raster</type><br /> <xo>-160000</xo><br /> <yo>200000</yo><br /> <xi>-50000</xi><br /> <yi>290000</yi><br /> <xpixels>4400</xpixels><br /> <ypixels>3600</ypixels><br /> </cal><br /></mapmate></span></span><br />
<br />
The key parameters being the bottom right and top left in MapMate's internal co-ordinate system, and the size of the raster file in pixels. To make life easier for myself I took the initial MapMate map and used the option to re-frame the map to the 10km grid. I could then write down both the grid co-ordinates and the internal MapMate co-ordinates for the map bounds. I did this for all Irish Vice Counties (although MapMate wouldn't snap to the grid for a couple, and its bounds for Fermanagh <br />
were broken), and recorded all the values in a <a href="https://github.com/SK53/Irish-Vice-Counties/blob/master/vice_counties/MapMate%20Helpers/vc_image_dimension.ods">spreadsheet</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/26559890885/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="H38 County Down : MapMate test map"><img alt="H38 County Down : MapMate test map" height="500" src="https://farm2.staticflickr.com/1442/26559890885_c49048db9d.jpg" width="450" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">First MapMate test picture map for County Down<br />
Data: (c) OpenStreetMap contributors</td></tr>
</tbody></table>
Now using QGIS all I had to do was create a raster file with exactly the same bounds as the MapMate map. By setting the print option to 254 dpi I get as close as I can to 100 dots per cm which makes for easy computation of image sizes. I scaled maps to be 1:250k which is convenient both in terms of detail and effective for all vice counties. The largest dimension was around 5600 pixels, but most rasters were roughly 4000x4000. Once I'd tested out a quick draft of County Down, I then produced rather more considered maps of the 2 Donegal vice counties. These were made by using <a href="http://wiki.openstreetmap.org/wiki/Overpass_turbo">Overpass-Turbo</a> queries to get specific features within a vice county and saving them as geojson files.<br />
<br />
<h3>
Hillshading </h3>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/stml/3693067553/in/photostream/" style="margin-left: auto; margin-right: auto;" title="You can find all these places on this map because it shows a much larger area on a much smaller scale than the previous maps."><img alt="You can find all these places on this map because it shows a much larger area on a much smaller scale than the previous maps." height="375" src="https://farm3.staticflickr.com/2531/3693067553_b13de288a4.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Collins Progressive Atlas page 12 : my introduction to hill-shading, contours & hypsometric tints.<br />
I was about 6 years & 4 months of age.</td></tr>
</tbody></table>
I also wanted to add <a href="https://en.wikipedia.org/wiki/Terrain_cartography#Shaded_relief">hill shading</a> and <a href="https://en.wikipedia.org/wiki/Hypsometric_tints">hypsometric tints</a>. These help a bit in quickly identifying locations for data entry, but are of much greater value for someone looking at the distribution of a plant. Ultimately I may produce multiple versions suitable for different contexts: for instance the recently published <a href="http://floraofderbyshire.blogspot.co.uk/">Derbyshire Flora</a> uses geology, or hypsometry as a background where this informs interpreting the distribution of a plant.<br />
<br />
Another motivation for doing this lies way back in my childhood. I was given a book at Christmas when I was six called <i>Collins Progressive Atlas</i>. I still have it, and flipping through its pages I'm amazed at its ambition in teaching map reading & many of the principles of cartography to young children. The image above shows on a 2-page spread it introduces some rather sophisticated concepts. (An aside: I'd love a way to do something like the 3 opening page spreads for anywhere in the world. OSM is a perfect starting point).<br />
<br />
To do this I downloaded raster DEM files from <a href="http://www.viewfinderpanoramas.org/dem3.html">ViewFinder Panormas</a>, and processed <a href="http://www.qgistutorials.com/en/docs/working_with_terrain.html">them in QGIS</a> to produce hill-shading, contours and tints. I did so pretty much using the standard defaults : these can be improved upon as can be seen from the work of folk like <a href="https://twitter.com/gravitystorm">Andy Allan</a> (<a href="http://www.opencyclemap.org/?zoom=9&lat=53.14281&lon=-6.5082&layers=000B0">Thunderforest</a>) and <a href="https://twitter.com/richardf">Richard Fairhurst</a> (<a href="http://cycle.travel/">cycle.travel</a>).<br />
<br />
Having done this I worked through all 40 of the Vice Counties: the images are currently available on <a href="https://www.flickr.com/search/?sort=date-taken-desc&safe_search=1&tags=praeger&user_id=11264009%40N00&view_all=1">Flickr</a>, but I have <strike>will eventually add</strike> added them on <a href="https://github.com/SK53/Irish-Vice-Counties/tree/mapmate-files/mapmate">github</a> along with the relevant calibration files. Thes should be usable directly in MapMate if all three files with the same base name are copied to the same directory.<br />
<br />
<h3>
MapMate's Co-ordinate system</h3>
The last thing I wanted to do was understand MapMate's internal co-ordinate system.<br />
<br />
To do this I created two geolocated images which were 100 km square (basically the Irish Grid Square C and the British Grid Square composed of half each of <a href="http://www.cucaera.co.uk/grp/&map=road&vcs=true&graticule=true&nsgrids=false&zoom=8&lat=54.90962660253353&lon=-6.722087617611318?refs=NW&map=road&vcs=true&graticule=true&nsgrids=false&zoom=8&lat=54.90962660253353&lon=-6.722087617611318">NW </a>& <a href="http://www.cucaera.co.uk/grp/&map=road&vcs=true&graticule=true&nsgrids=false&zoom=8&lat=54.90962660253353&lon=-6.722087617611318?refs=NW&map=road&vcs=true&graticule=true&nsgrids=false&zoom=8&lat=54.90962660253353&lon=-6.722087617611318">NR</a> I made sure I had both grids visualised in the images. I then created MapInfo calibration files based either on bottom-left (corresponding to Irish Grid) or top-right co-ordinates (assumed British Grid).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEobX1VQcZUQSufWw7Z3Vl8YUUGJikyGoueECQ2x5H8Jn1j93QBT-dVSixW1bFpWuqikD1zYqnr80XnV8QttFtOc2IfJg4BJCHr6bwKH1EQC47xuBK9Cx6iyxf83s5QyKlbGTay9vE1ecS/s1600/mm_test_osgrid.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEobX1VQcZUQSufWw7Z3Vl8YUUGJikyGoueECQ2x5H8Jn1j93QBT-dVSixW1bFpWuqikD1zYqnr80XnV8QttFtOc2IfJg4BJCHr6bwKH1EQC47xuBK9Cx6iyxf83s5QyKlbGTay9vE1ecS/s320/mm_test_osgrid.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Using QGIS I created raster images with both Irish (grey-brown) and British (blue) grids.</td></tr>
</tbody></table>
I then loaded them into MapMate, and overlayed these with the coastline data which comes with MapMate.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9gV3x5JOi80PWb5JmJML_PhEDokOe3gM_UXzMAcIJ9rLsEy8un-4YfNb6Z9LUdqcEPOGP00XeSXgkjdXbk8sqmfdZxWuKDJdVvdbnwLHe6Po2zrJnYBnw_FoaU6nPb28UrhP2uyr2oLeS/s1600/irish_grid_eg.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9gV3x5JOi80PWb5JmJML_PhEDokOe3gM_UXzMAcIJ9rLsEy8un-4YfNb6Z9LUdqcEPOGP00XeSXgkjdXbk8sqmfdZxWuKDJdVvdbnwLHe6Po2zrJnYBnw_FoaU6nPb28UrhP2uyr2oLeS/s320/irish_grid_eg.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">100 km sq. British Grid (from NW05 to NR49)</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn06V3lWP8k4oP2Ze8i_kNeE-7zLIwbMMV493Hw1LjW2afeHDcI2HWIlPqwwdi8_jIHiRRg8_xEwPj2bWx-js1-rBS2787nCEeyQQfzLNvsFF6mTXsebMZIfPP2-4ug7h-bSJfX8P1kfnH/s1600/british_grid_eg.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhn06V3lWP8k4oP2Ze8i_kNeE-7zLIwbMMV493Hw1LjW2afeHDcI2HWIlPqwwdi8_jIHiRRg8_xEwPj2bWx-js1-rBS2787nCEeyQQfzLNvsFF6mTXsebMZIfPP2-4ug7h-bSJfX8P1kfnH/s320/british_grid_eg.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">100km Irish Grid sqaure C. British Gird in blue.</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
As was already clear, MapMate uses a kludge. Irish Grid co-ordinates are mapped internally onto squares of the British Grid. It should be possible to work out which 10km squares belong to which grid system along the boundary (at least I assume the border is aligned on 10km grid lines).<br />
<br />
MapMate internally uses an Access database without any geospatial features. It also only allows its notional recording units to be grid squares (AFAIK, of any size from 1m to 10km), which helps because most geospatial calculations can be done using simple arithmetic. It does recognise that any particular grid square is within one of its internal polygons (local authority & vice county boundaries). As it does this quickly there may be a bit more to the code than just sums.<br />
<br />
The decision to rely on the useful properties of a regular kilometric grid, and effectively hard-wiring two grids into the code, has an unfortunate consequence for MapMate. It is not a practical tool for biological recording outside the British Isles (although I think at one stage it was adapted for Luxembourg). Whilst the Dutch & Swiss use 10km grids for recording, most other countries with well-established recording systems use some variant of a latlon grid: in Germany sheetlines of 1:25k maps are used in Baden-Württemberg, elsewhere it's often a 7.5 or 15 minute 'square'. (MapMate also suffers from being based on old technology, but that is another (huge set of) issue(s)).<br />
<h3>
Coda</h3>
MapMate is still a phenomenally useful piece of software, widely appreciated by its users as being straightforward to use, but it increasingly suffers from both being based on very old software, and a business model which does not provide enough recompense for continued development.<br />
<br />
At the same time the newer approaches of web-based and app based data entry do not yet deliver all the same functionality. I've tried address one deficiency of MapMate with more accurate maps delivered as images. However it cannot fix how MapMate internally validates records based on boundaries. I do believe that this and similar approaches can be used to maintain or extend MapMate's utility until newer software reaches a similar level of functionality.<br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-62719721409390479632018-09-15T13:16:00.001+01:002018-09-16T12:07:11.670+01:00A few oddities of OpenStreetMap history filesI've been working with the OSM history data for Great Britain <a href="http://sk53-osm.blogspot.com/2018/07/can-we-identify-completeness-of.html">for a while</a> now. This has mainly removing bugs in the initial processing (yup, out-by-one errors mainly) and, as a side effect, working out how to improve the speed of extracting way geometries. En route I have noted a few quirks which may be helpful for others working with the data.<br />
<br />
Some of these should be obvious, but I feel that they are worth stating nonetheless.<br />
<br />
<br />
<a name='more'></a>Here they are:<br />
<ul>
<li>Elements can s<b>tart their existence with a deleted status</b>. Node <a href="https://www.openstreetmap.org/node/1/history">#1</a> is the best example of which I'm aware. Here is the <a href="https://wiki.openstreetmap.org/wiki/OsmChange">OSM Change</a> XML (.osc) from that deletion:</li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><osmChange version="0.6" generator="CGImap 0.6.1 (1966 thorn-01.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/"><br /><create><br /><node id="1" visible="false" version="1" changeset="9257" timestamp="2006-05-10T18:27:47Z" user="τ12" uid="1298"/><br /></create></osmChange> </span></span></blockquote>
<blockquote class="tr_bq">
No idea how this came about.</blockquote>
<ul>
<li><b>Ways can have nodes which come into existence later </b>than the way itself. Way <a href="https://www.openstreetmap.org/way/180/history">#180</a> was <a href="https://www.openstreetmap.org/changeset/34555">created by</a> Deanna Early in June 2006:<br /><br /><blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><way id="180" visible="true" version="1" changeset="34555" timestamp="2006-06-04T00:35:25Z" user="Deanna Earley" uid="2231"></span></span> <span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><br /><nd ref="635109"/><br /><nd ref="15433291"/><br /><nd ref="635110"/><br /><nd ref="15433288"/><br /><nd ref="15433290"/><br /><nd ref="635111"/><br /><tag k="highway" v="residential"/><tag k="name" v="Green Lane"/></way></span></span>
But some of the nodes only appear to have been created rather later, like this one in September 2006:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><node id="635109" visible="true" version="1" changeset="109674" timestamp="2006-09-13T00:37:15Z" user="Deanna Earley" uid="2231" lat="50.8889009" lon="-1.3269173"/><br /><node id="635109" visible="true" version="2" changeset="6669766" timestamp="2010-12-15T17:48:55Z" user="0123456789" uid="55782" lat="50.8888938" lon="-1.3270150"/></span></span></blockquote>
Again I don't know why.</blockquote>
</li>
</ul>
<ul>
<li>The opposite is true, <b>nodes can be deleted whilst they still remain in the list of way</b> nodes. Here'a a road in Thurso, way <a href="https://www.openstreetmap.org/way/4839614/history">#4839614</a>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><way id="4839614" visible="true" version="1" changeset="122653" timestamp="2007-07-02T11:09:11Z" user="rjmunro" uid="729"><br /><nd ref="31140984"/><br /><nd ref="31140986"/><br /><nd ref="31140987"/><br /><nd ref="31140988"/><br /><nd ref="31140991"/><br /><nd ref="31140992"/><br /><nd ref="31140994"/><br /><nd ref="31140996"/><br /><nd ref="31140997"/><br /><nd ref="31140999"/><br /><tag k="created_by" v="Tways 0.2"/><br /><tag k="highway" v="unclassified"/><br /></way><br /><way id="4839614" visible="true" version="14" changeset="52740714" timestamp="2017-10-08T19:56:51Z" user="woodpeck_repair" uid="145231"><br /><nd ref="31140984"/><br /><nd ref="31140986"/><br /><nd ref="31140987"/><br /><nd ref="31140988"/><br /><nd ref="766071173"/><br /><nd ref="31140991"/><br /><nd ref="2344890723"/><br /><nd ref="2347341994"/><br /><nd ref="31140992"/><br /><nd ref="766073124"/><br /><nd ref="31140994"/><br /><nd ref="766071341"/><br /><nd ref="31140996"/><br /><nd ref="538050262"/><br /><nd ref="538049944"/><br /><nd ref="2409031255"/><br /><nd ref="31140997"/><br /><nd ref="538049236"/><br /><tag k="highway" v="unclassified"/><br /><tag k="ref" v="U4135"/><br /><tag k="source:ref" v="official"/><br /><tag k="source_ref:ref" v="http://www.highland.gov.uk/download/downloads/id/11641/list_of_adopted_roads_-_u_class"/><br /></way></span></span></blockquote>
Node <a href="https://www.openstreetmap.org/node/31140999/history">#31140999</a> was deleted by user Ollie in 2010:<br /><br />
<node id="31140999" visible="true" version="1" changeset="121208" timestamp="2007-07-01T18:51:46Z" user="rjmunro" uid="729" lat="58.5975437" lon="-3.5137181"><tag k="created_by" v="JOSM"/></node><node id="31140999" visible="false" version="2" changeset="4923978" timestamp="2010-06-06T23:45:04Z" user="Ollie" uid="10785"><tag k="created_by" v="JOSM"/></node><br />
<br />
In this case it's clear what has happened. All versions of the way from 2 to 13 <a href="https://www.openstreetmap.org/changeset/52740714">needed to be redacted</a>, and redaction cannot hope to restore all changes completely.<br />
</li>
</ul>
<ul>
<li>A logical corollary of redactions is that <b>version numbers are not</b> guaranteed to be <b>sequential</b>, whilst they do always increase monotonically, . This is particularly important if you use SQL window functions. I've found it useful to have an element sequence number which is sequential.<br /></li>
<li>Element <b>versions can share the same time stamp</b>. OSM XML only stores timestamps on objects to the nearest second. Whereas it must be possible to adjust the position of a node several times in one second, it's harder to see how these can be saved to the API so quickly. One case where this might work is where work is saved, and whilst the data is being uploaded some fault is noted, corrected and immediately saved. Another possibility, which I haven't investigated is in the mechanisms by which editors & the API generate timestamps. Effectively I havent worried about this and have 'thrown away' the earlier elements sharing the timestamp, but it does mean that you can't rely on date order alone to correctly sequence element history.<br /><br />Simon Poole asked me for an example, so here is one, node <a href="https://www.openstreetmap.org/node/107298/history">#107298</a>, versions 2 through 4 all have the same timestamp:<br /><br /><blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><node id="107298" visible="true" version="1" changeset="66" timestamp="2005-06-05T18:31:25Z" user="zool" uid="131" lat="51.5496368" lon="0.0051554"/><node id="107298" visible="false" version="2" changeset="415456" timestamp="2007-11-05T22:02:42Z" user="Paul Todd" uid="12503"/><node id="107298" visible="false" version="3" changeset="415456" timestamp="2007-11-05T22:02:42Z" user="Paul Todd" uid="12503"/><node id="107298" visible="true" version="4" changeset="415456" timestamp="2007-11-05T22:02:42Z" user="Paul Todd" uid="12503" lat="51.5496390" lon="0.0051490"><tag k="created_by" v="Potlatch 0.4c"/></node><br /></span><span style="font-family: "courier new" , "courier" , monospace;"></span></blockquote>
This was back in the early days of Potlatch when edit changes went live immediately on the API database. This 'live mode' of Potlatch prior to Potlatch 1.0 may explain the existence of most of these very transient versions.
</li>
</ul>
<ul>
<li><b>History files for an area may contain data outside that area</b>. I've used Geofabrik's history files (now requiring an OSM logon) for Great Britain, but they contain all versions of node #1. It spent some of it's existence in Argentina. I suspect the same is true if one creates files using Osmium. Mainly this is because one needs the history including when a node is deleted which lacks any geographical information. As far as I know it adds little overhead.</li>
</ul>
A quick re-cap of the key points again:<br />
<ul>
<li>Element history in OSM full planets is not complete (at least for early stages of the project).</li>
<li>There can be gaps in the sequence of OSM element version numbers (usually caused by redactions).</li>
<li>Redactions may affect the integrity of the relationships of geometries.</li>
<li>Time granularity is not enough to separate all object versions.</li>
<li>History files do not cleanly contain only data from the relevant area.</li>
</ul>
Taken together these facts means that processing OSM history data requires a fair degree of defensive programming measures. I've introduced a few as I've been working through data for GB, clearly I need more, because it's otherwise hard to separate out programming bugs from data glitches. For instance, I am now introducing node counts, point counts and deleted node counts for geometries as I assemble them.<br />
<br />
This also means that I'm delaying placing the basic SQL processing code for <a href="https://osmcode.org/opl-file-format/">Osmium OPL</a> data on github until I done some more checking. However, the specific examples I've posted above represent decent candidates for building up a small dataset for tests.<br />
<br />
<h4>
<span style="font-size: x-small;">Revisions:</span></h4>
<span style="font-size: x-small;">2018-09-16 12:00: Example of a node with multiple versions with a single timestamp added. </span><br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com1tag:blogger.com,1999:blog-2082422783692631078.post-64880319405555658952018-09-09T15:00:00.001+01:002018-09-09T15:00:29.553+01:00Where the streets have no name: Coach Road Estate, Washington<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnwIiukusqlSXO6n1UHyiHzIfvwV0POswDR0_Xvu_13cnVm2OPRzWn9LP3AhrdL8vUernRA-p_oKdi2ow8WfO58jVZTby69mgbn3F5RxLCUj32YRgOJjo8eFPYErcvLSU-mAyhkFdtU_to/s1600/2028309_61eb9eff.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnwIiukusqlSXO6n1UHyiHzIfvwV0POswDR0_Xvu_13cnVm2OPRzWn9LP3AhrdL8vUernRA-p_oKdi2ow8WfO58jVZTby69mgbn3F5RxLCUj32YRgOJjo8eFPYErcvLSU-mAyhkFdtU_to/s320/2028309_61eb9eff.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Serlby Close, Usworth. The houses on the left are part of Coach Road Estate<br />
<span style="font-size: xx-small;">Source: © <a href="http://www.geograph.org.uk/profile/45095">Alex McGregor</a> at <a href="http://www.geograph.org.uk/reuse.php?id=2028309">Geograph</a> <a href="https://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA-2.0</a></span></td></tr>
</tbody></table>
Microsoft, like a number of <a href="https://theodi.org/article/how-are-facebook-apple-and-microsoft-contributing-to-openstreetmap/">other large tech firms</a>, now has teams working to improve OpenStreetMap. A couple of weeks ago they turned their attention to the UK, and <a href="https://lists.openstreetmap.org/pipermail/talk-gb/2018-August/021923.html">asked</a> about some roads apparently missing street names.<br />
<br />
I checked a few of these examples using available Open Data sources, primarily those of the Ordnance Survey. For the most part, our existing mapping seems correct: there is no official street name. Many were places like caravan parks and industrial estates, but one really stood out. This was an area on the north side of <a href="https://www.openstreetmap.org/node/316252572#map=12/54.8987/-1.5288">Washington</a>, County Durham called Coach Road Estate. As I investigated it soon became apparent that the estate is interesting for other reasons too.<br />
<br />
<br />
<a name='more'></a>It's very unusual for roads in an urban setting to lack names. Although it is not uncommon in the countryside where addresses will be of the form <housename>,<village-name>. However it was very easy to verify that this was indeed the case for the Coach Road Estate. The main reason being the existence of an <a href="http://osm-nottingham.org.uk/?z=17&lon=-1.52783&lat=54.91850&bgl=OSM,1,15&s=coach%20road%20estate&st=SearchOpendataJson&uc=1">abundance of government open data</a> about the site (nearly 700 records in all):<br />
<ul>
<li><b>Food Hygiene</b>: a single convenience store <a href="http://ratings.food.gov.uk/enhanced-search/en-GB/%5E/coach%20road%20estate/Relevance/0/429/%5E/0/1/10">is present.</a></li>
<li><a href="http://naptan.dft.gov.uk/"><b>Naptan</b></a>: bus stops on the estate all use the name, and conveniently often include the nearest house number.</li>
<li><a href="https://data.gov.uk/dataset/cf3aac3e-f3f3-4564-9df4-099346246251/national-register-of-social-housing-nrosh"><b>National Register of Social Housing</b></a>. Although this stopped being maintained in 2011 it is still invaluable not just for addresses, but also to identify planned estates of social housing (council estates, colliery villages etc). There are hundreds of addresses ranging from 1 to 589 in this dataset (there are 648 records in total, but this includes a lot of garages). As they have postcodes they can also be geolocated.</li>
<li><a href="http://download.companieshouse.gov.uk/en_output.html"><b>Companies House</b></a>: five companies are registered at four addresses on the estate.</li>
</ul>
As can be seen by the geography of the data which can be reasonably accurately located (i.e. postcode centroids), the bounds of the area are self-evident even before viewing aerial images.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhX5w5frQseQaT-t-6qxpvpWLsapQdOWek2K2yCurgSh9NmFtKONjUQlYyVWCd7KkiKSbUGX1CZRJ_aPAbV6hFFH-YszTo5_-FXW2u3YHZDBqLEIo-jReTgo-5DeGHpEmNqFvuNx71W_7OG/s1600/Screenshot+%25286%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="651" data-original-width="795" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhX5w5frQseQaT-t-6qxpvpWLsapQdOWek2K2yCurgSh9NmFtKONjUQlYyVWCd7KkiKSbUGX1CZRJ_aPAbV6hFFH-YszTo5_-FXW2u3YHZDBqLEIo-jReTgo-5DeGHpEmNqFvuNx71W_7OG/s320/Screenshot+%25286%2529.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Screenshot from Will Philips <a href="http://osm-nottingham.org.uk/?z=17&lon=-1.52783&lat=54.91850&bgl=OSM,1,15&s=coach%20road%20estate&st=SearchOpendataJson&uc=1">OSM-Nottingham</a> website<br />
showing geolocated OpenData with "Coach Road Estate" in the address, name or other field. <br />
Unfortunately, I failed to screenshot this before mapping the area in detail.<br />
(Although the site is focused on the Nottingham area it now allows searches on most UK open data useful to OSM).</td></tr>
</tbody></table>
Having established from multiple sources that the name belonged to the estate as a whole and not to it's roads, it was time to add it to OpenStreetMap. Although the estate has perhaps 600 houses it's too small to be a suburb, so I decided to add it as <a href="https://wiki.openstreetmap.org/wiki/Tag:place%3Dneighbourhood">place=neighbourhood</a>. Unusually, because it has such precisely delineated boundaries, I could map the area <a href="https://www.openstreetmap.org/way/621256595">itself</a>.<br />
<br />
It was also possible to add a few addresses from Naptan. This is a classic case where the <a href="https://wiki.openstreetmap.org/wiki/Key:addr:place">addr:place</a> tag can be used in addresses. Fortunately there were two bus stops close together on the NE corner of the estate which allowed, with some judicious guesswork, many more addresses to be inferred.<br />
<br />
By now I had multiple other questions: When was the estate built? What types of houses? Was it planned as part of the <a href="https://en.wikipedia.org/wiki/New_towns_in_the_United_Kingdom">New Town</a>, or earlier? Was it constructed to provide housing for colliery workers? Can we find out anything about the layout?<br />
<br />
Fortunately the first question was easy to resolve: the estate is not shown on maps from the 1940s and early 1950s, but is partially complete on those dated around 1960. This in turn suggests that it pre-dates the New Town, which was formally designated in 1964.<br />
<br />
Washington, unlike other new towns, was not built on a predominantly green-field site. Rather it was already an area of piecemeal scattered housing and associated industrial areas (mainly collieries).<br />
<br />
Dispersed settlement patterns of this type were quite common on British coalfields (the area where my father comes from Hollinwood, Oldham is <a href="https://maps.nls.uk/geo/explore/#zoom=14&lat=53.5245&lon=-2.1432&layers=1&b=1">an example</a> from elsewhere, although the pattern has been obscured by later development). The development sits in the former parish of Usworth, which at the <a href="https://maps.nls.uk/geo/explore/#zoom=14&lat=54.9168&lon=-1.5341&layers=6&b=1">start of the 20th century</a> contained a nucleated village, Great Usworth, a development of long terraces of workers housing extending piecemeal westward from Usworth Colliery, and scattered terraces elsewhere. These terraces would have provided very small basic houses, mainly for colliers. My great-grandfather, born around 1860, lived in such a house in Hollinwood. It was not a "Parlour House", but the main living area was entered directly from the street, behind this was a scullery with a slop stone (a large sink), and perhaps 2 bedrooms upstairs. The toilet was in the back yard. My father remembers that the front door of my great-grandfather's house was rarely closed, and a large coal fire was kept burning on even the warmest days (as a miner his coal allowance was more than they needed). Amazingly one of these terraces still survives in Usworth, <a href="https://www.openstreetmap.org/way/168759782">Pensher View</a>. Estate agent details suggest that this survival is because the houses were larger than typical, and capable of being updated.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/29295370@N07/15099551610" style="margin-left: auto; margin-right: auto;" title="Old prefabricated housing in Washington"><img alt="Old prefabricated housing in Washington" height="350" src="https://farm3.staticflickr.com/2941/15099551610_fc565dc4a0.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Prefabs in Sulvgrave Village (presumably those at Usworth Green). The terrace behind is probably Pensher View<br />
Photo from Tyne & Wear Archives and Museums via Flickr. Crown Copyright. More details on their <a href="https://www.flickr.com/photos/29295370@N07/15099551610">Flickr page</a>.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
By 1950 other developments had taken place: an estate focused on a green space, <a href="https://www.openstreetmap.org/way/67494745">The Oval,</a> with a layout very suggestive of 1930s and 1940s social housing (and checking NROSH data, we can see it was developed as such). This was part of a development, named <a href="http://www.raggyspelk.co.uk/washington_pages/selections1/concord_aerial.html">Concord</a>, around a new crossroads, created when a road labelled as "New Road" on 1:25k maps of the 1950s was built northwards from New Washington. The road has since been re-named. A batch of <a href="http://www.communityarchives.org.uk/content/organisation/prefab-museum">pre-fabricated houses</a> were erected on the open space between terraces by the colliery, and were called Usworth Green.<br />
<br />
For many fantastic photos, and other documentary evidence (including annotated maps), of this housing, the <a href="http://www.raggyspelk.co.uk/">Raggy Speik</a> website has been an invaluable resource. There is also a fascinating article about the local bus operator with great local colour on the <a href="http://www.washingtonhistorysociety.co.uk/community/washington-history-society-12965/the-looey-bus/">Washington History Society website</a>.<br />
<br />
It was on this palimpsest of earlier developments that Washington New Town came into being.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/jrjamesarchive/9245116721" style="margin-left: auto; margin-right: auto;" title="Masterplan of Washington New Town"><img alt="Masterplan of Washington New Town" height="333" src="https://farm4.staticflickr.com/3686/9245116721_9057f083e6.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Masterplan of Washington New Town from JR James Archive on Flickr (<a href="https://creativecommons.org/licenses/by-nc/2.0/">CC-BY-NA-2.0</a>). <br />
Coach Road Estate is in the northernmost block of residential landuse zoning.</td></tr>
</tbody></table>
<br />
The immediate antecedents of the New Town designation lay in the Hailsham report of 1963 ( written by <a href="https://en.wikipedia.org/wiki/Quintin_Hogg,_Baron_Hailsham_of_St_Marylebone">Quentin Hogg, Viscount Hailsham</a>). The North-East suffered from a number of problems: a ageing and, often out-moded, housing stock with over half from before WWI; the demise, or anticipated demise, of traditional sources of employment (mining, shipbuilding); poor communications, and much land in need of remediation. Unemployment, and housing shortages were leading to outward migration from the area, which meant that Hailsham's report was something of an emergency rescue measure. (See <a href="http://etheses.dur.ac.uk/2335/2/2335_345-vol2.pdf">this thesis</a> for more detail).<br />
<br />
The existing new towns in County Durham, <a href="https://en.wikipedia.org/wiki/Peterlee">Peterlee</a> and <a href="https://en.wikipedia.org/wiki/Newton_Aycliffe">Newton Aycliffe</a>, needed strengthening, and finally central government was now receptive to the demand from local authorities to grant the area around Washington the same status. Previous resistance had come from several sources: Sunderland Council, central planners who wanted greenfield sites, and ones which were self-contained for employment. However, it seems that at a local level the area had already been chosen for development, at least of housing: Concord, and, I suspect Coach Road Estate, are testament to that. The predecessor of the New Town, Washington Urban District was formed in 1922 and grew by <a href="http://www.visionofbritain.org.uk/unit/10042860/boundary">absorbing</a> adjacent areas.<br />
<br />
The actual development of the New Town is beyond the scope of this post, but there are some nice images on John Grindrod's <a href="http://dirtymodernscoundrel.blogspot.com/2014/01/imagining-washington-new-town.html">site</a> from the master plan (including the fantasy by one of the artists of a Waitrose supermarket in the North East from the 1960s). Here I'll just note that the development area was divided into about 17 numbered communitie.s They later acquired formal names, and the area around Usworth Colliery became Sulgrave (named after George Washington's <a href="https://en.wikipedia.org/wiki/Sulgrave">other ancestral village</a> in Northamptonshire).<br />
<br />
Coach Road Estate was earmarked for one of the local centres. This is the area next to St Bede's Catholic church, and incorporates a small number of shops, and a (former) pub, the Coach and Horses.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Q_3hUKaQ9ZY1Dd1txSWVEtaN3CCfoUkQi_TmYmre8dFS_bCkhFtWKgf1HNL8gK91C56WcMfRdRUe485URF0KiLU-AOsc6wV3oonIMC7X3S_dgBVq_zVM3-r5HfbXhXORokrIvmd7H4Op/s1600/geograph-2028317-by-Alex-McGregor.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="640" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Q_3hUKaQ9ZY1Dd1txSWVEtaN3CCfoUkQi_TmYmre8dFS_bCkhFtWKgf1HNL8gK91C56WcMfRdRUe485URF0KiLU-AOsc6wV3oonIMC7X3S_dgBVq_zVM3-r5HfbXhXORokrIvmd7H4Op/s320/geograph-2028317-by-Alex-McGregor.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Coach & Horses, Usworth in 2010 (now closed)<br />
<span style="font-size: xx-small;">Source: © <a href="http://www.geograph.org.uk/profile/45095">Alex McGregor</a> at <a href="http://www.geograph.org.uk/photo/2028317">Geograph</a> <a href="https://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA-2.0</a></span></td></tr>
</tbody></table>
The pub is a typical post-war brick-built estate pub. Much of the housing is traditional brick-built construction, although there are also some less conventional bungalows, and maisonettes. None of the built structures visible in available pictures really help in dating the place any better than do maps.<br />
<br />
Of rather more use is the (probably) unusual layout of the estate. Firstly, it clearly incorporates provision for car ownership: something even early-post WWII estates did not do. Groups of houses face a service road at the rear giving access to garages and parking areas. The service roads do not appear to have pavements, but there may be very narrow ones. Virtually all houses front onto green spaces, either generous verges on the access roads, or, in the main, onto to the grass extents which separate clusters of houses. The primary access to the estate is via a loop road with just two access points to the main distributor road system. Pedestrians have an extensive path network in the estate and many more ways to leave it.<br />
<br />
The obvious arrangements to both provide for, and separate out, the motor car, are strongly redolent of the <a href="http://sk53-osm.blogspot.com/2011/04/radburn-exemplar-or-nightmare.html">Radburn</a> approach. However, instead of motor access being via an external loop road with service roads penetrating into a doughnut of housing surrounding a green core, here the loop road is within the estate, and housing and green space are inter-digitated. For want of a better term I've <a href="https://twitter.com/MunicipalDreams/status/1035826714771894272">called this</a> 'Inside-out Radburn'. Ian Waites is currently researching <a href="http://instancesofachangedsociety.blogspot.com/2017/12/the-radburn-idea-1-nothing-lost-and.html">Radburn </a>designs, so I hope this might pique his interest.<br />
<br />
This post has entirely been based on my delving into the likely circumstances which led to one small estate of houses having road name.<br />
<br />
I think it demonstrates just one of the reasons why the history of such places is so fantastically rich. Indeed, when I read John Boughton's new book <a href="https://www.bloomsbury.com/au/municipal-dreams-9781784787394/"><i>Municipal Dreams</i></a> this summer, I was struck that the history of housing in Britain in the 20th century is pretty much the history of social housing. Compared with privately developed housing, there was more innovation (not always successful) more design, more depth of planning, and a greater commitment from architects, planners, engineers, and, a surprising variety, of industries. John's achievement in condensing this richness into 300 pages leaves me in awe, especially, as his own research as detailed on <a href="https://municipaldreams.wordpress.com/">his blog</a> is very much rooted in the intimate history of individual estates.<br />
<br />
Nor is this the last word. It is clear there are abundant untapped areas of investigation in this whole field. It's a fascinating one, and as with other aspects of social history OpenStreetMap can play a useful part. <br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-58726328757518050272018-07-25T15:21:00.000+01:002018-07-25T15:21:33.017+01:00Coda on shop completion rates on OSMThanks to John Baker (<a href="https://www.openstreetmap.org/user/Rovastar">Rovastar</a>) for a few suggestions discussing my <a href="http://sk53-osm.blogspot.com/2018/07/can-we-identify-completeness-of.html">recent blog</a> post in the <a href="https://wiki.openstreetmap.org/wiki/Nottingham/Pub_Meetup">pub</a> last night:<br />
<br />
<ul>
<li>What do the graphs of numbers of unique shop tags look like with heavier filtering of relatively poorly used tags.</li>
<li>E-cigarette shops are a recent phenomenon, and should represent a genuinely novel tag rather than the mix of typos, synonyms etc which characterise much of the long tail of shop tags. </li>
</ul>
<br />
These were easy to follow up, so I present the graphs here:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvB6pROPx-x9OhkNPQcFMd-6m0M-bxrg5DrtPmrfkPYRsNhdxg7GpSDXGAtWyuDLbP5G1THXl6Hn3MSj5NliohTj-XJnqCAiQVrnvgP4EWjZFoJnE8uPmKRz4p0Ccs8DNz1nyJlg11kryN/s1600/shop_tags_dist.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvB6pROPx-x9OhkNPQcFMd-6m0M-bxrg5DrtPmrfkPYRsNhdxg7GpSDXGAtWyuDLbP5G1THXl6Hn3MSj5NliohTj-XJnqCAiQVrnvgP4EWjZFoJnE8uPmKRz4p0Ccs8DNz1nyJlg11kryN/s320/shop_tags_dist.png" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Unique shop tags over time on OpenStreetMap for Great Britain, <br />
filtered to remove tags with a restricted number of uses as at June 2017.</td></tr>
</tbody></table>
For virtually any level of filtering the curves level out around 2010-2011. Thus the core set of shop tags looks to be very stable. A good place to judge the extent of likely synonymy for shops in Britain is the LUA script used by <a href="https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua#L3095">SomeoneElse</a> for his "Useful Maps".<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKlP3L1HcHa1z9VOJl0EkhXbIL9gAFugnSl-5k1d7SD0qjafC7Q5-73cSZJZ5ZKEFaWCUaqhT74WW4gGKA8Wzlb3TycEY1T69nAm3z6tvU_bQOz7c0qBH83cA85ygUdZKxBbctGNbLMbuk/s1600/shop_ecig.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKlP3L1HcHa1z9VOJl0EkhXbIL9gAFugnSl-5k1d7SD0qjafC7Q5-73cSZJZ5ZKEFaWCUaqhT74WW4gGKA8Wzlb3TycEY1T69nAm3z6tvU_bQOz7c0qBH83cA85ygUdZKxBbctGNbLMbuk/s320/shop_ecig.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Growth of mapped e-cigarette shops in GB on OSM</td></tr>
</tbody></table>
As expected e-cigarette shops first appeared rather late, at the end of 2013, and there are a decent number mapped (over 200 by mid 2017). I haven't checked, but I suspect the sharp increase in 2017 was caused by some tagging rationalisation. It's not unusal for new things to acquire a range of synonyms before tagging stabilises and one value becomes favoured. (It's equally true that in some cases this does not happen).<br />
<br />
I've had a couple of other requests which it will take rather longer to look at, but if you have ideas relating to shops in Great Britain I can look at the data right now.<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-78244440079164167022018-07-24T16:01:00.003+01:002018-07-24T16:35:24.510+01:00Can we identify 'completeness' of OpenStreetMap features from the data?At the Milan SotM conference <a href="https://twitter.com/sfkeller">Stefan Keller</a> from the <a href="https://www.ifs.hsr.ch/index.php?id=12520">Geometalab</a> at HSR (Rapperswil) will <a href="https://2018.stateofthemap.org/2018/A30-Areas-of-Interest_for_OpenStreetMap_with_Big_Spatial_Data_Analytics_/">talk</a> about recent work of his group on identifying "Areas of Interest" (<b>AoI</b>) from OpenStreetMap data. Stefan has been kind enough to involve me in some discussions about this work as it has progressed, but in this post I am solely concerned with a separate issue arising from the use of points of interest in this work.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-aczFGxmw0u3Nee0jezGjv64iH-RFibe4xYxQiOdfywGEaIO6w-ViTyDKVyFf2f0qe6hhei2qnr7Aj0lrXVi03_8ZzjhOqs9DQ3WFD2JsFL8U9d9GGpP1q4r849kXGLaGF-pGHSoKcgh/s1600/shop_time_by_la3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-aczFGxmw0u3Nee0jezGjv64iH-RFibe4xYxQiOdfywGEaIO6w-ViTyDKVyFf2f0qe6hhei2qnr7Aj0lrXVi03_8ZzjhOqs9DQ3WFD2JsFL8U9d9GGpP1q4r849kXGLaGF-pGHSoKcgh/s640/shop_time_by_la3.png" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Growth of shops mapped on OSM for selected Local Authorities<br />
(See Analysis section below for commentary)</td></tr>
</tbody></table>
<br />
<br />
Areas of Interest were <a href="https://blog.google/products/maps/discover-action-around-you-with-updated/">introduced</a> on Google Maps back in 2016. Loosely they correspond to shopping, entertainment and cultural areas with large clusters of relevant points of interest. No doubt Google not only used map features, but also other sources of data such as location of Android phones to calculate the footprints for Areas of Interest (shown in a pale orange or salmon colour on Google Maps).<br />
<br />
There are issues with the Google implementation, some discussed in this <a href="https://www.citylab.com/design/2016/08/google-maps-areas-of-interest/493670/">CityLab article</a> from 2016. My own examination of Google Maps confirms that shopping areas which are otherwise equivalent in range and type of shops are chosen as AoI in wealthy areas, but not in poorer areas dominated by social housing. I also found some places, notably the UBS IT centre in Altstetten, Zurich, which have erroneously been identified as AoI by Google. The work of Geometalab is therefore interesting not just in terms of whether OSM data can be used to calculate similar areas, but also to provide suitable data where biases based on socioeconomic status can, at least, be identified and corrected because data and code are open.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQW-W1I4WqWeWA_xEbOazKOLVc-yP7Z1vc1Sh8Ww9VRWTa1mMCA21e9erBE44Zd8x6lHcaZwdE5rBH-DbNowCyIJsQpQvn3oAR31Rj_hJobIyxrzXgQ_pXUzkGhW8m2MFRt3axZNQbcFoh/s1600/Screenshot+%25282%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="600" data-original-width="1138" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQW-W1I4WqWeWA_xEbOazKOLVc-yP7Z1vc1Sh8Ww9VRWTa1mMCA21e9erBE44Zd8x6lHcaZwdE5rBH-DbNowCyIJsQpQvn3oAR31Rj_hJobIyxrzXgQ_pXUzkGhW8m2MFRt3axZNQbcFoh/s400/Screenshot+%25282%2529.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Zurich, centre and Aussersihl districts, showing Areas of Interest.<br />
Work of Geometalab, derived from OpenStreetMap data.</td></tr>
</tbody></table>
The starting point for this type of work relies on areas where POI mapping density is high and reasonably complete (for instance, the areas of Switzerland which Stefan's group have looked at, and areas
of the English <a href="http://sk53-osm.blogspot.com/2013/06/the-shopping-news-mapping-retail.html">East Midlands</a> and Germany which I have looked at
both recently, and <a href="http://sk53-osm.blogspot.com/2013/04/segmentation-of-retail-landuse-why-do.html">in the past</a>). Given that it is possible to calculate reasonable AoIs from OSM data where PoI density is high, the question arises "Can we identify which areas are 'reasonably' complete?". Normally, this type of work has involved comparing OSM data to some external reference data which are assumed for the purposes of comparison to be complete (for instance Peter Reed's <a href="http://tlatet.blogspot.co.uk/2015/08/osm-retail-survey-conclusions-1.html">work</a> on UK retail). However, in many parts of the world, and for many topic domains there is no readily usable data for this purpose. So the ancillary clause for the question is ", and we do this with OSM data alone?"<br />
<br />
This post is a first look at the problem for one class of POIs: shops.<br />
<br />
<br />
<a name='more'></a><br />
<br />
<h4>
Species Accumulation Curves</h4>
My starting point comes from familiarity with something called a <a href="https://en.wikipedia.org/wiki/Species_discovery_curve">Species Accumulation Curve</a>. I believe that there are strong points of commonality between how OSM data is accrued and these curves.<br />
<br />
For many groups of plants, animals, and other biota, it is nigh on impossible to find, in a single survey, all the different species which grow or live in a particular area. Numerous factors influence this:<br />
<ul>
<li><b>Surveyors' skills.</b> Not every surveyor has the same skill set, training, or even just visual acuity. One of the best naturalists I know is a care worker, who can trump national and international scientific authorities by finding more species than they can in the field.</li>
<li><b>Seasonality</b>. Plants flower at different times, birds migrate, some insects are on the wing for a short time.</li>
<li><b>Weather</b>. The hot dry weather in Britain has greatly reduced the number of flowers I have seen in the past few weeks, and consequently their insect visitors. On Sunday I was heartened to lead a field meeting where we found 44 species in our target group; but 10 years ago in the same location & at the same time of year we found nigh on 30 more.</li>
<li><b>Predator Prey</b> relationships. Many species numbers go in cycles (for instance Lemming years), but at least for some insects population density has been estimated to be an order of 10^12 between the troughs and the peaks. Ideally one surveys through 2-3 full cycles: problematic if they are 17-year cicadas, or bamboos which flower and die on a 70-year cycle.</li>
<li><b>Increasing knowledge</b>. Sharing of techniques for searching or recognising different plants and animals can have an amazing influence on total numbers of species found. This is true even in Britain for as well studied a group as the higher plants. The BSBI's Atlas 2020 project which will be completed in 3 years time, will not only show changes in plant distribution brought about by agricultural intensification, increased urbanisation and climate change, but also changes from looking more closely for a wider range of plants (notably urban weeds and garden escapes).</li>
<li><b>Sheer cussedness.</b> Fungi are particularly awkward customers. Most spend their time invisibly underground, only showing fleetingly as fruiting bodies (mushrooms) when they feel the time and weather is right. Even with the most capable surveyors in the world the full extent of species complexity can only be appreciated by continual regular surveying of the same place. There are two locations in England which demonstrate this point. Esher Commons have been regularly surveyed for fungi by scientists, including global authorities on some fungal groups, from Kew Gardens for many years. No other place <a href="http://www.brc.ac.uk/psl/content/esher-commons"><b>in the world</b></a> is know to have as many fungi, and around 20% of all fungi known in the British Isles have been found at Esher. <a href="https://www.slnnr.org.uk/wildlife.aspx">Slapton Ley</a> in Devon has also received decades of regular surveying effort for fungi, and has over 2000 known fungi species. It may be come second to Esher for known fungal diversity!</li>
<li><b>Recorder effects</b>. Even for professional scientists it is often difficult to maintain a constant recording effort. Most biological data is gathered by citizen scientists who can only devote what leisure time they can spare for the activity. Recorders tend to be located in larger cities, rather than in potentially highly species diverse remote areas. For many species groups only a few people are seriously dedicated (as in my own interest in Plant Galls).</li>
</ul>
The <b>key advantage</b> of species accumulation curves is that, whilst not impervious to these effects, they are a relatively robust measure. For my fungi-loving friends they are a useful tool to work out when to move on from one area to another. At the scientific level the curves are well studied and there is a good framework of statistical techniques for analysing them.<br />
<br />
<h3>
Differences between OpenStreetMap & Biological Record Data</h3>
<br />
Data collection for biological recording differs from that for OSM in one particularly important aspect. For biological records all observations in each survey count. In OSM every repeat observation of the same POI (pub, shop, restaurant) is never collected. This means we can make no use of properties of each individual survey activity which contributed POIs. Also there is absolutely no equivalent of an OSM import for biological records. I've focused on Great Britain, so the latter has little impact on the results I present below.<br />
<br />
We can still look at two types of accumulation: <br />
<ul>
<li>of individual shops;</li>
<li>of shop tags.</li>
</ul>
Note that empty shops are meaningful data in the first grouping.<br />
<br />
<h3>
Null Hypothesis</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVaQ5iq88LH1dfKKsv79hvT94gJpE_CKKRNdebtxiKYINqjOBjoio9kKZ0LkTFX4wLHqaCpvaOaVCk07my2GtMtGzcxyWULRc2HMm8l_sFnfZ4ay-ZWxsL0UYbWyPzxzWk6Q-GLfFYdvRO/s1600/IMG_8359a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVaQ5iq88LH1dfKKsv79hvT94gJpE_CKKRNdebtxiKYINqjOBjoio9kKZ0LkTFX4wLHqaCpvaOaVCk07my2GtMtGzcxyWULRc2HMm8l_sFnfZ4ay-ZWxsL0UYbWyPzxzWk6Q-GLfFYdvRO/s320/IMG_8359a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Typical small shops which get mapped piecemeal on OSM: <br />
near Conde de Casal Metro station<br />
Av. Mediterraneo, Madrid, Nov. 2016</td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<br />
My null hypothesis is that over time we should see the number of shops on OSM for a given area tailing off towards an asymptote once the area is well-mapped. I know that surveys I did in the Spring of 2013 changed the percentage of shops mapped in Nottingham from around 40% towards 90%. Even earlier <a href="https://twitter.com/JLZIMMERMANN">Jean-Louis Zimmerman</a> and <a href="https://twitter.com/tonyemery">Tony Emery</a> had mapped <a href="https://wiki.openstreetmap.org/wiki/Orange_(Vaucluse)#Plans_de_Ville">Orange</a> in great detail and a map of the town was published. I therefore took these two towns and a few others to see if this was plausible. Data were gathered point by point using Overpass-turbo, and plotted in LibreOffice.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtkP5NTvLWtal9ew9eBo0F8ABrlgPqKKj1IltZTJ0aoVT3j_Wb2RwhMVNy_qKJrY0geOpu8FQ0IVzLUctfdNaLmcIVYZWclwBbJJyfzdkq0iwhKSVmRN9PiHrKNESchfLDxts6p6dNNzvd/s1600/shop_graph2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="287" data-original-width="688" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtkP5NTvLWtal9ew9eBo0F8ABrlgPqKKj1IltZTJ0aoVT3j_Wb2RwhMVNy_qKJrY0geOpu8FQ0IVzLUctfdNaLmcIVYZWclwBbJJyfzdkq0iwhKSVmRN9PiHrKNESchfLDxts6p6dNNzvd/s400/shop_graph2.png" width="400" /></a></div>
<br />
The two towns (Nottingham & Orange) where I had good reason to believe that shops were reasonably complete some years ago formed a baseline and did appear to show curves with asymptotic properties.<br />
<br />
The remaining places I chose on the basis that I knew that they are well-mapped, but without knowing if retail properties had been mapped to completion. I thought that it was plausible that this would prove to be true for Zurich & Karlsruhe, certainly not for Madrid, and likely not for Dakar. San Francisco was chosen as a well-mapped location in North America. In practice none of the graphs for number of shops over times suggests that effort to map shops has reached an inflexion. Even for somewhere like Karlsruhe which has had active mappers for as long as anywhere the graph suggests that there is still scope for mapping shops.<br />
<br />
Gathering data point-by-point is fine for a quick test, but far too tedious (& expensive in use of free resources), so my next step was to wrestle with OSM History files.<br />
<br />
<h3>
Extracting OSM History data</h3>
I already had an OSM History file for Great Britain for June 2017 downloaded from Geofabrik. Unfortunately, these files do not appear to have been updated since Geofabrik changed the user metadata available on their public servers. Also because history files contain user information protected by GDPR, these files are now only available through using an OSM sign-on now.<br />
<br />
Manipulating history files effectively means either using the command line <a href="https://osmcode.org/osmium-tool/">osmium tool</a> or writing programs using osmium library. This in turns means installing osmium. I therefore did this under Ubuntu 16.04. There is a packaged version of osmium for Ubuntu, but it is ancient, so it is necessary to compile and install the current version 1.8.<br />
<br />
Osmium is very much designed for heavy duty sophisticated processing of OSM data. It's not really a toolset for quick-and-dirty <i>ad hoc</i> investigations of the kind I do. I was apprehensive about getting tied up in knots getting the Osmium tool installed, particularly when I read the list of <a href="https://github.com/osmcode/osmium-tool/blob/master/README.md">dependencies</a>.<br />
<br />
In practice the only problem I had initially was due to not cloning a couple of packages into the right location in my osmium build directories. As I've never used Cmake in my life I was certainly intimidated by the simple statement "Please read the CMake documentation and get familiar with the <code>cmake</code> and
<code>ccmake</code> tools which have many more options.". However, reassured by Richard Fairhurst & Andy Townsend that it wasn't too difficult to install I preserved and soon had it installed. One thing I would have found helpful would have been an outline of the directory tree for a build.<br />
<br />
I also had a 5 minute attempt to compile Peter Mazdermind's <a href="https://github.com/MaZderMind/osm-history-renderer">OSM History</a> tool, but this has not been maintained and uses very ancient versions of osmium, so I did not preserver.<br />
<br />
The key reason for using the 1.8 version is that it has better support for extracting dependent data. Thus in a two step process it is possible to filter a history file for all elements tagged with shop and then find all their dependent elements. This is well covered in the osmium manual.<br />
<br />
For pragmatic reasons I chose the One Per Line (OPL) format, as I could very quickly load this data "as is" into a Postgres database.<br />
<h3>
Wrangling the Data</h3>
As I have done for years I loaded the data exactly as it was stored in the source file, so that I could start from the raw data at any time all within Postgres. In practice I loaded nodes, ways & relations with distinct COPY TO statements.<br />
<br />
I then processed each element type into base tables: transforming the main columns from strings to the proper datatype. Tags involves converting the string into an array separated by commas in the form key, value, key1, value1 .... This in turn converts simply to hstore.<br />
<br />
For each element the next thing was to calculate the end date for each version and add this to the . This can be done with a window function, or by joining the base table to itself (a left join). (See my very <a href="http://sk53-osm.blogspot.com/2011/05/on-histories-of-openstreetmap-data.html">old post</a> for one way to do this).<br />
<br />
The major disadvantage of my pragmatic approach is that one has to reassemble geometries, but before one can do that it is necessary to determine the potential number of distinct geometries for each version of a way or relation element. As others have done before me I ignored relations (very few shops are mapped as relations), and just worked with ways. To do this I first found all distinct start dates for all the nodes versions which contributed to any given way element, which can then be treated as minor versions of each way version. I actually prefer the term geometry version. You can see something very similar if you look at the history of a way in Potlatch2.<br />
<br />
Once I had the start and end dates for each geometry version the linestrings for the way can be assembled by joining the way_node_history, way_geom_history and node_history tables. All ways should produce valid linestrings. By storing the linestrings it is possible to preform multiple checks so that the code only attempts to assemble valid polygons (st_npoints(geom) > 3 and st_isclosed(geom)) worked for me.<br />
<br />
The shop data is relatively small, under 70k ways, totalling around 125k versions, which expands to 180k geometry versions. For nodes of course versions is the total : 100k elements, 200k versions. Given the data goes back to 2007, the increase in data volume to handle history is very modest.<br />
<br />
The last thing I did was calculate a centroid for all the data (this can include ways which do not form polygons). All analysis used the shop centroids.<br />
<br />
It's worth noting that another paper in the SotM academic track by Alexander Zipf's group at Heidelberg may presage much easier analysis by anyone of OSM historical data without the need for this kind of data manipulation.<br />
<br />
<h3>
Analysis</h3>
<h4>
Shops by Local Authority </h4>
My starting point was to look at how many shops have been mapped within each local authority across Great Britain. This enables looking at a much more representative sample than the few cities I looked at earlier. There is a disadvantage in that local authorities do not correspond to cities and therefore may not make natural mapping units.<br />
<br />
A first couple of quick plots show that there is a huge diversity in numbers of shops mapped, when they are mapped, intensity of mapping activity, and so on. When all are plotted together it's difficult to pick out any other trends:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-ziH5igd4N8c3LXYgy889hrwtr9qAFrYiruLNMG8a_ZnQHFGR9aVQ4GGz1N6bGVFIcyR6zFO4g8J_LnHUmWeIwKJbTsUpg3q-c_-RFN8drfOvbe8ncpu2sNPWhE0FbYsUQCZZQ0Krj6Tu/s1600/shop_hist_1stdraft.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="660" data-original-width="1366" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-ziH5igd4N8c3LXYgy889hrwtr9qAFrYiruLNMG8a_ZnQHFGR9aVQ4GGz1N6bGVFIcyR6zFO4g8J_LnHUmWeIwKJbTsUpg3q-c_-RFN8drfOvbe8ncpu2sNPWhE0FbYsUQCZZQ0Krj6Tu/s640/shop_hist_1stdraft.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Progress in shop mapping for all Local Authorities in Britain<br />
(the top lines with over 2000 shops are : Birmingham, Bristol, Edinburgh, Leeds, Nottingham and London Borough of Westminster)</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis9OrPjQ6LpZmYhXKRWsMJ0AD6hoQ0_TVzLZz3mGk-DJSEqVYiGPuggMhFPJCEPlzsghBHQAMskhWq0eFtBErsdZUc-25Z-ElTj0RRebC6L7fFi94srWlbyW-SbE6r2jsQ60Easjm0C7Pd/s1600/shop_hist_2stdraft.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="660" data-original-width="1366" height="231" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis9OrPjQ6LpZmYhXKRWsMJ0AD6hoQ0_TVzLZz3mGk-DJSEqVYiGPuggMhFPJCEPlzsghBHQAMskhWq0eFtBErsdZUc-25Z-ElTj0RRebC6L7fFi94srWlbyW-SbE6r2jsQ60Easjm0C7Pd/s640/shop_hist_2stdraft.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Percentage of shops mapped at June 2017 in prior months, <br />
all Local Authorities Great Britain.</td></tr>
</tbody></table>
If we just look at the raw number of shops mapped, the accretion curve is more or less flat, with no sign of tapering off:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf95pDHJxjfrnQFHExHDV8hlugMPQrrkw9RC-kUi7YbjgpgkL7p82Ac9SYALcpXELyE2sL-k5ZIIr8v52MHC5ybtXMWv0eTgvuSbfE6FZU0p_H8P14LDLiXikloPLOpHp_AfU4yelTtyCP/s1600/shop_time_by_all.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf95pDHJxjfrnQFHExHDV8hlugMPQrrkw9RC-kUi7YbjgpgkL7p82Ac9SYALcpXELyE2sL-k5ZIIr8v52MHC5ybtXMWv0eTgvuSbfE6FZU0p_H8P14LDLiXikloPLOpHp_AfU4yelTtyCP/s400/shop_time_by_all.png" width="520" /></a></div>
<br />
Just looking at the top local authorities can highlight a few other features:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKYmphXb12sr_Ckp5G5K_WiH3fHszqByUsLarsTToxZl1qyTD01OqwD0Xeg70BIGBh_PQ_MflXtYfWiE8Unpu-lCETCllXdGWI3h_BuGKr13sqdExXgpPcZQ_FmaH8cW1408rxAxKbcLJJ/s1600/shop_s2000.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKYmphXb12sr_Ckp5G5K_WiH3fHszqByUsLarsTToxZl1qyTD01OqwD0Xeg70BIGBh_PQ_MflXtYfWiE8Unpu-lCETCllXdGWI3h_BuGKr13sqdExXgpPcZQ_FmaH8cW1408rxAxKbcLJJ/s320/shop_s2000.png" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Local Authorities with more than 2000 shops mapped mid-2017</td></tr>
</tbody></table>
<br />
Most of these places have seen a fairly steady increase in total shops mapped, but there are a few step changes:<br />
<ul>
<li>Birmingham, 2015: Mattijs Melissen (<a href="https://www.openstreetmap.org/user/Math1985">Math1985</a>)<span class="d-block Box"></span> was a very active shop mapper at this time. The tailing off subsequently may merely because he returned to The Netherlands on completing his post doc.</li>
<li>Edinburgh, 2014. The <a href="https://www.ed.ac.uk/history-classics-archaeology/research/research-projects/mapping-edinburgh-s-social-history">Edinburgh MESH</a> (social history) project were actively mapping the inner city during this period.</li>
<li>Nottingham, 2013. My own deliberate attempt to map most shops from around March to June 2013.</li>
</ul>
Clearly spurts of activity such as these are fairly typical of many areas. The extreme is Darlington where virtually all shops on OSM were mapped over at most a couple of months. Pulses of activity may therefore result in curves which are apparently asymptotic, but these only reflect individual mapper activity. I have not included counts of mappers touching elements tagged with shop, but this suggests some such metric may need to be used to avoid false positives from dedicated mapping of shops by single individuals.<br />
<br />
A much easier way to look at the data is by looking at graphs side by side. I selected all LAs with over 800 shops mapped, which gives a convenient set of 45 different ones. The graphs are at the head of the blog. Out of these 45, only 2 suggest there might be an asymptotic relationship in the data: Nottingham and Tendring. There are plenty more examples of many shops mapped over short periods (e.g., Gateshead, Sefton).<br />
<br />
<h4>
Shops by Tag</h4>
We can also look at whether there is any indication that we have mapped a given category of shop to exhaustion. Here are accumulation curves for the top 35 tag values (those with over 1000 elements mapped:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS_kIatjLCwmHSJkvc_bzXE0UWJnBqwZs1j5Lxj_AwYfgeGfB8Luv3yRMRdUeuJ03GFIKsgLxqzGZ_o7qNMlbYXmDpX01VaCVQXku6AUS0sumnYQ-1-jIgYoNGKRLmF-lu7V1HRjWBxc9j/s1600/shop_tags_facet_s1000.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS_kIatjLCwmHSJkvc_bzXE0UWJnBqwZs1j5Lxj_AwYfgeGfB8Luv3yRMRdUeuJ03GFIKsgLxqzGZ_o7qNMlbYXmDpX01VaCVQXku6AUS0sumnYQ-1-jIgYoNGKRLmF-lu7V1HRjWBxc9j/s400/shop_tags_facet_s1000.png" width="520" /></a></div>
Only shop=supermarket and shop=doityourself have any appearance of slowing down, and it would be a long stretch to say they were trending towards a given number.<br />
<br />
In pretty much all cases the accumulation curves are linear. Thus individual shop tags are much less vulnerable to individual mapper activity. The one step function is shop=bookmaker where Math1985 initiated an effort to reduce the number of synonyms. Slightly worrying is the steady increase in shop=yes.<br />
<br />
Lastly we can look at the total number of shop tags:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMIFC2EBtFHIyB22AyjD0M6wVEAY8cHmZCvY94qcxAOzcBQoDkJuFg_Tqxp01pc4eSYITULWzzPwHOJcZmzldVpzQ1YpR3AKHK_nkkitbLWnNEFiZNUfHsEZJyBFT4b_sHXMcaAqgewZkJ/s1600/shop_tags.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMIFC2EBtFHIyB22AyjD0M6wVEAY8cHmZCvY94qcxAOzcBQoDkJuFg_Tqxp01pc4eSYITULWzzPwHOJcZmzldVpzQ1YpR3AKHK_nkkitbLWnNEFiZNUfHsEZJyBFT4b_sHXMcaAqgewZkJ/s400/shop_tags.png" width="400" /></a></div>
This looks more like the kind of graph I had been hoping to see. The drop around early 2015 was again, no doubt, due to Matt85's rationalisation efforts. It perhaps suggests no more than 2000 tags are needed to map shops in the UK.<br />
<br />
In <a href="http://sk53-osm.blogspot.com/2015/05/retail-outlets-on-openstreetmap.html">previous work</a> I showed that there is a long tail of very low usage shop tags, and that the presence of these tags is usually in the noise (I was able to assign 98-99% of all shop tags to specific general categories). It struck me that removing this noise may provide a more informative graph. I therefore excluded any shop tag which had been used 5 times or less in June 2017:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihGS_cLI8YnxcOBVpUtNKceUUdhnWk8e7x3RINroV2Hi2YeHlSXIST692rhBS6RpSm3hrHGCbplOGnFMunIKdnJWXAKnpmSKclhz_onqaC8l23EBRL9DizKHZ9szaJiDvNMWEPEzVtVGTa/s1600/shop_tags_gt5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1600" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihGS_cLI8YnxcOBVpUtNKceUUdhnWk8e7x3RINroV2Hi2YeHlSXIST692rhBS6RpSm3hrHGCbplOGnFMunIKdnJWXAKnpmSKclhz_onqaC8l23EBRL9DizKHZ9szaJiDvNMWEPEzVtVGTa/s320/shop_tags_gt5.png" width="520" /></a></div>
Finally, I have the kind of graph I predicted. It actually looks as though 500 shop tags pretty much meets all our tagging needs.<br />
<br />
Even without trying to fit curves to the rest of the data, it is clear that even in well-mapped cities, the data from 2017 suggest that there are plenty of shops to map. It may be with more tightly constrained boundaries we may see more curves suggestive of saturation. I'll look at this in the next post.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com2tag:blogger.com,1999:blog-2082422783692631078.post-27395917091615330642018-04-25T21:18:00.002+01:002018-04-25T21:19:35.128+01:00Linear or 1D maps from OpenStreetMap<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/26829630157/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="1-D map of Clumber Street"><img alt="1-D map of Clumber Street" height="177" src="https://farm1.staticflickr.com/970/26829630157_f7647c0451.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Clumber Street, a pedestrian shopping street in Nottingham</td></tr>
</tbody></table>
<br />
We are all familiar with 1D, or linear, maps.<br />
<a name='more'></a><br />We see them in graphics at bus stops; use them to work out when it's our stop on the metro; there's even a <a href="https://www.theguardian.com/news/datablog/gallery/2013/jan/09/london-underground-alternative-tube-maps">whole genre</a> of using the style of the London Underground map for other purposes. Here are one or two examples: <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:KK_A_New_Service_Brochure_Map.gif" style="margin-left: auto; margin-right: auto;" title="By New York City Transit Authority (The JoeKorNer) [Public domain], via Wikimedia Commons"><img alt="KK A New Service Brochure Map" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/45/KK_A_New_Service_Brochure_Map.gif/512px-KK_A_New_Service_Brochure_Map.gif" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">New York Subway line map from 1960s</td></tr>
</tbody></table>
<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg" style="margin-left: auto; margin-right: auto;" title="By Niconaike [CC0], via Wikimedia Commons"><img alt="Extrait Tablede Peutinger Secteur Rhône et Alpes" src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d2/Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg/512px-Extrait_Tablede_Peutinger_Secteur_Rh%C3%B4ne_et_Alpes.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">An excerpt from the Tabula Peutingeriana </td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:R%C3%A9seau_Grand_R,_poteau_d%27arr%C3%AAt.jpg" style="margin-left: auto; margin-right: auto;" title="By P.poschadel [CC BY-SA 2.0 fr (https://creativecommons.org/licenses/by-sa/2.0/fr/deed.en)], from Wikimedia Commons"><img alt="Réseau Grand R, poteau d'arrêt" src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/02/R%C3%A9seau_Grand_R%2C_poteau_d%27arr%C3%AAt.jpg/256px-R%C3%A9seau_Grand_R%2C_poteau_d%27arr%C3%AAt.jpg" width="256" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Bus stop showing typical public transport use of 1D maps</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File:Jefferys%27s_itinerary;_or_travellers_companion_Fleuron_T221156-3.png" style="margin-left: auto; margin-right: auto;" title="By Jefferys, Thomas [Public domain], via Wikimedia Commons"><img alt="Jefferys's itinerary; or travellers companion Fleuron T221156-3" src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Jefferys%27s_itinerary%3B_or_travellers_companion_Fleuron_T221156-3.png/256px-Jefferys%27s_itinerary%3B_or_travellers_companion_Fleuron_T221156-3.png" width="256" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A 18th century road itinerary for Northumberland<br />
Jeffrey's <i>Fleuron </i>(perhaps also a strip map)</td></tr>
</tbody></table>
<br />
If you want to know more, <a href="https://twitter.com/Lorp">Laurence Penney</a>, from Bristol, gave a highly condensed version of <a href="http://www.youshouldliketypetoo.com/blog/misc/one-dimensional-maps/">his talk</a> on these maps at the international State-of-the-Map conference at
Birmingham back in 2013. He has a fantastic, and ever growing,
collection of examples going back at least as far as the <a href="https://en.wikipedia.org/wiki/Tabula_Peutingeriana">Tabula Peutingeriana</a> a
Roman scroll of road itineraries (as a 19th century facsimile. <br />
<br />
Like <a href="http://sk53-osm.blogspot.co.uk/2013/06/cartograms-and-openstreet-map-data.html">cartograms</a>, 1D maps were always a map style which I have wanted to create using OpenStreetMap data. Over the last few years I have made sporadic efforts to see how I might create them, and for quite some while I have had a reasonable notion for a workable approach. However, doing so required a fairly concerted effort to break through the little annoyances which occur at every step. In the past couple of days I have made just that effort, and seem to have something which is reasonable, and certainly meets my initial goals.<br />
<br />
<h4>
Why Now? </h4>
The reason why I knuckled down to the task was pragmatic.<br />
<br />
For several years we have spent an hour before our summer pub meetings in the East Midlands (Nottingham and Derby) mapping areas close to our meeting place. These partly help to improve OSM, but were also conceived as a way to show interested parties what was involved in mapping. This year I felt that it would be worthwhile to focus our energies in this hour on checking the centres of Derby & Nottingham for change. <br />
<br />
As OpenStreetMap data becomes richer the issue of maintaining existing data becomes more pressing. Amenities, shops and other points of interest in town and city centres are particularly prone to change and are often changes are often difficult to spot in any systematic way. It seems axiomatic that we should strive for a means which makes checking existing data somewhat less arduous than the effort it took to create in the first place, but often that is not true. Some editors, do help a bit: Vespucci, for instance, highlights objects which have not been touched by an edit within the last year.<br />
<br />
Clearly something which reduces the quantity of information to validate ought to help: hence 1D maps.<br />
<br />
<h4>
Outline of the technique.</h4>
As I stated above, I've had a reasonable idea of how to approach the problem for a while. These are the key steps I envisaged :<br />
<ul>
<li>Merge all ways belonging to a particular road (identified by a common name and/or ref).</li>
<li>Create a single centre line for the road (to eliminate spurs, dual carriageways, roundabouts, service roads etc).</li>
<li>Identify all POIs belonging to a given street: this might be because they have the appropriate street name or are within a given distance of the street centre line (corner buildings, for instance may belong to another street, but be equally prominent on the intersecting one). </li>
<li>Locate the POI along the centre line, and, importantly, which side of the line it is located.</li>
<li>Calculate distances along the centre line.</li>
</ul>
If this all goes according to plan one ends up with a tabulation like this:<br />
<div class="de1">
<br />
<table border="1" style="border: 1px black; width: 90%;">
<tbody>
<tr>
<th>Name
</th>
<th>POI Type
</th>
<th>Distance Along (m)
</th>
<th>Side of Road
</th>
<th>Distance Offset (m)
</th>
</tr>
<tr>
<td>Curry 77</td><td>restaurant</td><td>173</td><td>R</td><td>21</td></tr>
<tr><td>The Salutation</td><td>pub</td><td>192</td><td>R</td><td>24</td></tr>
<tr>
</tr>
<tr><td>Cumin</td><td>restaurant</td><td>218</td><td>L</td><td>18</td></tr>
<tr>
<td>Nottingham Credit Union</td><td>bank</td><td>227</td><td>R</td><td>29</td>
</tr>
</tbody></table>
</div>
<br />
This format gives enough data to work on a suitable rendering.<br />
<br />
In practice the second point is what took most time, so in the rest of the post I'll describe the steps in detail. I make no claims for elegance, and some of the steps I certainly didn't use the perhaps the best tools. Be warned there is <b>a lot</b> of nitty-gritty stuff below!<br />
<h3>
Step by Step guide</h3>
<h4>
The Road Network & POIs</h4>
I used two Overpass-Turbo queries to download all highway ways and a discrete set of POIs (all amenities, shops and offices) from the same bounding box. Data was downloaded as Geojson and immediately uploaded into PostGIS (using the QGIS DBManager). I converted everything to the British Grid on upload which simplifies distance calculations, but also gives some 'safe' areas to locate transformed data at the end.<br />
<br />
Next I grouped all highways with the same name into multilinestrings:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">SELECT nextval('street_seq') street_id<br /> , name<br /> , st_multi(st_union(geom)) geom<br /> FROM hwy_upload<br /> GROUP BY name</span></span></blockquote>
The additional <span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">st_multi</span></span> is needed because most roads will still be a single linestring when merged with the union function. Operations also need to work with more complex roads, so they are all treated in the same way.<br />
<h4>
Generating a Centre Line 1 : creating an approximate medial axis </h4>
The first operation in generating a central line is to buffer the entire multilinestring by some arbitrary value (20 metres worked for me), and then use <a href="http://postgis.net/docs/ST_ApproximateMedialAxis.html">st_approximatemedialaxi</a>s (discussed extensively in an <a href="http://sk53-osm.blogspot.co.uk/2016/10/skeletons-in-water.html">earlier post</a>) to create a first cut medial axis.<br />
<br />
There are several problems with the medial axis generated by the PostGIS functions:<br />
<ul>
<li>In many cases it cant be resolved to a line. </li>
<li>It's a multilinestring (I think even if the linear segments can be merged into a linestring).</li>
<li>It appears that it is constructed of many 2-point lines.</li>
<li>Some of the piecewise linear segments are minute (nanometres in length IIRC)</li>
<li>There can be many short or very short stubs.</li>
</ul>
These problems were what stymied me when I first looked into using the medial axis.<br />
<br />
Some can be reduced by using <span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><a href="https://postgis.net/docs/ST_SnapToGrid.html">st_snaptogrid</a>.</span></span> Using a 1 m snapping grid is absolutely fine for my purposes and eliminates some of the more absurdly short segments. After considering various ways of pruning the medial axis graph I instead decided that routing through it was easier to try.<br />
<h4>
Generating a Centre Line 2: using pgrouting </h4>
As I had the data in PostGIS <a href="http://pgrouting.org/">pgrouting</a> was a fairly obvious candidate to generate routes. It is a relatively lightweight set of tools for generating routes and trivial to install<br />
<br />
To prepare the data for pgrouting I decomposed the medial axis multtistrings using st_dump and stored them in a new table.<br />
<br />
I also extracted all the points (node) from the original road linestrings and for each street found the pair of points which were most distant from each other. These two points would be used later as the source and target nodes for routing:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">WITH st_pt_distance AS (<br /> SELECT street_id, <br /> a.pt_id source_pt_id, b.pt_id target_pt_id, <br /> a.pt_geom source_pt_geom, b.geom target_pt_geom,<br /> st_distance(a.pt_geom, b.pt_geom) distance<br /> FROM hwy_pts a, hwy_pts b<br /> WHERE a.street_id = b.street_id<br /> AND a.pt_id <b.pt_id<br /> )<br />SELECT street_id, <br />FROM st_pt_distance <br />WHERE distance = (SELECT street_id, max(distance)<br /> FROM st_pt_distance<br /> GROUP BY street_id)</span></span></blockquote>
As I only want routing along an individual street it was not obvious if I could safely use a single table for pgrouting. I therefore decided to use a temporary table for each street, and iterate over each street using a Postgres stored procedure. Streets with inconvenient geometries (such as those having two exterior rings when buffered) are eliminated at this step.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3zPnFgr2gr2BWdM6X3mFWcOQzMUjrox8tTe8jGKSviNo1Ptg2ZtIoPSyD2AzuyA8vuYyP1fhNc8UzmMIrKRAbIwtFs-bCJbK7S389UL9-4U6JiDP_FYDCrCcYoifUM_Ot5J9iZ-OYQWID/s1600/ng_roads1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="897" height="288" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3zPnFgr2gr2BWdM6X3mFWcOQzMUjrox8tTe8jGKSviNo1Ptg2ZtIoPSyD2AzuyA8vuYyP1fhNc8UzmMIrKRAbIwtFs-bCJbK7S389UL9-4U6JiDP_FYDCrCcYoifUM_Ot5J9iZ-OYQWID/s320/ng_roads1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Examples of buffered roads with rejected geometries (mostly in 2 or more parts).<br />
Some are correct others needed fixing</td></tr>
</tbody></table>
The basic means of generating a route was as follows:<br />
<ul>
<li>Create the temporary routing table: <span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><br />CREATE temp_table AS<br /> SELECT a.street_id, (st_dump(aml_geom)).path street_seg_id,<br /> (st_dump(aml_geom)).geom as geom,<br /> st_length(st_dump(aml_geom)).geom) as cost,<br /> </span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">st_length(st_dump(aml_geom)).geom) as reverse_cost<br /> FROM hwy_upload<br /> WHERE a.street_id = <value></span></span></span></span></li>
<li><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-size: small;">Prepare the table for pgrouting:</span></span><br />SELECT pgr_createtopology('temp_table', 0.0000001,'geom'); <br /><br /><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;">This routine creates another table <span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">temp_table_vertices</span></span> which contains the nodes (vertices) of the routing network). Useful to remember when cleaning up the temporary tables. (I'm not sure why I used such a small value for the second parameter. I think I was worried about the very short segments which I later eliminated).</span></span></span></span></li>
<li><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;">Create a route. I used the standard <a href="http://docs.pgrouting.org/latest/en/pgr_dijkstra.html#pgr-dijkstra"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">pgr_dijkstra</span></span></a> call which returns a list of all vertices and edges making up the route:<br /><br /><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">SELECT * <br />FROM pgr_dijkstra('select street_seg_id,<br /> source, <br /> target, <br /> cost, <br /> reverse_cost from temp_table',<br /> (SELECT id FROM temp_table_vertices_pgr<br /> WHERE geom = (SELECT start_pt from hwy_upload <br /> WHERE street_id =<value>)),<br /> (SELECT id FROM t</span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">emp_table_vertices_pgr</span></span></span></span></span></span><br /> WHERE geom = (SELECT end_pt FROM </span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">hwy_upload <br /> WHERE </span></span></span></span></span></span></span></span></span></span></span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">street_id =<value></span></span></span></span></span></span>))</span></span></span></span> </span></span></li>
<li>In principle I should have been able to just use <a href="https://postgis.net/docs/ST_LineMerge.html"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">st_linemerge</span></span></a> to assemble the parts of the route, but I found some small gaps which prevented the function from working. Instead I used the ordered set of nodes and made the line from them with <a href="https://postgis.net/docs/ST_MakeLine.html">st_makeline</a>. This approach rather relies on the medial axis segments only having 2 points. (This may have been related to the short segments, but I haven't tried the original technique since then):<br /><br /><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">SELECT st_makeline(pts) geom from (<br /> SELECT c.*<br /> FROM <br /> ((SELECT b.street_seg_id,seq,node,edge,agg_cost, geom,<br /> st_pointn(b.geom,1) pts <br /> FROM pgr_dijkstra(<see above>) a <br /> JOIN hwy_aml_dump b on edge=</span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">street_seg_</span></span>id)<br /> ...</span></span></li>
</ul>
And <b>VOILA</b>! we have a centre line, almost home and dry.<br />
<h4>
Adding the POIs</h4>
It is now trivial to calculate position along the centre line for each of the selected POIs using <a href="https://postgis.net/docs/ST_LineLocatePoint.html"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">st_linelocatepoint</span></span></a>:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;">SELECT street_id, poi_id, <br /> st_linelocatepoint(<br /> st_pointn(<br /> st_shortestline(poi.geom,street.centreline_geom),<br /> 2),<br /> centreline_geom)<br /> * st_length(centreline_geom) distance_along_street<br /> FROM streets street, pois poi<br /> WHERE street.street_id = poi.street_id</span></span></blockquote>
<span style="font-family: "times" , "times new roman" , serif;"><span style="font-size: small;">The offset from the centre line is given by st_shortestline, but we need one other piece of information: where the POI is to the left or right of the centre line. Unfortunately it is not possible to rely on the shortest line dropped to the street centre line. It needs to be extended slightly to ensure it crosses the street centre line. Doing this is fairly painful in PostGIS (best managed with a simple function, see <a href="https://gis.stackexchange.com/questions/104439/how-to-extend-a-straight-line-in-postgis">GIS Stack Exchange</a> for examples):</span></span><br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">SELECT st_makeline( /* line from POI to 5 m across centre line) */<br /> geom, /* POI location */<br /> st_translate(geom, /* xsect_geom is intersection point */<br /> sin(st_azimuth(geom, xsect_geom ) * <br /> (st_length(st_shortestline(geom,</span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">xsect_geom </span></span>))+5),<br /><br /> cos(st_azimuth(geom, </span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">xsect_geom </span></span> ) * <br /> (st_length(st_shortestline(geom,</span></span><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">xsect_geom </span></span>))+5))<br /> FROM pois</span></span></blockquote>
Now the <a href="https://postgis.net/docs/ST_LineCrossingDirection.html">st_linecrossingdirection</a> function works every time, and because these are simple straight lines we can assign POIs to left or right.<br />
<h4>
Pulling it together</h4>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDoDyoY4rKCkY-jHqjvv8tsLTXvDX3AVy8dyWYBwaNtLEYwoCgo-4CbBLLWOXISPFD2LWQFNVr1QX8_yriHEBBvLmzNe_gLaEHw7-GHcZ03es7KuWvxPqrLh7yb1rPck4FDgUIcXib1ulK/s1600/ng_roads2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="897" height="288" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDoDyoY4rKCkY-jHqjvv8tsLTXvDX3AVy8dyWYBwaNtLEYwoCgo-4CbBLLWOXISPFD2LWQFNVr1QX8_yriHEBBvLmzNe_gLaEHw7-GHcZ03es7KuWvxPqrLh7yb1rPck4FDgUIcXib1ulK/s320/ng_roads2.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;"><span style="font-family: "times" , "times new roman" , serif;">Maid Marian Way in <a href="https://www.openstreetmap.org/way/102189757#map=16/52.9516/-1.1518">central Nottingham</a> <br />showing the calculated central line (orange) <br />and short lines (red) from POIs.<br />This is a topologically complex road.</span></span></td></tr>
</tbody></table>
Now all the components exist and it's possible to generate the tabulation of POIs shown above. This is in itself quite useful, we used my <a href="https://www.dropbox.com/s/soc802n8ynn53s7/nottingham_pois_by_road.csv?dl=0">initial tabulation</a> a little last night until the rain made paper-based mapping infeasible. However, the original goal is some kind of more usable visualisation such as that shown at the head of the post. I'll discuss how I used QGIS to achieve this in the next post.<br />
<br />
I will add queries & other code to github in the near future, probably as GISTs for now. Much of this was driven by wanting something to hand for our first mapping evening, and in many cases I went with what I could get to work quickly rather than investigating why a given result was unexpected.<br />
<br />
One other unexpected benefit is that I discovered a few bugs in OSM data. As is often the case when one looks at a larger set of data inconsistencies and minor errors stand out rather quickly.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com2tag:blogger.com,1999:blog-2082422783692631078.post-48544086164999624862017-08-30T22:25:00.001+01:002017-08-30T22:30:27.571+01:00Mapping a specific building form<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXBjPg8h4mMwxqkducRT3eiQikeemXbcbihUzgYbTqItgmA4USNjmQqpTecHRoy8CwFwoS2FINX0gz8mEM38W0Bv0bncbrahGaORAtqkFdUeQcrL3JQP0aG5qtw0iPj5A5TC1xm8Hfiuii/s1600/IMG_1158a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXBjPg8h4mMwxqkducRT3eiQikeemXbcbihUzgYbTqItgmA4USNjmQqpTecHRoy8CwFwoS2FINX0gz8mEM38W0Bv0bncbrahGaORAtqkFdUeQcrL3JQP0aG5qtw0iPj5A5TC1xm8Hfiuii/s320/IMG_1158a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Arts-and-Craft style semi-detacheed houses, Edwards Lane Estate, Nottingham</td></tr>
</tbody></table>
<span id="goog_1201215879"></span><span id="goog_1201215880"></span><br />
My interest in many aspects of urban environments has increased greatly since I started contributing to OpenStreetMap.<br />
<br />
I suppose this was always there but largely latent. Wandering around familiar places to capture details to add to OSM often forces me to ask questions about the area. Why is it there? Why is it laid out in that way? Who designed the buildings? When was it built? Why are there gaps in house numbering? What was planned for the little stub street? What used to be on the land with newer houses?<br />
<br />
<a name='more'></a>A particular class of urban areas which I find interesting are social housing estates. I've written a little about these before: <a href="http://sk53-osm.blogspot.co.uk/2011/01/twitchell_13.html">here</a> and <a href="http://sk53-osm.blogspot.co.uk/2011/04/radburn-exemplar-or-nightmare.html">here</a>. Fortunately I am not alone in this interest. Planners, Architects, Social Historians, and others share it. As a result there is more documentation readily available to satisfy casual queries. In the lead is an inspirational blog called <a href="https://municipaldreams.wordpress.com/">Municipal Dreams</a>. Although this has, an understandable, London bias, <a href="https://twitter.com/MunicipalDreams">John Boughton</a> has managed to cover plenty of other cities across the country. Other blogs may cover individual estates in detail, for instance <a href="https://twitter.com/iwaites60">Ian Waites</a>' <a href="http://instancesofachangedsociety.blogspot.co.uk/">blog</a> on the Middlefield Estate in Gainsborough, . For Nottingham, I'm lucky that <a href="https://twitter.com/chrismatthew82">Chris Matthews</a> has written (an all too short) history of council housing in the city.<br />
<br />
This particular blog post originated when Municipal Dreams tweeted a picture of a characteristic Arts-and-Craft style house in Osmaston, Derby:<br />
<br />
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Another Addison Road, this one in Derby. Built, I would guess, under his great 1919 Housing Act. <a href="https://t.co/CwAaroDUcF">pic.twitter.com/CwAaroDUcF</a></p>— Municipal Dreams (@MunicipalDreams) <a href="https://twitter.com/MunicipalDreams/status/888819536807514112">July 22, 2017</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<br />
This house looked very familiar: there are hundreds of similar houses scattered across Nottingham on estates built by the city council between the 1920s and the 1940s. It prompted me to systematically add all such houses to OSM. So far, I may have missed a few, I have found over 900 across the city as shown in the map below.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoNx6kIOnkaUZWQ8dbX5ASCUtE0b8j06xUm606r1w9Gclv6ikadSSCpNwiqYU7uoCGVB5QNjGUCmVau_8QYt4m4r6L8plmhRl9TIad8Is7kcHBZQfMdgkOXqibPBsl4DcXvTE0fZfVj9cP/s1600/corner_semi_locs.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1132" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoNx6kIOnkaUZWQ8dbX5ASCUtE0b8j06xUm606r1w9Gclv6ikadSSCpNwiqYU7uoCGVB5QNjGUCmVau_8QYt4m4r6L8plmhRl9TIad8Is7kcHBZQfMdgkOXqibPBsl4DcXvTE0fZfVj9cP/s320/corner_semi_locs.jpg" width="452" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Distribution of this particular building style across the city of Nottingham.</td></tr>
</tbody></table>
<br />
Just adding the footprints of the houses is not enough. Additional tags are needed to be able to identify this particular design. At present I am using personal tags as an interim measure. I hope that ultimately it will be possible to accurately identify the design reference used by the Nottingham City Housing department (see examples in <a href="https://twitter.com/Alexander_Ball">Alex Ball</a>'s post on their early work). I am documenting my approach on the wiki.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ASherwood_Perry_Road_2235.jpg" style="margin-left: auto; margin-right: auto;" title="By ClemRutter (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons"><img alt="Sherwood Perry Road 2235" src="https://upload.wikimedia.org/wikipedia/commons/thumb/a/ac/Sherwood_Perry_Road_2235.jpg/512px-Sherwood_Perry_Road_2235.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">One of the earliest examples of the design. Perry Road, Sherwood Estate.<br />
Later examples (see below and above) either have hanging tiles covering first floor at the front, or are not rendered at all.</td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjj0QEf7LlZIYiPBJB9ZxKVc3DeTjIsEYmhcBWbCEW1KNJNJIhQQbFtY_bzGpoaT3vgiDCYCMAAVn-2W1C4cdA6YYnxOvBsqBxC_1ml3fs4POuIFhX4bQ7zKIkq8zkkIwEK64IexMuSSf5/s1600/P1020760-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjj0QEf7LlZIYiPBJB9ZxKVc3DeTjIsEYmhcBWbCEW1KNJNJIhQQbFtY_bzGpoaT3vgiDCYCMAAVn-2W1C4cdA6YYnxOvBsqBxC_1ml3fs4POuIFhX4bQ7zKIkq8zkkIwEK64IexMuSSf5/s320/P1020760-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Houses on Charnock Avenue, Wollaton Park Estate<br />
Only 4 pairs of houses of this type were built on this estate.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwEcILp0OnrRd4Xi1e1pWvMB1r5wS85R5gTZ2qaFyX-EXnta1T6gzoLDeIrsDTkxZq2CN1o0qeibBueDEWq5vPdlokuUrEMOsZojrFbuJayPoxpwb3qP9OIxSSxq_tB8ruQqE1BZtv0ImI/s1600/P1020758-1-2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwEcILp0OnrRd4Xi1e1pWvMB1r5wS85R5gTZ2qaFyX-EXnta1T6gzoLDeIrsDTkxZq2CN1o0qeibBueDEWq5vPdlokuUrEMOsZojrFbuJayPoxpwb3qP9OIxSSxq_tB8ruQqE1BZtv0ImI/s320/P1020758-1-2.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Side view of Charnock Avenue Houses.</td></tr>
</tbody></table>
Using Environment Agency Lidar open data and <a href="https://wiki.openstreetmap.org/wiki/Simple_3D_buildings">Simple 3D Building</a> tags I've had a go at capturing the form of a couple of these houses too: the image below shows how they are rendered on F4 Map.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJjCj9IJMlkdTHTdz9Y2EtOyItGFx1x8op5WDGz7hz_YQSqnS9KwfcUwG-bmAuQdLRLYFr9ED78Q2GyCA9UN8KpCtJ3P6HdWuYB7q5h1TVmAYSQh0-_F1u0naktTyLCdqQfY_hhVetQ67r/s1600/Screenshot+%252869%2529-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="934" data-original-width="1024" height="291" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJjCj9IJMlkdTHTdz9Y2EtOyItGFx1x8op5WDGz7hz_YQSqnS9KwfcUwG-bmAuQdLRLYFr9ED78Q2GyCA9UN8KpCtJ3P6HdWuYB7q5h1TVmAYSQh0-_F1u0naktTyLCdqQfY_hhVetQ67r/s320/Screenshot+%252869%2529-1.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Screengrab from <a href="http://demo.f4map.com/#lat=52.9482375&lon=-1.1864589&zoom=21&camera.theta=65.054&camera.phi=58.671">F4 map</a>.<br />
The S3DB rules for the two houses are slightly different.</td></tr>
</tbody></table>
<br />
<br />
Given that the houses were built over a period of about 20 years it can be awkward to date them. In addition to old maps and various information in the Matthews book and on-line, there are archival aerial photographs of the estates largely taken shortly after they were built.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOXIn2RGekPotrfbsnVQ6mUe-32zcX7XcglJ5FetKJyMaPVjbkQEA5P-P0m9K1A5mbNnHTa4zRy3bWMZjOdlhzSOoRdJzvv0KKZnfTjijlgym-G6CWl9yc-QWzEU3dYQfIBe1zaHxp2LNX/s1600/Bulwell_Hall_Estate_EPW034135.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="491" data-original-width="580" height="405" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOXIn2RGekPotrfbsnVQ6mUe-32zcX7XcglJ5FetKJyMaPVjbkQEA5P-P0m9K1A5mbNnHTa4zRy3bWMZjOdlhzSOoRdJzvv0KKZnfTjijlgym-G6CWl9yc-QWzEU3dYQfIBe1zaHxp2LNX/s320/Bulwell_Hall_Estate_EPW034135.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/way/324227126#map=16/53.0150/-1.1913">Bulwell Hall Estate</a> photographed from the South-west 1930. <br />
The semi-detached houses discussed in this post can be seen at the far end of the road in the foreground.<br />
Source: <a href="https://britainfromabove.org.uk/en/image/EPW034135">Britain from Above</a>.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ-H3AOoQgTuFhP9BR8VZJefoQFmLLubRyQwEDEMv9JY57tbXuT1NiiJE0jAUffFCPpg_TAF3H_cpYKjc-iqJKkNPZp8c3sA-03xifZWRMA7rex0akSjvNwMnhJBijWidPTfRROyeabOxo/s1600/Aspley_Estate_EPW036010.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="488" data-original-width="580" height="405" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ-H3AOoQgTuFhP9BR8VZJefoQFmLLubRyQwEDEMv9JY57tbXuT1NiiJE0jAUffFCPpg_TAF3H_cpYKjc-iqJKkNPZp8c3sA-03xifZWRMA7rex0akSjvNwMnhJBijWidPTfRROyeabOxo/s320/Aspley_Estate_EPW036010.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/node/469813521#map=16/52.9759/-1.2000">Aspley Estat</a>e from the S in 1931.<br />
Again the houses are most prominent in the corners of street intersections.<br />
Source: <a href="https://britainfromabove.org.uk/en/image/EPW036010">Britain from Above</a>. </td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLgqHkJxERiZwWsm7K3ifABKul6gk6xYTPtkhy6R0rCDyLorK3YLD26miwonjqFB5l7p47PdvqurHrV-JTpQONi6k1f8D5PwV5DOn4xbLnZTSezdDYghNJtp1JlUhx1P1xqr_wK_YxB70P/s1600/Sherwood_EPW021828.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="481" data-original-width="580" height="397" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLgqHkJxERiZwWsm7K3ifABKul6gk6xYTPtkhy6R0rCDyLorK3YLD26miwonjqFB5l7p47PdvqurHrV-JTpQONi6k1f8D5PwV5DOn4xbLnZTSezdDYghNJtp1JlUhx1P1xqr_wK_YxB70P/s320/Sherwood_EPW021828.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/#map=17/52.98554/-1.15113">Sherwood Estates</a> from the SW in 1928.<br />
The rendered houses shown above are in front of the school centre left.<br />
Source: <a href="https://britainfromabove.org.uk/en/image/EPW021828">Britain from Above</a>.</td></tr>
</tbody></table>
These pictures also give a good impression of the sheer scale of Nottingham's social housing programme between the wars. These three estates were more or less complete by 1930 and several others followed before the start of WWII. For instance the Broxtowe Estate consists of <a href="http://overpass-turbo.eu/s/rmQ">around 1700 houses</a>. It's difficult to determine the exact year of construction, but surprisingly easy to identify the decade, so for some I have tagged them with decade_built=*.<br />
<br />
Mapping this level of detail: a single design for a single type of house is feasible within a city and when they are all built by a single organisation. It may also be suitable for many of the private housing developments built post-war by national firms (Wimpey, Taylor Woodrow, etc) as they often used identical designs. It becomes more problematic when very similar, but not identical houses turn up across the country. Being alerted to this by Municipal Dreams, I noted a pair at <a href="http://www.openstreetmap.org/#map=18/52.96562/-1.30327">the junction</a> of Park Drive and Nottingham Road in Ilkeston, and then I came across these houses in Station Road Awsworth:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsoii1TjsbjsgOihUb4rvkAy7Qir0h6STYoJoNEOf-naUMLj4pyi6eN4YFJ1Aq1oUILNB-rmtI2kMVE3ZqvymuPBLJLeqM0fTJk0SMctnXlnj3HQcSr6vfbBYX11gYi-dS4OEELQ2oxj4W/s1600/P1030997-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsoii1TjsbjsgOihUb4rvkAy7Qir0h6STYoJoNEOf-naUMLj4pyi6eN4YFJ1Aq1oUILNB-rmtI2kMVE3ZqvymuPBLJLeqM0fTJk0SMctnXlnj3HQcSr6vfbBYX11gYi-dS4OEELQ2oxj4W/s320/P1030997-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.openstreetmap.org/#map=18/52.99328/-1.28342">Station Road, Awsworth</a>. Note minor differences (absence of bay window, single window at side etc)</td></tr>
</tbody></table>
Although we can identify these houses with a range of tags:<br />
<ul>
<li>building=semi-detached,</li>
<li>building:architecture=arts_and_crafts,</li>
<li>decade_built=1920,</li>
<li>architect=* (if known),</li>
<li>developer=* (or building:developer);</li>
</ul>
it is still very difficult to envisage a single typology which is likely to satisfy the demands of mappers, historians of social housing, architecture and planning. However, I hope I have shown here that much is possible with OSM data at a local level.<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-60206181784285255372016-10-24T12:40:00.004+01:002016-10-24T12:40:53.746+01:00Using Open Data for Statistical Purposes<div class="separator" style="clear: both; text-align: center;">
</div>
A <a href="https://twitter.com/owenboswarva/status/789541162583490561">tweet</a> by <a href="https://twitter.com/owenboswarva">Owen Boswarva</a> drew my attention to a <a href="https://publichealthmatters.blog.gov.uk/2016/10/21/obesity-and-the-environment-the-impact-of-fast-food/">recent report</a> by <a href="https://twitter.com/PHE_uk">Public Health England</a> (PHE) on the correlation of density of fast food outlets and deprivation.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheWCxUoQyCWIXAyYl9HYi-9OTMFiVZnRlns2PZaUOvEsybANjkJwYkNK4_mv899nyhzRhQ-PFhbKtxZ-r1DYJVUvtFbRYW4TfvQ3qdLdq_ILIVgJZDK-8mR4edFihA_mQ9Bk8OUO-ylkO9/s1600/ff_outlet_density_fhrs.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheWCxUoQyCWIXAyYl9HYi-9OTMFiVZnRlns2PZaUOvEsybANjkJwYkNK4_mv899nyhzRhQ-PFhbKtxZ-r1DYJVUvtFbRYW4TfvQ3qdLdq_ILIVgJZDK-8mR4edFihA_mQ9Bk8OUO-ylkO9/s320/ff_outlet_density_fhrs.jpg" width="450" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets normalised to 100,000 population for Local Authorities in England<br />
Source: Food Hygiene Rating Scheme (Takeaway class)</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
Specifically my interest was directed at the source of fast food outlet counts. PHE used data from <a href="http://www.pointx.co.uk/about.htm">PointX</a>, a joint venture of Landmark Information and the Ordnance Survey. I instantlly wondered if one could do the same thing with <a href="http://ratings.food.gov.uk/">Food Hygiene Ratings</a> (FHRS) open data. This is a quick report on doing exactly that.<br />
<br />
I already had a complete set of FHRS data for September 2016. I needed to download various administrative and census geographies, population figures for <a href="https://en.wikipedia.org/wiki/ONS_coding_system">Lower Layer Super Output Areas</a> (LSOAs), <a href="https://en.wikipedia.org/wiki/Indices_of_deprivation_2007">Index of Multiple Deprivation</a> (IMD) Scores for LSOAs and various files showing the linkages between the geographies.<br />
<br />
A certain amount of data wrangling was needed to merge this data (for instance linkages, population and IMD) all came in spreadsheets with awkward column names, multiple sheets and other minor inconveniences. Once these were sorted out I had a table with base figures at LSOA level which could be readily aggregated to Middle Layer Super Output Areas (MSOAs) and local authorities. The IMD score is rebased by summing LSOA scores multiplied by population and then dividing by total population.<br />
<br />
Using R I constructed simple scatter plots with a regression line and 95% confidence limits for both MSOA and Local authorities.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivwthKpqdKUOmD1TENCQsFisCAoIO_E4dPWtB24n2W0u6aPRCaPByfLmQb2SLa6M8MBukBJRLeQqqtlx4j-folt_AZKmqyowccvVGHH5W5O5vFCjGxplS9oOVeGRx3GI5sPtWd0vFfeEmj/s1600/imd_msoa_plot-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivwthKpqdKUOmD1TENCQsFisCAoIO_E4dPWtB24n2W0u6aPRCaPByfLmQb2SLa6M8MBukBJRLeQqqtlx4j-folt_AZKmqyowccvVGHH5W5O5vFCjGxplS9oOVeGRx3GI5sPtWd0vFfeEmj/s320/imd_msoa_plot-1.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets (normalised) vs calculated <br />
Index of Multiple Deprivation for Middle Super Output Areas</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjVSlG7MNI-ax-I1LOXS7PbubNeDM8exiQiN1NetyJYV4BuP3vkgOgdYgZgTrOhZfIHVjQWG8MMQ_VxPgorRimjcV5lezT3vy0PO76E4iiIC577JyQCqkLcZ4lwVqW2lGx5FUKWBQ47UXE/s1600/imd_plot2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjVSlG7MNI-ax-I1LOXS7PbubNeDM8exiQiN1NetyJYV4BuP3vkgOgdYgZgTrOhZfIHVjQWG8MMQ_VxPgorRimjcV5lezT3vy0PO76E4iiIC577JyQCqkLcZ4lwVqW2lGx5FUKWBQ47UXE/s320/imd_plot2.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Number of Fast Food outlets (normalised) vs calculated <br />
Index of Multiple Deprivation for Local Authorities<br />
(outlier of City of London excluded)</td></tr>
</tbody></table>
<br />
For comparison the relevant plot from the PHE report is shown below:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://publichealthmatters.blog.gov.uk/wp-content/uploads/sites/33/2016/10/fast-food-graph.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="405" src="https://publichealthmatters.blog.gov.uk/wp-content/uploads/sites/33/2016/10/fast-food-graph.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Scatter plot from PHE report for Local Authorities</td></tr>
</tbody></table>
<br />
The final comparison I made was perhaps one I should have done at the outset. Comparing raw counts of fast food outlets from the Open Data source (FHRS) and the PointX data. PHE provided a table of counts at ward level. It took me a while to find a shape file and codes which fitted (the codes change year-on-year), but then it was easy to do a Point-in-Polygon count of the FHRS data for a direct comparison. The correlation of values was plotted in R again.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXftxkaPN_7u1vF2HBJHh_jXc5jkFvAtZZE1V1DxCOoKHiDsR2PFVFp4nFqnfy5KomP-z8BgAhKWcHDGW8Pqe3tequsCzXdcywfWJK3Oun0tkEBfjjCraOVJPV830ECoCX9V_BsujeSlGy/s1600/ff_plot2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXftxkaPN_7u1vF2HBJHh_jXc5jkFvAtZZE1V1DxCOoKHiDsR2PFVFp4nFqnfy5KomP-z8BgAhKWcHDGW8Pqe3tequsCzXdcywfWJK3Oun0tkEBfjjCraOVJPV830ECoCX9V_BsujeSlGy/s320/ff_plot2.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of number of Fast Food outlets by 2015 ward boundaries<br />
derived from Food Hygiene Data or from Landmark/Ordnance Survey</td></tr>
</tbody></table>
<br />
Doing this took longer than I hoped: but almost entirely because I don't know my way around the various formats of boundary data related to the census and more changeable boundaries such as the wards.<br />
<br />
I haven't done a formal comparison of the outputs, but the visuals presented above strongly suggest that FHRS data is just as useful as the PointX data for this purpose. The main explanation for the lower count coming from FHRS is that the PointX data includes outlets which do food delivery which may include places classified as Restaurants in FHRS.<br />
<br />
I had expected more issues with FHRS because there is clearly an under-reporting issue in inner city areas due to rapid turnover of management of takeaways (see the <a href="https://www.theguardian.com/world/2016/sep/25/day-in-life-food-safety-inspector-newham-east-london-hygiene">recent Guardian article</a> for an in-depth appreciation of this issue). The other week at the <a href="http://wiki.openstreetmap.org/wiki/London#Past_events">London OpenStreetMap</a> pub meeting in Islington I insisted that we should check the 'scores-on-the-doors' before choosing where to eat our Burritos (a habit I've learnt from <a href="https://twitter.com/drsiant">Dr Sian Thomas</a>). The three fast food outlets next to the pub didn't feature at all on the FHRS data.<br />
<br />
In conclusion: now that FHRS data covers nearly every major authority in the country (Rutland were the last still hold out) it is entirely suitable for a range of statistical purposes.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-6008134696771961252016-10-07T11:48:00.001+01:002016-10-07T11:48:23.706+01:00Skeletons in the WaterFor a number of years now I have, from time-to-time, made the odd stab at trying to find the flowline of a river from the mapped surface area of the watercourse using OpenStreetMap data.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AWindermere_Lake_District_from_hill.JPG" style="margin-left: auto; margin-right: auto;" title="By Abbasi1111 (Own work) [CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons"><img alt="Windermere Lake District from hill" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/20/Windermere_Lake_District_from_hill.JPG/512px-Windermere_Lake_District_from_hill.JPG" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windermere in the English Lake District, one of my test cases.</td></tr>
</tbody></table>
I not infrequently find, being neither trained as a geospatial specialist nor a mathematician, that, although I have a fairly clear idea of what I want to do with some particular manipulation of geodata, I am stymied. More often than not this is simply because I don't know the most widely used term for a particular technique. It was therefore really useful to learn from <a href="http://www.imagico.de/">imagico </a>that the generic term for what I was trying to do is <a href="https://en.wikipedia.org/wiki/Topological_skeleton"><b>skeletonisation</b></a>. (I do hope my relative ignorance is not on <a href="https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/">this </a>scale.)<br />
<br />
Armed with this simple additional piece of knowledge immediately opened out the scope of resources available to me from wikipedia articles, blog posts, to software implementions. Unfortunately when I first tried to get the relevant extensions (<a href="http://www.sfcgal.org/">SFCGAL</a>) installed in PostGIS I was not able to get them to work, so I shelved looking at the problem for a while.<br />
<br />
Very recently I re-installed Postgres and Postgis from scratch with the latest versions and the <a href="http://postgis.net/docs/reference.html#reference_sfcgal">SFCGAL extensions</a> installed fine. So it was time to re-start my experiments.<br />
<br />
Once I was aware of skeletonisation as a generic technique I also recognised that it may be applicable to a number of outstanding issues relating to post-processing OpenStreetMap data. Off the top of my head & in no particular order these include:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/4647841258" style="margin-left: auto; margin-right: auto;" title="Wiggly River Trent"><img alt="Wiggly River Trent" height="438" src="https://c3.staticflickr.com/5/4005/4647841258_9d9f480ef7.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My earliest experiment using Ordnance Survey Open Data for the River Trent<br />
Voronoi triangles based on modes of polygon, clipped back to polygon</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
<ul>
<li><b>Waterway flowlines</b>. Replacing rivers mapped as areas by the central flowline where such a flowline has not already been mapped. Such data can then be used for navigation on river systems or for determining river basins (and ultimately watersheds/hydrographic basins). (It is this data which much of the rest of the post is concerned with).</li>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVfytyoQEYreYjoLypQlRhBv0G-Gp466tBiFZ5Ns2U5O8llVSouDFt-2olNEgnBry74rsZw_EtUcykCG2afBhLRxXUUK_zZZbmHg_zcIZuYIG4_ljEFj60Mu1NDly2jxxfkWUVFZQQCfyi/s1600/annapurna_himal_eg.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVfytyoQEYreYjoLypQlRhBv0G-Gp466tBiFZ5Ns2U5O8llVSouDFt-2olNEgnBry74rsZw_EtUcykCG2afBhLRxXUUK_zZZbmHg_zcIZuYIG4_ljEFj60Mu1NDly2jxxfkWUVFZQQCfyi/s320/annapurna_himal_eg.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Earlier experiments with OpenStreetMap glacier data for the <a href="http://www.openstreetmap.org/#map=13/28.5394/83.8310">Annapurna region</a><br />
Height (contours) & slope(shading) data via Viewfinderpanorams.com<br />
Voronoi triangulation clipped to glacier used to try & find flowlines for the main Annapurna Glacier.<br />
Some ideas originated from conversations with Gravitystorm.<br />
Map data (c) OpenStreetMap contributors 2014.</td></tr>
</tbody></table>
<li><b>Glaciers</b>. Similarly for rivers although height also needs to be factored in. The idea is not just to identify flows on a glacier, but also simulate likely regions of higher speed flow with a view to creating an apparently more realistic cartographic depiction of the glacier. (Only apparent because in reality one needs lots of good aerial photography to correctly map ice-falls, major bergschrunds, crevasses, crevasse fields etc.).<br /></li>
<li><b>Creating Address Interpolation lines</b>. A small subset of residential highways have quite complex structures and therefore it is non-trivial to add parallel lines for address interpolation. Buffering the multilinestring of the highway centre lines & then resolving that to a single line would help. (More on this soon).<br /></li>
<li><b>Dual Carriageways</b>. Pretty much the same issue as above except there is the additional problem of pairing up the two carriageways. Resolving them to a single way would make high-level routing and small scale cartography better (i.e., it's a cartographic generalisation technique). </li>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieJ1qXx_0uPcOOouThY6pGNyvVYCE7EOphx1mRmejUW5MiwgTGkzzIybNWhS1-bZGnONRk74Tat3YJAfaSUhdn7WVuLytW83E-10T4-l6VeW5kei7NiSm0W5zJdm67HEOyIjX7HDx1B9a7/s1600/old_market_square_ss.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieJ1qXx_0uPcOOouThY6pGNyvVYCE7EOphx1mRmejUW5MiwgTGkzzIybNWhS1-bZGnONRk74Tat3YJAfaSUhdn7WVuLytW83E-10T4-l6VeW5kei7NiSm0W5zJdm67HEOyIjX7HDx1B9a7/s320/old_market_square_ss.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The straight skeleton of Old Market Square Nottingham which allows routing across and close to most of the square<br />
The skeleton does not take account of some barriers on the square, <br />
but the hole at the left (a fountain shows the principle).<br />
Data source: (c) OpenStreetMap contributors 2015.</td></tr>
</tbody></table>
<br />
<li><b>Routing across areas for pedestrians</b>. Pedestrian squares, parks car parks etc. Skeletonisation of such areas may offer a quick & dirty approach to this problem.</li>
</ul>
What follows are some experiments I've done with water areas in Great Britain. I have mainly used the <a href="http://postgis.net/docs/manual-dev/ST_StraightSkeleton.html">ST_StraightSkeleton</a> function, with rather more limited time spent looking at <a href="http://postgis.net/docs/manual-2.2/ST_ApproximateMedialAxis.html">ST_ApproximateMedialAxis</a>. The two images below show my initial attempt to find hydrographic basins: this works merely by chaining together continuous waterway linestrings. These results are not bad, but several major rivers are divided into multiple watersheds. The map of Ireland shows the problem better because the Shannon system appears as a number of discrete watersheds, largely because the Shannon flows through a number of sizeable lakes. Other major rivers illustrating the issue in the UK are the Dee, Trent and Thames.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22944754896/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="River Systems of Great Britain (derived from OSM)"><img alt="River Systems of Great Britain (derived from OSM)" height="667" src="https://c1.staticflickr.com/6/5784/22944754896_68b1af138f.jpg" width="472" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Identification of watersheds in Great Britain by contiguous sections of waterway in OpenStreetMap</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23379593606/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Irish Watersheds from OpenStreetMap"><img alt="Irish Watersheds from OpenStreetMap" height="640" src="https://c7.staticflickr.com/6/5802/23379593606_28cce6a05d_z.jpg" width="453" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Watersheds in Ireland derived from linear watercourses on OpenStreetMap.<br />
Waterways are generally less well-mapped in Ireland, but also several major waterways pass through large lakes (e.g., the Bann (Lough Neagh), Shannon (Lough Ree, Lough Derg), and the <a href="https://en.wikipedia.org/wiki/River_Erne">Erne </a>(Upper & Lower Lough Erne)) and no centre line is available. </td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
So the naive approach raised two problems:<br />
<ul>
<li>Lakes, rivers mapped as areas etc also needed to be included in creating the elements of the watershed</li>
<li>Actual watersheds can be created by creating Concave shells around their constituent line geometries. <strike>Unfortunately I get a PostGIS non-noded intersection error when trying this, so wont discuss it further (although if someone can walk me through how to avoid such problems I'm all ears)</strike>. As later versions of PostGIS seem more robust I return to this later.</li>
</ul>
Of course the simple way to address the first one is just to include areas of water as additional objects in the chain of connected objects. However I would also like to replace rivers as areas, and smaller lakes with linestrings as this type of generalisation can greatly assist cartography at smaller scales. The lack of a source of generalised objects derived from OSM has been a criticism of its utility for broader cartographic use, so this is another aspect of this investigation.<br />
<br />
So now with skeletonisation routines working in PostGIS time to look at some of the basics.<br />
<br />
I've taken Windermere, the largest lake in England, as an example to work through some of the issues. Windermere is a long thin lake which should have a fairly obvious median line. However, it does have some islands which complicate the matter.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi69JhprcDLV6mW-6YnTyHXry3aGUHS2cVIpQkGMSnEt6vTQ2VIPlWqpXnEld7x7-fbMeqbMacxnLGGYWjzFu7mFTkhVKoPnz-gtty6oR-QXNY-kLZDRUQVIAc0dreIyi6qbGUwGzfdSH49/s1600/windermere5.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="285" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi69JhprcDLV6mW-6YnTyHXry3aGUHS2cVIpQkGMSnEt6vTQ2VIPlWqpXnEld7x7-fbMeqbMacxnLGGYWjzFu7mFTkhVKoPnz-gtty6oR-QXNY-kLZDRUQVIAc0dreIyi6qbGUwGzfdSH49/s400/windermere5.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Six versions of Windermere showing area, media axis (red), straight skeleton (thinner lines)<br />
for different degrees of simplification (parameters of 0,5,25,125..). <br />
Original shape is shown as a blue outline.`<br />
All created as a single query using st_translate.</td></tr>
</tbody></table>
Both the straight skeleton & the medial axis are complicated multi-linestrings if I use raw OSM data for Windermere. Progressive simplification of the shape reduces this complexity with reasonable desirable medial axis appearing when simplified with the parameter of around 100 (assumed to be meters in Pseudo-Mercator). Unfortunately there are two problems: the derived axis passes through large islands; and inflow streams are not connected.<br />
<br />
I therefore took a different approach. I disassembled Windermere using ST_Dump and cut the line forming the outer ring at each point a stream or river way touched the lake. I then simplified each individual bit of shoreline between two streams & then re-assembled the lake.<br />
<br />
When this is done all inflows & outflows are connected to the straight skeleton of the simplified lake area. This can be input directly into my routines for collecting all ways making up a watershed.<br />
<br />
Additionally the straight skeleton can be pruned. The simplest one is to just remove all individual linestrings which dangle (i.e., are not connected to a waterway). Presumably one can iterate this until one has the minimum set necessary to a connected set of flows, but I haven't tried this.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw-J7rbfs65v-2L9J5F1HUlYAOpxE9CknygN2sH1RU6U-VN3rrKXF1u_8gdSQy5BRVH83mwd3e8hMGIiEfyNcCs4P50Lw15lquXvHlmucu_03E2pOukRehF2dQSuSxhpkRev5X1LeV9APn/s1600/windermere_skelton_detail.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="281" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhw-J7rbfs65v-2L9J5F1HUlYAOpxE9CknygN2sH1RU6U-VN3rrKXF1u_8gdSQy5BRVH83mwd3e8hMGIiEfyNcCs4P50Lw15lquXvHlmucu_03E2pOukRehF2dQSuSxhpkRev5X1LeV9APn/s400/windermere_skelton_detail.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Straight Skeletons for Windermere calculated for different simplification parameters.<br />
The grey lines represent a parameter where details of islands are kept but the number of edges in the skeleton is greatly reduced.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAaytrlcWRZPDSzNbwLJ1mk3a6wq9HDaDKx3boW_cHtre8hFJmzzQpYAGQMTDaKneGkvPv_9-b2NXJC5aaSnMQ9RBw7nVLrbGh6kn4N_X3i__JEBhLyes49yeh-QmJcDpijJTvAWPThToP/s1600/windermere_skelton_joined.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAaytrlcWRZPDSzNbwLJ1mk3a6wq9HDaDKx3boW_cHtre8hFJmzzQpYAGQMTDaKneGkvPv_9-b2NXJC5aaSnMQ9RBw7nVLrbGh6kn4N_X3i__JEBhLyes49yeh-QmJcDpijJTvAWPThToP/s320/windermere_skelton_joined.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windermere showing inflow & outflow waterways</td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiyGrzoDSXGTevdbFupjgQp7JC0liFDjcnah1GM3UNt9q3PYqv9Op_O15fZklt3VAnhph6fPu9ZKdvyonw1pfbOSzELoTBQEms7fUdNYP37G5YudsXj5ery1295edNvlzSgxdvwHMMcgW1/s1600/windermere_skelton_joined_pruned.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiyGrzoDSXGTevdbFupjgQp7JC0liFDjcnah1GM3UNt9q3PYqv9Op_O15fZklt3VAnhph6fPu9ZKdvyonw1pfbOSzELoTBQEms7fUdNYP37G5YudsXj5ery1295edNvlzSgxdvwHMMcgW1/s320/windermere_skelton_joined_pruned.jpg" width="280" /></a></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizX4nTZUJhnYit_Gqpn_IwzVw2ahlksMBmYo0Fyo-GEEIolutW-u_ArqpmNberA1tnXA-DYv3JfOV_wcQ8olblMnT5hm-Xkc1O3-OZOBBpX5QlzhVJNZU6asC1j0v4O8fbbNJaZMGzndI9/s1600/windermere_skelton_joined_pruned_detail.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="270" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizX4nTZUJhnYit_Gqpn_IwzVw2ahlksMBmYo0Fyo-GEEIolutW-u_ArqpmNberA1tnXA-DYv3JfOV_wcQ8olblMnT5hm-Xkc1O3-OZOBBpX5QlzhVJNZU6asC1j0v4O8fbbNJaZMGzndI9/s320/windermere_skelton_joined_pruned_detail.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Detail of the centre of Windermere showing a reduced straight skeleton linked to inflowing streams (blue). The equivalent without reassembly and preserving stream topology is in red</td></tr>
</tbody></table>
For a single lake it is possible to determine the appropriate degree of simplification to apply, but the complete set of lakes & ponds in Great Britain is a completely different matter.<br />
<br />
Over simplification will result in too big a discrepancy between the original shape and adjacent geometries. Even for Windermere trying to include islands in a reassembly fails with too great a degree of simplification because geometries now cross each other.<br />
<br />
My approach has been to simplify geometries with parameters from 50 to 250 metres in ST_Simplify. I then compare a number of factors with the original:<br />
<ul>
<li>Do I get a valid geometry</li>
<li>Number of interior rings </li>
<li>A measure of surface area</li>
</ul>
With these I then choose one of the simplified geometries for further processing. In general large lakes and riverbank polygons will tolerate more simplification. The overall result is less complicated straight skeletons for further processing. (As an aside I think <a href="http://www.cs.nuim.ie/~pmooney/">Peter Mooney</a> of Maynooth did some work on comparing lake geometries using OSM data around 2010 or 2011).<br />
<br />
For my immediate practical purposes of finding watersheds I did not perform further pruning of skeletons, but such a process is needed for other applications such as cartographic generalisation.<br />
<br />
Even with my first approach which I thought was fairly robust I'm losing a fair number of waterways with simplification. I haven't looked into this further because it will delay finishing this particular post: and it's been on the stocks long enough.<br />
<br />
For further posts on the problems of skeletonisation read <a href="https://twitter.com/smathermather">Stephen Mathers</a> <a href="https://smathermather.wordpress.com/">blog </a>which I found very useful. <a href="https://github.com/StyXman">StyXman </a>is <a href="https://github.com/StyXman/osm-centerlines/tree/master/src/org/openstreetmap/josm/plugins/centerlines">developing a JOSM plugin</a> which uses some of these techniques to create centrelines too. A big thank you to him, and, of course, to Christoph Hormann (<a href="http://www.imagico.de/">imagico</a>).SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-34403203048250806332016-07-01T11:58:00.000+01:002016-07-01T11:58:12.422+01:00How far are Hedgehogs from a road?<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/27777325701/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="My last hedgehog siting (2010)2887a"><img alt="My last hedgehog siting (2010)2887a" height="375" src="https://c6.staticflickr.com/8/7444/27777325701_3f62054431.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My last <a href="https://en.wikipedia.org/wiki/European_hedgehog">hedgehog </a>sighting in Britain: <a href="https://en.wikipedia.org/wiki/Elston">Elston</a>, Nottinghamshire 2010.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script><br />
<br />
One of my great joys with OpenStreetMap (and other (mainly) geographical Open Data) is that it provides a way into answering intriguing analytical questions.<br />
<br />
A few weeks ago the query was from a <a href="https://twitter.com/hedgehoghugh">Hedgehog ecologist</a>: naturally I learnt of the query through OSM (via IRC to be precise).<br />
<br />
The question was very simple: <i> </i><br />
<br />
<blockquote class="tr_bq">
<i>What proportion of Britain's land area is more than 100 m from a road? </i> </blockquote>
<br />
The reason it is germane for hedgehogs is that historically they have had a very high mortality from crossing roads. These days they are so rare, that spotting a squashed hedgehog is itself a rarity. Certainly <a href="https://www.cartoonstock.com/cartoonview.asp?catref=emc0071">this cartoon</a> would not have the same resonance it did when it first appeared in the 1970s.<br />
<br />
To answer the query is fairly straightforward: providing one has either a GIS tool or database to hand <b>AND</b> a full data set of British roads. QGIS and PostGIS were available & I also have a full set of OSM data for May 2015 in the latter.<br />
<br />
<br />
<a name='more'></a>The steps involve in answering the query were simple ones as follows:<br />
<ul>
<li>Select a relevant subclass of highway elements from OSM (in this case deliberately selecting the main types of motorable roads)</li>
<li>Re-project into <a href="https://en.wikipedia.org/wiki/British_Grid">British Grid</a> (which enables more precise answers to this type of query)</li>
<li>Buffer all the roads by 100m</li>
<li>Eliminate overlaps</li>
<li>Sum the area covered </li>
<li>Subtract that from the known land area of Great Britain.</li>
</ul>
In practice for this type of work I nearly always grid the data so that costly queries relying on the GIST indexes on geometries have to do less work. So I clip roads using a 2 or 5 km grid. All roads in a given grid square can either be merged into a multiline structure & then buffered, or buffered and then merged into a single multipolygon. One way is probably more efficient than the other. Once buffered the buffer area<br />
<br />
In my initial analysis which I carried out very quickly I made three errors:<br />
<ul>
<li>Failing to include trunk roads in the initial road selection through missing quotes in the SQL predicate</li>
</ul>
<ul>
<li>Using <span style="font-family: "Courier New",Courier,monospace;">st_within</span> & not <span style="font-family: "Courier New",Courier,monospace;">st_intersects</span> to find roads crossing a grid square.</li>
</ul>
<ul>
<li>Clipping back to grid square boundaries in all cases. A road ending on the edge of one grid square still creates a 100 m semicircle in the adjacent square</li>
</ul>
As usual it was visualisation of the data which hinted at the things I needed to fix. Certain main roads were missing, there were islands of roads in central Wales, and the grid boundaries were too obvious when visualised.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiss10RDU1Xbzft4CVIaxkKGx8ComY8FUA3ipeiJ6WCKnfaDtbrJZ2l43qSWiY5rOK8RWOUwKr9bLkvMeifnnLcbHKrZym1yFFKBNArZWwjmsAH9YpS4-q2TYRnydhvqaj7Yw7iXuAUzY88/s1600/100m_roads.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="318" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiss10RDU1Xbzft4CVIaxkKGx8ComY8FUA3ipeiJ6WCKnfaDtbrJZ2l43qSWiY5rOK8RWOUwKr9bLkvMeifnnLcbHKrZym1yFFKBNArZWwjmsAH9YpS4-q2TYRnydhvqaj7Yw7iXuAUzY88/s320/100m_roads.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Initial visualisation. Note big gaps in Central Wales and absence of continuous lines on motorways.</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPxjmmdEzelgiQJPvO8cnpTV3wzMgHMqtR7l8YCFvwTKQ450WpMCKmBRa3XaV3tZWb3VE89NNvZ19gi2V9EvbU3UWh1Mi1bFbyjjap5Cb5FsFoALwAOmOG2oliyVV-Mqonahev0F9h5W8P/s1600/road100b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPxjmmdEzelgiQJPvO8cnpTV3wzMgHMqtR7l8YCFvwTKQ450WpMCKmBRa3XaV3tZWb3VE89NNvZ19gi2V9EvbU3UWh1Mi1bFbyjjap5Cb5FsFoALwAOmOG2oliyVV-Mqonahev0F9h5W8P/s320/road100b.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Visualisation with fixed data. Compare Central Wales and motorways.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8NurA5UoE_t7Gvb9UJEtBtH2TVkgCnqqt2t3jpxIVm8S_5lHxxM-q_TGXhJazD3eGN1EV10SW32L0GJbrxyqoPEttv07f5W43uFmmQAJ6xlAj-fDuhXatgR-glwMVHrTvhwVwV5IDX2qB/s1600/road100a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8NurA5UoE_t7Gvb9UJEtBtH2TVkgCnqqt2t3jpxIVm8S_5lHxxM-q_TGXhJazD3eGN1EV10SW32L0GJbrxyqoPEttv07f5W43uFmmQAJ6xlAj-fDuhXatgR-glwMVHrTvhwVwV5IDX2qB/s320/road100a.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Area W of Birmingham enlarged. The main population centres are <a href="https://en.wikipedia.org/wiki/Shrewsbury">Shrewsbury </a>(top centre) and <a href="https://en.wikipedia.org/wiki/Telford">Telford </a>(right of top centre) with the <a href="https://en.wikipedia.org/wiki/West_Midlands_conurbation">West Midlands conurbation</a> on right edge. Welsh hill country is top loeft.</td></tr>
</tbody></table>
<br />
The first I suspected anyway. I'd written the selection rather hurriedly & there are 11 types of highway I include normally, so it's rather easy to mis-spell one, or as in this case not spot missing quotes. Easy to fix.<br />
<br />
The second was less immediately obvious, but it's one which has caught me out in the recent past too. This is writing <span style="font-family: "courier new" , "courier" , monospace;">st_within</span> in a query where <span style="font-family: "courier new" , "courier" , monospace;">st_intersects</span> is required. I suppose I do point-in-polygon type queries more frequently so instinctively type <span style="font-family: "courier new" , "courier" , monospace;">st_within</span>. Also <span style="font-family: "courier new" , "courier" , monospace;">st_within<span style="font-family: "georgia" , "times new roman" , serif;"> <span style="font-family: "times" , "times new roman" , serif;">is specifically a function which works on polygons whereas </span></span></span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "times" , "times new roman" , serif;"><span style="font-family: "courier new" , "courier" , monospace;">st_intersects</span> has broader applicability.</span></span></span><br />
<br />
Either way these were easy errors to fix. To do the calculation I used an Ordnance Survey Open Data file to hand (one of local authority boundaries) which results in a slight overstatement of land area as some LAs boundaries include marine areas (notably in <a href="https://en.wikipedia.org/wiki/The_Wash">The Wash</a> & in the <a href="https://en.wikipedia.org/wiki/Bristol_Channel">Bristol Channel</a>) together with the OSM outline for the <a href="https://en.wikipedia.org/wiki/Bristol_Channel">Isle of Man</a> (as this is not available in OSGB Open Data). The area for Great Britain and Isle of Man which I used was 238,905 square km, but this can be corrected to any other widely available figure, such as that given on <a href="https://en.wikipedia.org/wiki/Countries_of_the_United_Kingdom">Wikipedia</a> of 229,519 square km. Equally the area of road buffers will include some sea for coastal roads. Ideally I need my grid cells to be land area only, something I'm thinking of doing for other purposes too.<br />
<br />
The sum of the area of the road buffers was for 100 m, for 200 m, for 500 m and 1000 m. I was also asked for a 5km buffer and I calculated that separately using a larger grid size. Note that gridding is purely something done to speed up the process. A recent talk highlights (sorry, I can't find the link for this slidedeck but it was from a recent Geo-conference) that many standard geometry routines have poor <a href="https://en.wikipedia.org/wiki/Computational_complexity_theory">computational complexity</a> properties so one often has to resort to divide-and-conquer approaches such as gridding the data.<br />
<br />
So here are the approximate figures, given the caveats above:<br />
<ul>
<li>Within <b>100 m</b> of a road: 53,964 km2 or about 23.5% of land area of Great Britain
</li>
<li>Within <b>200 m</b> of a road: 87,971 km2 or about 38.3% of land area</li>
<li>Within <b>500 m</b> of a road: 150,147 km2 or about 65.4% of land area</li>
<li>Within <b>1000 m</b> of a road: 189,562 km2 or about 82.6% of land area</li>
<li>Within <b>5000 m</b> of a road: 260,419 km2 or rather more than the land area concerned. </li>
</ul>
The last figure made it abundantly clear that I needed to address the sources of approximation. Coastlines & other borders introduce edge effects which can seriously <a href="http://sk53-osm.blogspot.co.uk/2015/05/retail-outlets-on-openstreetmap.html">complicate this type of analysis</a> and often require ensuring that one has appropriate data to handle it. So I did the following:<br />
<ul>
<li>Created a shape file of the land area for Great Britain & Isle of Man</li>
<li>Clipped each buffered result set by that shape file</li>
<li>Used each of these inputs for the final calculation</li>
</ul>
In order to make the shape file needed something based on the coastline. Fortunately I remembered Geolytix had used OSGB Open Data of coastlines to produce refined postal area polygons. This was immediately available to me and I therefore used this as the starting point. I created a separate Isle of Man shapefile from OSM data and merged the two together. So here are the revised figures:<br />
<ul>
<li>Within <b>100 m</b> of a road: 53,760 km2 or about 23.3% of land area of Great Britain
</li>
<li>Within <b>200 m</b> of a road: 87,353 km2 or about 37.9% of land area</li>
<li>Within <b>500 m</b> of a road: 147,740 km2 or about 64.2% of land area</li>
<li>Within <b>1000 m</b> of a road: 186,533 km2 or about 79.7% of land area</li>
<li>Within <b>5000 m</b> of a road: 226,418 km2 or 98.3% of the land area. </li>
</ul>
And just to enable checking here are some visualisations:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5JXewQyljE-56aNWSuhfTIF7u6XQCt6A-tO_ShaLdyoHHPRGopcX2j_seAUVl46u_zrBSTXBt9RtEldtQdfz44oy-sK-0-1uRvpm4PhyIzQeE2xJ81cTqC0Hxf_HJYZEcQp30hqxOJWXF/s1600/hedgehog_roads_e%2526W.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5JXewQyljE-56aNWSuhfTIF7u6XQCt6A-tO_ShaLdyoHHPRGopcX2j_seAUVl46u_zrBSTXBt9RtEldtQdfz44oy-sK-0-1uRvpm4PhyIzQeE2xJ81cTqC0Hxf_HJYZEcQp30hqxOJWXF/s320/hedgehog_roads_e%2526W.jpg" width="450" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Distances for England & Wales</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh29UrwTtJ_yWH8hCOesJnZhSHbEHvsFtHOIm0Yr39kwp2YYBGiDAhLakNaYRZ77z867eJUwxZBbABemdwHpskPUr87nRA33WXtbiBWDy9gInmC6gtTbTc9jZgg4PHBrri0_pLOIRAR7-FQ/s1600/hedgehog_roads_scotland.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh29UrwTtJ_yWH8hCOesJnZhSHbEHvsFtHOIm0Yr39kwp2YYBGiDAhLakNaYRZ77z867eJUwxZBbABemdwHpskPUr87nRA33WXtbiBWDy9gInmC6gtTbTc9jZgg4PHBrri0_pLOIRAR7-FQ/s320/hedgehog_roads_scotland.jpg" width="452" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mainland Scotland</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4INWweMfHg-ovFvhY0wd-Ekmw3st4-xc6rB0k52TyHJSiZFIB38PyimqsRk4Lr1F2fuP3t5iuWWd3uWtMY-hTT7FT7Ysgs3gAb1E_rvBl8ToUh9qaEsiFB9sAta25PXKD3_x49i90FBM0/s1600/hedgehog_roads_dark_peak.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4INWweMfHg-ovFvhY0wd-Ekmw3st4-xc6rB0k52TyHJSiZFIB38PyimqsRk4Lr1F2fuP3t5iuWWd3uWtMY-hTT7FT7Ysgs3gAb1E_rvBl8ToUh9qaEsiFB9sAta25PXKD3_x49i90FBM0/s320/hedgehog_roads_dark_peak.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">English <a href="https://en.wikipedia.org/wiki/Peak_District">Peak District</a> (<a href="https://en.wikipedia.org/wiki/Dark_Peak">Dark Peak</a>)<br />
<a href="https://en.wikipedia.org/wiki/Greater_Manchester">Manchester </a>is on W, <a href="https://en.wikipedia.org/wiki/Sheffield">Sheffield </a>on R</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRsI6KCHuVhN_vvvQFhTDQAepDxp2zdluD8q7P9ZIG4Y7JjsR8oFh3plufTUx7jJPX_gHrD0-Lg3b5BkuQeYRdlTPsLap6tAsjYvLt4F0IaG4tSTWdLuCnYmkwPQPERVRG7OI_doFCrHBG/s1600/hedgehog_roads_north_wales.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRsI6KCHuVhN_vvvQFhTDQAepDxp2zdluD8q7P9ZIG4Y7JjsR8oFh3plufTUx7jJPX_gHrD0-Lg3b5BkuQeYRdlTPsLap6tAsjYvLt4F0IaG4tSTWdLuCnYmkwPQPERVRG7OI_doFCrHBG/s320/hedgehog_roads_north_wales.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/North_Wales">North Wales</a> & <a href="https://en.wikipedia.org/wiki/Merseyside">Merseyside</a></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhu_FkNh5-JhegViHulkWUqbL7xnxEx_eKu6FDGTsQUqD0zyEViOqTWUnjNArj71qGVyyLLubb1IIbYIc5-ma8DkA2GGTp0kYbq61hzVfSHELrcmExXW2Bdp8W8ugFAklPzXOgxx8G56_xb/s1600/hedgehog_roads_south_coast.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhu_FkNh5-JhegViHulkWUqbL7xnxEx_eKu6FDGTsQUqD0zyEViOqTWUnjNArj71qGVyyLLubb1IIbYIc5-ma8DkA2GGTp0kYbq61hzVfSHELrcmExXW2Bdp8W8ugFAklPzXOgxx8G56_xb/s320/hedgehog_roads_south_coast.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Isle_of_Wight">Isle of Wight</a> and South Coast of England</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdJE5c08Av0wYVdi9OU8LziDdjeQHwxdH2zge0ak9T8B2VTshdtzIDZYj4a9GyX2xEv1s4dmbeuBfcAamDxN2n2JzcTcvlAo2KRd_hlgNJGqEwOg9Xz2edVGPSkqV4exJzZxjbluZh6RL8/s1600/hedgehog_roads_south_ebudes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj1MJsqepL2dpxTatVkXWTVfUCc67_Nsg8WlG1I6HqhtpWhbOBV-3DuEAZkrh0ew5TWXVBjA7QV6_nhAHZF1LSYckXdAmEpD0OCm0smCKZrTdrWQAuW99Ajw4fdKjSOf3cR9q07_6n3sK-/s1600/hedgehog_roads_south_west.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj1MJsqepL2dpxTatVkXWTVfUCc67_Nsg8WlG1I6HqhtpWhbOBV-3DuEAZkrh0ew5TWXVBjA7QV6_nhAHZF1LSYckXdAmEpD0OCm0smCKZrTdrWQAuW99Ajw4fdKjSOf3cR9q07_6n3sK-/s320/hedgehog_roads_south_west.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/South_West_England">South West England</a></td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Y5k93FM_TOi2aG7egcpoKID7SaybjSn7K2-kWEN76KxY0qfzvSLAHwo41aC5oQO7EXsDXPogCM6WUBNmTxVDIFGvp1e0UqXS2zpobCSyyatLGD-HuetK7j7XXT83nzVrK37PThINA-lZ/s1600/hedgehog_roads_south_ebudes.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Y5k93FM_TOi2aG7egcpoKID7SaybjSn7K2-kWEN76KxY0qfzvSLAHwo41aC5oQO7EXsDXPogCM6WUBNmTxVDIFGvp1e0UqXS2zpobCSyyatLGD-HuetK7j7XXT83nzVrK37PThINA-lZ/s320/hedgehog_roads_south_ebudes.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Central_Belt">Central Belt</a> & Southern <a href="https://en.wikipedia.org/wiki/Inner_Hebrides">Inner Hebrides</a>, Scotland<br />
<a href="https://en.wikipedia.org/wiki/Kintyre">Kintyre </a>(bottom left of centre) shows forest tracks erroneously mapped as roads</td></tr>
</tbody></table>
<br />
Even now there are one or two problems: specifically forestry tracks on Kintyre mapped as unclassified roads (I think I caught most of these and corrected them sometime ago). Some roads on Scottish Islands have a status which is unclear too. However the visualisations of the data do help to highlight such cases.<br />
<br />
The other thing is that for much of the country the patches of land which are further than 1 km from any road are often easily recognisable to the extent that they can often be readily names. The two big patches in the Peak District are <a href="https://en.wikipedia.org/wiki/Bleaklow">Bleaklow</a> and <a href="https://en.wikipedia.org/wiki/Kinder_Scout">Kinder Scout</a>. So the visualisation also shows other things which may be of interest for other reasons<br />
<br />
<br />
The sad part of this post is that it wouldn't be necessary if hedgehogs were not in a steep decline across Britain.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-86808748596885196912016-05-20T17:42:00.000+01:002016-05-20T17:42:36.469+01:00Bristol (& New Brighton) Buildings from Lidar<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img alt="West front of Bristol Cathedral" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/44/West_front_of_Bristol_Cathedral.jpg/512px-West_front_of_Bristol_Cathedral.jpg" width="512" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">West Front, Bristol Cathedral<br />
One of the buildings where we could use Lidar data to enhance its representation in OSM</td></tr>
</tbody></table>
At <a href="http://odcamp.org.uk/">OpenData Camp 3</a> over the weekend I asked <a href="https://twitter.com/murraydata">John Murray</a> if he could give me a set of polylines extracted from the <a href="http://www.geostore.com/environment-agency/survey.html#/survey?grid=SJ39">Environment Agency Lidar Open Data</a>. Do read John's <a href="http://odcamp.org.uk/turning-lidar-data-into-actionable-insight/">amazing post</a> about the tools he has built for doing nifty things with Lidar data: <i>Turning Lidar Data into Actionable Insight</i>.<br />
<br />
I thought if might be a bit of fun to actually show directly how opendata produced by one of the ODCamp sponsors might end up in OpenStreetMap.<br />
<br />
In practice John got keen over the lunch break and wrote a bit of code to turn his polylines into polygons. So just before the last session of the day kicked off I had been sent a shape file for the 1km Ordnance Survey gird square where the meeting was taking place.<br />
<br />
We had one initial teething problem that the data was out by 1 km, but notwithstanding that it was very simple to perform some simple manipulations in the off-line OpenStreetMap editor JOSM.<br />
<br />
JOSM can read shapefles (and some other geo-formats such as geojson too) and automatically transforms these into OSM elements projected in WGS84. Therefore the additional data manipulation steps were pretty trivial:<br />
<ul>
<li>Select all way elements (using a type:way search)</li>
<li>Add a source=EA Lidar Open Data tag</li>
<li>Select all type:relation elements to find multipolygons</li>
<li>Add building=yes tag</li>
<li>Select all way elements which were not part of multipolgons (type:way and not child type:relation)</li>
<li>Add building=yes tag</li>
<li>Select all way elements again and simplify them.</li>
</ul>
The image below shows how this looks in the editor<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR5uOKI8zn-QVj5v311whbflYKbbfJ09I5QRk7z9ZA_Cqkx9ECW8Jy1Q5Gsu7dMqIK1D-B5P5l4QK7qf2O7HvAU2fHAiueZl4p0VM-3Hrd4Cb7-2bRRkFoofRGjEeeyOuvOmNTi7Jcw4Zm/s1600/watershed_in_josm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="289" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhR5uOKI8zn-QVj5v311whbflYKbbfJ09I5QRk7z9ZA_Cqkx9ECW8Jy1Q5Gsu7dMqIK1D-B5P5l4QK7qf2O7HvAU2fHAiueZl4p0VM-3Hrd4Cb7-2bRRkFoofRGjEeeyOuvOmNTi7Jcw4Zm/s400/watershed_in_josm.png" width="520" /></a></div>
<br />
Now this is all pretty amazing, and if you read John's blog post there's lot more info which can be gleaned. However a close inspection of the data still shows a sizeable number of artefacts which would need cleaning up. Some John has dealt with in the intervening few days, but turning any automatically extracted feature into something of the sort of quality which can be one in OSM is another matter: and is not too dissimilar to the points I made some time ago about <a href="http://sk53-osm.blogspot.co.uk/2016/01/uk-open-data-and-buildings-in.html">OpenMap Local</a>.<br />
<br />
For me the real advantage is that it's a major step in making if more feasible to use Lidar data to enrich OSM data. For instance data on roof orientations could be combined with the algorithms & crowd-sourced validation methods from OpenSolarMap. I hadn't realised until listening to John's talk just how valuable gable or eaves heights are in building datasets. It certainly persuaded me that they belong in OSM.<br />
<br />
Another downside is that it takes a whiz like John to create this software and it makes use of a powerful machine, powerful algorithms, optimised hardware & proprietary storage. I have therefore spent a little time this week looking again at what is available in QGIS to do similar (but much less powerful) manipulation of the Lidar data.<br />
<br />
Basic transformations of Lidar have been described elsewhere (for instance see Chris Hill's <a href="http://chris-osm.blogspot.co.uk/2015/09/extracting-building-heights-from-lidar.html">posts</a>) so I won't dwell on them here. Suffice it to say I presume that the following have been created for a given area:<br />
<ul>
<li>Combined Digital Surface Model (DSM). I usually do this as a virtual time set (can be done directly in QGIS)</li>
<li>Combined Digital Terrain Model (DTM). As above.</li>
<li>Delta of the two. DSM-DTM. This gives things (buildings, cars, trees etc) which are elevated above ground level.</li>
</ul>
To get somewhere near what John's approach involves ideally requires:<br />
<ul>
<li>Filtering out shorter objects (mainly cars, garages & some street furniture)</li>
<li>Filtering out smaller objects (mainly trees)</li>
<li>Edge detection</li>
<li>Polygonisation</li>
</ul>
In practice I found it relatively easy to do the first & last and did not find a simple way of doing the other 2 in QGIS (although in part that might be because I'm short of disk right now).<br />
<br />
The other two can be achieved easily:<br />
<ul>
<li>Filtering by Height: this is merely another raster calculation using the QGIS Raster Calculator. In my <a href="http://www.openstreetmap.org/#map=16/53.4361/-3.0497">test area</a> (New Brighton on the Wirral, OS grid ref SJ3093) most houses are Edwardian and much higher than 3 metres, whereas garages are usually a touch over 2 metres. I therefore used 3 metres as a cut-off.</li>
<li>Polygonisation. I used the height filtered data directly with the Raster...Conversion...Polygonize option in QGIS. This is a much cruder and more naive method than I was hoping to use, but there it is.</li>
</ul>
I show the results of these steps below (in separate images to allow easier inspection & then combined).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYDCeAKDuapgjTlTLn-oNjxplyXxMX5211wvBRrpUfkFk0vPVoirYulMqsBkz0E3NYGn3-VokehXr5HJmyev-1K0WKYls5jljD2umcr4lsuTzp6lYRFBQmGPDcK8ZE4LEyyKOczclDr6Ls/s1600/newbrighton_lidar_heights.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYDCeAKDuapgjTlTLn-oNjxplyXxMX5211wvBRrpUfkFk0vPVoirYulMqsBkz0E3NYGn3-VokehXr5HJmyev-1K0WKYls5jljD2umcr4lsuTzp6lYRFBQmGPDcK8ZE4LEyyKOczclDr6Ls/s320/newbrighton_lidar_heights.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lidar Height data (DSM-DTM) filtered for >3m</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsiDud8QR-pVCPJn7LguOW7MMnDhn3smxGLnv6MJpMAnlLy9OWN_RGMzbV6xVZjvGLUpcQc_1BBCvDrUOo-G7uo1J_xlXXE6tDv-3c0Y0JASf1H3A5bXAbayqn9liwXSHuc0qJOPXZdkM7/s1600/newbrighton_building_polygons_only.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsiDud8QR-pVCPJn7LguOW7MMnDhn3smxGLnv6MJpMAnlLy9OWN_RGMzbV6xVZjvGLUpcQc_1BBCvDrUOo-G7uo1J_xlXXE6tDv-3c0Y0JASf1H3A5bXAbayqn9liwXSHuc0qJOPXZdkM7/s320/newbrighton_building_polygons_only.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Extracted & OSM Building polygons compared<br />
(garages are deliberately excluded from OSM data)</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyPpVNtAWZ8tKp2reO9o1-v5hSTO8sF1Cu1rex2-S7ZzDcYWIYPm63SzJVvWnrxrGa5hsrtW8NV4CVh66QkVfTwVqD7OD2pw87xQp2va0GZSqq32sA61L7n0o9BgycfwUKhnMDX2kkVc6V/s1600/newbrighton_building_polygons.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyPpVNtAWZ8tKp2reO9o1-v5hSTO8sF1Cu1rex2-S7ZzDcYWIYPm63SzJVvWnrxrGa5hsrtW8NV4CVh66QkVfTwVqD7OD2pw87xQp2va0GZSqq32sA61L7n0o9BgycfwUKhnMDX2kkVc6V/s320/newbrighton_building_polygons.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Height data combined with Polygons</td></tr>
</tbody></table>
<br />
<br />
Firstly it's worth noticing a few features from the raw height data:<br />
<ul>
<li>Most buildings are tall, usually in excess of 8 metres (and probably at least that height at the gables).</li>
<li>There are a limited number of lower height buildings. The most obvious ones are near the top of the image and include two small factory premises N of the railway & the platform canopy of the railway station. S of these the road bridge over the railway is obvious; and immediately to the SE there are apparently two largish buildings of low height, albeit quite a bit of noise in the height profile. (These are, in fact, Victoria View a development of flats which halted for several years). Further S still there are a small number of bungalows.</li>
<li>Terraces with a lower rear service area are obvious. </li>
<li>There are a significant number of linear features above 3 m in height. Most look to be walls, and indeed garden walls in the area tend to be high as most gardens are small & given building heights would tend to be overlooked.</li>
<li>Isolated trees are obvious in one or two back gardens</li>
<li>Larger groups of trees are equally obvious along the railway line (& elsewhere)</li>
<li>Swirly patterns in the 3-4 metre range occur in a number of places. These are mainly scrub (mainly gorse) or shrubberies.</li>
<li>There are still parked vehicles giving returns in the 3-4 metre height range. </li>
</ul>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimaisu9hHl_SbDeoiZOMSeaqL6gT3-uj0BcnwaAcU5aQh9QJ-Wk96KKrs8qtVrGIsANGE9BhIKDMyLBMYLQNkDLtBi8XoKTeMLBMqDeRSuNYl5gztsu1tEcbsgcAEak7B_mNx3kCDPiCwp/s1600/IMG_6362-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimaisu9hHl_SbDeoiZOMSeaqL6gT3-uj0BcnwaAcU5aQh9QJ-Wk96KKrs8qtVrGIsANGE9BhIKDMyLBMYLQNkDLtBi8XoKTeMLBMqDeRSuNYl5gztsu1tEcbsgcAEak7B_mNx3kCDPiCwp/s320/IMG_6362-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Edwardian Streets <a href="http://www.openstreetmap.org/#map=18/53.43388/-3.05226">S of Mount Road</a> New Brighton (Dovedale & Langdale Rds)</td></tr>
</tbody></table>
<br />
I include a couple of photos of streets in the area to help with context. I would recommend strongly <a href="https://www.flickr.com/photos/liverpoolsuburbia/541180697/in/album-72157600356944866/">Russ Oakes' work</a> documenting suburban streets all over Merseyside for a much broader perspective. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7D__u6IwG9GA1DOpekCoAmBf-Wht8GVtaLFVpwm1NU8j2VscOqLp8xXlxb0MVeROazfxkSjIIbQZcYc6sEfV0KcGXZmFRDOW2w1aqxC9LVvpEoIOwNMG_MwEJRz4jgncnePRRyMLC9Tk0/s1600/IMG_6347-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7D__u6IwG9GA1DOpekCoAmBf-Wht8GVtaLFVpwm1NU8j2VscOqLp8xXlxb0MVeROazfxkSjIIbQZcYc6sEfV0KcGXZmFRDOW2w1aqxC9LVvpEoIOwNMG_MwEJRz4jgncnePRRyMLC9Tk0/s320/IMG_6347-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Junction of <a href="http://www.openstreetmap.org/#map=17/53.43528/-3.05107">Dudley & Hamilton Road</a>s, New Brighton</td></tr>
</tbody></table>
<br />
Comparing the extracted polygons with OSM (and ignoring some OSM data which is missing) shows:<br />
<ul>
<li>There is a fairly constant offset of OSM data (presumably inherited from the Bing imagery).</li>
<li>Building footprints are broadly comparable</li>
<li>Small gaps in terraces are resolve much better by tracing.</li>
<li>Some detail has not been added to some of the terraces in OSM which are still drawn as plain rectangles.</li>
<li>It's certainly possible to spot missing features & use Lidar data as an aid to add them in (notably Victoria View, but the W part of the development was started after the Lidar data.</li>
</ul>
Now as for deriving data to enhance OSM there's a fair more bit of processing needed.<br />
<br />
Absolute building height is relatively easy, one just needs to find the maximum height within a (location corrected) OSM polygon. Generating the other more useful <a href="http://wiki.openstreetmap.org/wiki/Simple_3D_buildings">Simple 3D building</a> (S3DB) tags is rather more involved, and certainly I have the impression that QGIS would be a fairly clunky way to do things. I really hope that some more technically-minded OSM folk can take inspiration from John's ideas and start thinking about tools to mainpulate Lidar data specifically for OSM.<br />
<br />
There is no doubt that the Environment Agency Lidar data was one of the most significant open data releases last year. Furthermore it is likely that other agencies & local government bodies will make Lidar available more widely in the near future. For instance I believe much data from Kanton Zurich is open, including Lidar. This <a href="http://maps.zh.ch/?topic=LidarZH&scale=6280&x=670560.53&y=250675.15&offlayers=dom2014hillshade%2Cblatteinteilung-lidar%2Cblatteinteilung-dtm%2Cblatteinteilung-labels_lidar%2Cblatteinteilung-labels_dtm&over=HoehenlOverlayZH%2CUpBackgroundZH">example </a>shows the extensive slumping caused by peri- and post-glacial phenomena in the woods near Bergietikon: so this is a reminder that it's not just buildings which are of interest.<br />
<br />
One last thing to note is that there's lots one can do with this data immediately (the subject of John's original article). Working how to add this data to OSM begins to look not dissimilar to creating authoritative datasets. It's of course worth spending time working out how to do this because once in OSM the data is potentially available for a multitude of purposes. SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com1tag:blogger.com,1999:blog-2082422783692631078.post-36926262322454029592016-05-04T12:45:00.001+01:002016-05-04T12:45:45.395+01:00Where have all the woods gone from Google Maps?Very recently there was a nice post by <a href="https://twitter.com/justinobeirne">Justin O'Beirne</a> about the <a href="http://www.justinobeirne.com/essay/what-happened-to-google-maps">cumulative effect of change</a>s to the cartography of Google Maps. <a href="https://twitter.com/richardf">Richard Fairhust</a> summarised his views on twitter:<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>
— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>
— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
On that <a href="https://twitter.com/justinobeirne">@justinobeirne</a> article: 1. Google Maps becoming less “map”, more “backdrop for search results”. Similar: road colours white/grey.</div>
— Richard Fairhurst (@richardf) <a href="https://twitter.com/richardf/status/727190431260377088">May 2, 2016</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
This is just my (very) minor contribution to the discussion.<br />
<br />
The <a href="http://bsbi.org/">Botanical Society of Britain & Ireland</a> (BSBI) uses Google Maps as the background to their <a href="http://bsbi.org/maps/?taxonid=2cd4p9h.cba">maps of plant distributions</a>. Over the past couple of weeks I've been using it a lot because I've been interested in two things:<br />
<ul>
<li>Where I might fund particular plants relatively close to where I live;</li>
<li>Which plants I see might be of interest to the county recorders.</li>
</ul>
As at this time of year many of the botanical highlights are to be found in ancient woodlands it's damn useful to see where the woods are when assessing the BSBI records. That's why I noticed woods disappearing from the Google cartography as one zooms in.<br />
<br />
This screenshots shows successive zooms of an area in central Nottinghamshire which includes Clumber Park an two old woods, Gamston & Eaton Woods. The latter two are centre right above the village of Askham.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghxp280O1MvqbowbofIfme6-jRTBs3o9g3S_KYd4gcSvd3c1IoU-EG3Fo9I-lhmAYmPnzv-Tr5k_zPD0xYNs41foFDgKqfsfiE6SsmThTYggdO7-qcl2abP81LSSngPUfdPgzgTwQmS1wZ/s1600/google_woods1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghxp280O1MvqbowbofIfme6-jRTBs3o9g3S_KYd4gcSvd3c1IoU-EG3Fo9I-lhmAYmPnzv-Tr5k_zPD0xYNs41foFDgKqfsfiE6SsmThTYggdO7-qcl2abP81LSSngPUfdPgzgTwQmS1wZ/s320/google_woods1.png" width="520" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLilCYPVe9Xnr7Ja3sHtBwTlxu818etM3FR-MpoE1E63YaKLPbLh-mL1afrkhqMe2EsGdjueyc1gahtAnkvfRHQxF2pka7iVdcUIDRQLyt-S-wWz1C_GaW2sto7i5keY_TFxtTsFQCOaN9/s1600/google_woods2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLilCYPVe9Xnr7Ja3sHtBwTlxu818etM3FR-MpoE1E63YaKLPbLh-mL1afrkhqMe2EsGdjueyc1gahtAnkvfRHQxF2pka7iVdcUIDRQLyt-S-wWz1C_GaW2sto7i5keY_TFxtTsFQCOaN9/s320/google_woods2.png" width="520" /></a></div>
<br />
All woodland just disappears between these two zoom levels.<br />
<br />
<div style="text-align: left;">
Here's the active map so one can play with zooming in & out.</div>
<br />
<div style="text-align: center;">
<br />
<iframe allowfullscreen="" frameborder="0" height="450" src="https://www.google.com/maps/embed?pb=!1m14!1m12!1m3!1d76338.19460595223!2d-0.9269806048847737!3d53.27934919391607!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!5e0!3m2!1sen!2suk!4v1462360791321" style="border: 0;" width="600"></iframe></div>
<br />
Losing woods at high zoom levels is another example of loss of functionality. In practice it makes the maps layer useless for interpreting botanical data: I have to resort to using the satellite layer. Even that is not always easy because sometimes fields also appear dark green.<br />
<br />
Google does use a couple of other green shades for things like parks, golf courses, and possibly nature reserves (see Sherwood Forest NNR near Edwinstowe). I don't know if these come on and off in a similar arbitrary pattern. SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com2tag:blogger.com,1999:blog-2082422783692631078.post-59723103370821543962016-02-14T14:43:00.001+00:002016-02-14T14:43:16.935+00:00Distribution of Contributions in Volunteer-generated Datasets : Gall or Fruit Fly RecordsI remarked in my <a href="http://sk53-osm.blogspot.co.uk/2015/04/interviewed-by-opencagedata.html">OpenCageData interview</a> that I see many similarities between biological recording and OpenStreetMap contributions. Indeed, I've had some interesting discussions about this with <a href="https://povesham.wordpress.com/">Prof. Muki Hakaly</a> at UCL. Muki's group now do extensive research across the gamut of activities which fall under the rubric of "citizen science", so I'm hopeful that they will elucidate which features are common across this spectrum.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/5817646991/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="Chaetorellia jaceae f : 5532b"><img alt="Chaetorellia jaceae f : 5532b" height="375" src="https://farm6.staticflickr.com/5240/5817646991_49307c1d98.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A female <i><a href="https://en.wikipedia.org/wiki/Chaetorellia">Chaetoraellia </a>jaceae</i>, a tephritid fly whose larvae feed on Knapweed.<br />
Photo: (c) mausboam, Flickr.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Basically, we know that there is a very long tail of smaller contributions to OpenStreetMap. Both <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_The_Long_Tail_of_OpenStreetMap">Harry Wood</a> and <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_More_than_Just_Data">Frederick Ramm</a> gave presentations on aspects of this at SotM-14 in Buenos Aires, and Richard Fairhurst also touched on this at SotM-US in 2013. Very recently <a href="http://www.openstreetmap.org/user/marczoutendijk/diary/37902">Marc Zoutendijk</a> has used data collected by Pascal Neis to examine a cohort of new Dutch OpenStreetMap contributors from 2014 and 2015.<br />
<br />
The usual hope expressed by people doing this type of analysis with OSM data is that by better understanding of these contributions we can improve the number of people who continue to contribute after the initial sign-up and first edit.<br />
<br />
My perspective is slightly different, because it is coloured by knowledge of the much longer history of biological recording.<br />
<br />
<br />
<a name='more'></a>In Britain this can roughly be dated to the 17th century, and <a href="https://en.wikipedia.org/wiki/John_Ray">John Ray</a>'s publication of a flora of the <a href="https://books.google.co.uk/books?id=hz0-AAAAcAAJ&dq=plantarum+circa+Cantabrigiam&source=gbs_navlinks_s">Cambridge area</a>. In the early days compilation of records relied on exchanges of letters, but by the 18th century the collation of data had grown to the extent that for many counties it was possible to produce a flora for the local area. For instance in Nottinghamshire, the first flora was produced by <a href="http://www.biodiversitylibrary.org/item/188261#page/245/mode/1up">Deering </a>in 1738, followed by a second by Ordanyo in 1807. (In fact there was another written in the 1830s, followed by a gap of over hundred years until the last county flora was written by the Howitts in 1963). The sheer scope of this literature can be seen in Tim Rich's digitised copy of Simpson's 900 odd page <i><span style="font-size: 12pt;">A Bibliographical Index of the British Flora</span></i><span style="font-size: 12pt;"> (see the <a href="http://archive.bsbi.org.uk/index.html">BSBI website</a>). Here's a small extract of works which mention aspects of the Nottinghamshire Flora from around 1750-1825</span><br />
<span style="font-size: 12pt;"><br /></span>
<br />
<blockquote class="tr_bq">
<span style="font-size: 12pt;"><span style="font-size: x-small;">(Nottinghamshire) A catalogue of plants ... about Loughborough; R. Pulteney 1747; Manuscript Leicester Museum 1749, and library Linnean Society</span></span><br />
<span style="font-size: 12pt;"><span style="font-size: x-small;">Nottingham; Historical account of the town of, C. Deering 1751, 90.<br />(Nottinghamshire) An account of some of the more rare English plants observed in Leicestershire; R. Pulteney Philosophical Transactions XLIX 2 (1757) 803.<br />(Nottingham) A catalogue of some of the more rare plants found in the neighbourhood of Leicester, Loughborough and in Charley Forest; R. Pulteney Philosophical Transactions XLIX (1757) 803, 866; and in ‘The history ... of Leicester', I. Nichols I (1795) clxxvii.<br />Nottinghamshire. Plantae Cantabrigienses; T. Martyn 1763, 83, [from Deering's Catalogue].<br />Nottinghamshire The present state of all nations; T. Smollett II (1768) 408.<br />Nottingham. Description of England and Wales; [Society of Gentlemen] VII (1769) 135.<br />Nottingham. The complete English traveller; N. Spencer 1771, 495; 1773, 495.<br />(Nottinghamshire) Manuscript notes in Ray's Synopsis iii; J. Lightfoot [died 1788] & J. Hill [died 1775] library Botany Department Oxford.<br />Nottingham; Topographical and statistical description of the county of, G. A. Cooke [c. 1802-10] 121.<br />Nottinghamshire. Botanist's guide; D. Turner & L. W. Dillwyn II (1805) 482.<br />(Nottinghamshire) Midland flora; T. Purton 1817 2 volumes; appendix, parts 1, 2 & 3 (in 2 parts) 1821.<br />Nottinghamshire. The scientific tourist in England ...; T. Walford II (1818).<br />Nottinghamshire. The new British traveller; J. Dugdale IV (1819) 6.<br />Nottinghamshire; Botanical calendar for, T. Jowett 1826. By "Il Rosajo" in local paper*.</span></span>
</blockquote>
Perhaps my favourite quote from this period is about the Maiden Pink <i>Dianthus deltoides</i>, for instance in an <a href="https://ebooks.adelaide.edu.au/c/camden/william/britannia-gibson-1722/p14010002.html">english language</a> edition of <a href="https://en.wikipedia.org/wiki/William_Camden#Britannia">Camden's Britannia</a> around 1722 :<br />
<blockquote>
John Ray says, " I find this to be the same pink
which groweth so plentifully by the road side on the sandy hill you ascend going from
Lenton to Nottingham." <i>Catalogus Plantarum</i> 2 ed., 1677. p. 57.
</blockquote>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJq1JQvTjzgJTp_9YVWu3grV3T_d7jEwUxeMB_g2RthVmgKcpx6MDZgocAf2dHYSs8TPEKC45WP3r4JUpOeBnArLbrMdpVpIy4Ga_uwn0WD42yygTH-BfB35J_0zcrYZ-aOjzyDTRSoLeR/s1600/IMG_5414-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJq1JQvTjzgJTp_9YVWu3grV3T_d7jEwUxeMB_g2RthVmgKcpx6MDZgocAf2dHYSs8TPEKC45WP3r4JUpOeBnArLbrMdpVpIy4Ga_uwn0WD42yygTH-BfB35J_0zcrYZ-aOjzyDTRSoLeR/s320/IMG_5414-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Dianthus_deltoides">Maiden Pink</a> in Nottingham,<br />
sadly not native but an escape from a green roof.<br />
Photo: copyright the author</td></tr>
</tbody></table>
<br />
<br />
Ray's correspondence has more (gruesome detail):<br />
<br />
<div style="text-align: center;">
<br />
<iframe frameborder="0" height="430px" src="https://archive.org/stream/correspondenceof48rayj?ui=embed#page/48/mode/1up" width="480px"></iframe>
</div>
<br />
<div style="text-align: justify;">
These details just emphasise how much has changed: the sandy hill between Lenton & Nottingham is a four-lane road, Derby Road. This <a href="http://www.mapillary.com/map/im/6Got-g2pXQmqT0a8ngdnKw/photo">Mapillary </a>sequence is roughly in the relevant location.</div>
<br />
<iframe frameborder="0" height="580px" src="http://www.mapillary.com/jsapi/?image=6Got-g2pXQmqT0a8ngdnKw&showMap=false" width="480px"></iframe>
Indeed the area was long known as Lenton Sands, although this name is, I think, falling into disuse.<br />
<br />
I wasn't not sure where the gallows were located, but a quick web search shows that <a href="http://www.ournottinghamshire.org.uk/page_id__943.aspx">they</a> were close to what is now the junction of <a href="http://www.openstreetmap.org/#map=16/52.9642/-1.1533">Forest and Mansfield Roads</a>.<br />
<br />
Another evocative historic plant location is <a href="https://en.wikipedia.org/wiki/Nottingham_Castle">Nottingham Castle Rock</a>. This is the type locality for the <a href="https://en.wikipedia.org/wiki/Silene_nutans">Nottingham Catchfly, <i>Silene nutans</i></a>, but it is long since extinct in the area. However, at the foot of the rock is another plant location known since John Ray's time: here Alexanders, <a href="https://en.wikipedia.org/wiki/Smyrnium_olusatrum">Smyrnium olusatrum</a>, still grows in a small patch along Peveril Drive.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik7g5vc0SiB3MOricskgnZDvU-xNc5uOAg8uUqtVVAKq29EgUCd8-twZ-qgK5k3oZaWy_nqhh46Nvb9y1EHg9ONpMiGI1BeeRf8DI3umZvPHzdCN7ZZWDVMGGZlTeaDiQapBiqfs7VwdMv/s1600/IMG_0148a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik7g5vc0SiB3MOricskgnZDvU-xNc5uOAg8uUqtVVAKq29EgUCd8-twZ-qgK5k3oZaWy_nqhh46Nvb9y1EHg9ONpMiGI1BeeRf8DI3umZvPHzdCN7ZZWDVMGGZlTeaDiQapBiqfs7VwdMv/s320/IMG_0148a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Foot of Castle Rock, Nottingham.<br />
The gates are the old entrance gates to The Park Estate.<br />
The green plants behind the left-hand gate are Alexanders.</td></tr>
</tbody></table>
These little examples are just some of the one's most familiar to me. Throughout the British Isles there are thousands of such places where rare plants grow which have been known for hundreds of years. Occasionally the places get lost, and then by chance, or as the result of diligent research and field work, get relocated (see <a href="http://www.museumwales.ac.uk/articles/2009-04-01/The-rarest-plants-in-Wales/">here</a> for examples from Wales).<br />
<br />
It's not just plants either, a friend, David Brown, runs a regular field course in Scotland called <a href="http://staging.field-studies-council.org/individuals-and-families/courses/2015/kd/special-spring-moths-52071.aspx"><i>Special Spring Moths</i></a>. He takes the participants to locations which have been known for at least 170 years to see such exotica as the <a href="http://www.ukmoths.org.uk/species/brachionycha-nubeculosa">Rannoch Sprawler</a> and the <a href="http://www.ukmoths.org.uk/species/lycia-lapponaria/female/">Rannoch Brindled Beauty</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ABrachionycha_nubeculosa2.JPG" style="margin-left: auto; margin-right: auto;" title="By Adam Furlepa (Own work) [CC BY-SA 4.0 (http://creativecommons.org/licenses/by-sa/4.0)], via Wikimedia Commons"><img alt="Brachionycha nubeculosa2" src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Brachionycha_nubeculosa2.JPG/512px-Brachionycha_nubeculosa2.JPG" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A Rannoch Sprawler in Poland, a rather better photo than <a href="https://flic.kr/p/4Dkhkn">mine </a>of a Scottish moth.<br />
By Adam Furlepa <a href="http://creativecommons.org/licenses/by-sa/4.0">CC BY-SA 4.0</a>, <a href="https://commons.wikimedia.org/wiki/File%3ABrachionycha_nubeculosa2.JPG">via Wikimedia Commons</a></td></tr>
</tbody></table>
It should be obvious by now, as to why naturalists are often keenly interested in topography. Rarer species highlight why accurate records and a means of sharing them have been so important to British naturalists for such a long time.<br />
<br />
Until the mid-20th century the principle means of record keeping were personal card indexes (David Brown still does things this way). Collation of records over a particular area would be entrusted to a particularly knowledgeable and enthusiastic individual. The ideal was that these records would be periodically consolidated and published, either as a journal article, or, for larger groups, as a book.<br />
<br />
In 1964 this changed when the <a href="https://en.wikipedia.org/wiki/Biological_Records_Centre">Biological Record Centre</a> was set-up and records started to be computerised. Slightly earlier, in 1962, the first <a href="https://en.wikipedia.org/wiki/Atlas_of_the_British_Flora">Plant Atlas</a> of Britain & Ireland was published. I think this was the first major publication to organise records on the basis of Ordnance Survey grid squares, which had only appeared relatively recently on consumer, rather than military, map products (the <a href="http://wiki.openstreetmap.org/wiki/New_Popular_Edition">New Popular Edition</a>). Since then major atlases have appeared for a number of groups, with birds and plants each having several editions.<br />
<br />
For these better known groups, atlas data pertains to surveys carried out in a defined period. For instance the <a href="http://www.bto.org/volunteer-surveys/birdatlas">last Bird Atlas</a> surveys ran from 2007-2011, and the next plant atlas recording period runs until <a href="http://www.bsbi.org.uk/atlas_2020.html">2020</a>. For most insects (the exceptions being <a href="http://butterfly-conservation.org/111/butterflies-for-the-new-millennium.html">Butterflies </a>and <a href="http://www.british-dragonflies.org.uk/content/national-dragonfly-atlas">Dragonflies</a>) there are just not enough people interested, or with the specialist knowledge, for such an effort. For these groups, all known records, however old, need to be used. There are currently around 100 beetles known from Nottinghamshire, where the last known sighting was pre-1916. But they can be re-found, as was the nationally rare Hazel Pot Beetle, <a href="https://en.wikipedia.org/wiki/Cryptocephalus">Cryptocephalus </a>coryli, by <a href="http://www.eakringbirds.com/eakringbirds6/insectinfocuscryptocephaluscoryli.htm">Trevor & Dilys Pendleton</a> in 2008. So these older records can still be totally relevant today.<br />
<br />
Now to return to the long tail phenomenon.<br />
<br />
The point about the various BRC Atlas initiatives is that, in addition to all the information about plants and animals, they also form a range of large and valuable datasets for understanding aspects of how people contribute data in this type of undertaking. (Of course they don't answer the <b>WHY?</b>, but the National Biodiversity Network has recently <a href="http://nbn.org.uk/News/Latest-news/NBN-Recorder-Motivation-Research-summary-findings.aspx?utm_source=eshot&utm_medium=email&utm_campaign=NBNeNewsFebruary2016_1">surveyed contributor</a>s, and answers there may be of interest to people concerned with similar issues with OpenStreetMap).<br />
<br />
I, of course, don't have access to any of these large datasets at a granularity at which one can ask questions about relative frequency of contributions. However, I have contributed to a small niche dataset, that for <a href="https://en.wikipedia.org/wiki/Tephritidae">Tephritid </a>flies (often called Picture-winged Flies, or Gall Flies, but more widely called Fruit Flies; unfortunately in Britain fruit fly usually means <i>Drosphila melanogaster</i> which belongs to a <a href="https://en.wikipedia.org/wiki/Drosophilidae">different family)</a>. These are small, but distinctive flies most of which spend their larval phase feeding on fruits of various plants, some galling their hosts. They are common, but not well recorded.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/6016864881/in/photolist-9S5XLt-bzr1nw-a7icYc-a7icpT-a7m4fs-a7m47h-aaG1iF-aaJQdE-aaG1oB-9S5XHK-9S8SNj-9S8SCU-5PTALX-5f2qr6-56qHaA-51L6dc-51Qj21-4W3aT2-4W7pfd-LVj3Z" style="margin-left: auto; margin-right: auto;" title="Tephritis bardanae (m)"><img alt="Tephritis bardanae (m)" height="640" src="https://farm7.staticflickr.com/6005/6016864881_181814a7a0.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Tephritis_bardanae"><i>Tephritis bardanae</i></a> on <a href="https://en.wikipedia.org/wiki/Arctium_tomentosum"><i>Arctium tomentosum</i></a>, <a href="https://en.wikipedia.org/wiki/Puchberg_am_Schneeberg">Puchberg-am-Schneeberg</a>, Austria, 2011<br />
Taken the day before SotM-EU 2011.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Laurence Clemons, the co-ordinator of the scheme, is a near contemporary from university. He has run it for over 30 years, combining it with his day job of teaching. From time to time he circulates updated maps for each tephritid species. Fortunately for me he also includes a list of all recorders with the number of records and species submitted. The records go back to the 19th century, so although the data set is relatively small, 3 572 observations, it has both broad temporal and geographical extent. I have used this list from around 2008-9 to look at recorder contribution distributions.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/mausboam/5979022584/in/photolist-9S5XLt-bzr1nw-a7icYc-a7icpT-a7m4fs-a7m47h-aaG1iF-aaJQdE-aaG1oB-9S5XHK-9S8SNj-9S8SCU-5PTALX-5f2qr6-56qHaA-51L6dc-51Qj21-4W3aT2-4W7pfd-LVj3Z" style="margin-left: auto; margin-right: auto;" title="Terellia tussilaginis (m) : 9037a"><img alt="Terellia tussilaginis (m) : 9037a" height="375" src="https://farm7.staticflickr.com/6142/5979022584_c6db6cbf41.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A male of <i><a href="https://en.wikipedia.org/wiki/Terellia">Terellia </a>tussilaginis</i> on <a href="https://en.wikipedia.org/wiki/Arctium_minus"><i>Arctium minus</i></a>, Nottingham.<br />
Despite the name, these insects only feed on Burdocks.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
A quick scan of the people contributing records shows many entomologists who are specialists in other fields, for instance Raymond Uffen, a lepidopterist with 1 record, Brian Spooner, former head of Mycology at Kew Gardens, with 2 records. The people with lots of records are well known dipterists: <a href="http://www.omnesamici.co.uk/CGSthenandnow/WhiteleyDerek.html">Derek Whiteley</a> (371 records) from the Sheffield area, <a href="http://www.stevenfalk.co.uk/">Steven Falk</a> (865) from Warwickshire, John Coldwell (170) from Barnsley. Quite a few are museum professionals: Bill Ely (161), Jerry Bowdrey (184). Many of the recorders with a small number of records will be general naturalists who happen to come across galls, flies or leaf mines in the course of other surveying work.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwI7dBYNbkPqZb7VnUArw68pYEbzb_GSE0lYy-0ML6Eq-nV2KO55NpWD1qy7cj8TEitaOrzzyIIE3Oiy4lXfEeZpxuG02uQmcg1YQi-0cHBhF4ISDC9GTmswp-blVWb2bDshHzhRVlH7Tf/s1600/carr_I_p468_1L.tif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwI7dBYNbkPqZb7VnUArw68pYEbzb_GSE0lYy-0ML6Eq-nV2KO55NpWD1qy7cj8TEitaOrzzyIIE3Oiy4lXfEeZpxuG02uQmcg1YQi-0cHBhF4ISDC9GTmswp-blVWb2bDshHzhRVlH7Tf/s320/carr_I_p468_1L.tif" width="422" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A page from Carr (1916) show early records of Tephritidae in Nottinghamshire<br />
Not all of these records have found there way to the national scheme.</td></tr>
</tbody></table>
I don't recognise many names of historical, rather than active recorders, but two stand out: J.L. Carr (1 record), and J.W. Saunt (209), see short biographies at <a href="http://www.coleopterist.org.uk/biogdict/c.htm">The Coleopterist</a>. Carr was the author of the formidable Invertebrate Fauna of Nottinghamshire (1916), keeper of the Nottingham Natural History Museum and Professor of Zoology. I know less of Saunt, other than he contributed many records to Carr's book, but he was apparently a car-body builder in Coventry.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgza8UTn6DXAdFaljTsyawm_qNxoEQ7XS2uTTMAB6rE1Zu1P1AQavziEqn_caDykFtpT9ICzKpZPmNAOB81r-wSTstHIPSK-SklCUzhqfYfBZ_RcBQznQ9Lbp5vi4BVSsACk_Ja4K07D55L/s1600/tr_graph3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgza8UTn6DXAdFaljTsyawm_qNxoEQ7XS2uTTMAB6rE1Zu1P1AQavziEqn_caDykFtpT9ICzKpZPmNAOB81r-wSTstHIPSK-SklCUzhqfYfBZ_RcBQznQ9Lbp5vi4BVSsACk_Ja4K07D55L/s320/tr_graph3.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Records / recorder GB & Ireland Tephritid Recording Scheme 2008</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
I took the list of records from Laurence's PDF and lightly munged the data into a CSV file which I then pulled into R to plot some histograms. (As usual I find I always have to consult a few webpages to even get started again with R after a break of a few weeks).<br />
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
Of course, and entirely as expected, the histogram shows an exponential decay of numbers of recorders against records. This is exactly like the graph which Marc created which set me of looking at these numbers. The extreme outlier with nearly 2000 records, is, again, as might be expected, Laurence Clemons. No-one devotes their leisure activity to running a recording scheme unless it is a passion. Also note that 2000 records implies considerably less than 100 a year: each record probably represents significant effort.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
If I change the size of bins and exclude the more extreme outlier values, the graph looks remarkably the same! Below I show the graph with 50 bins for the visible range with an upper limit of 200 and then 50.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG890u1CUKU0SMhBai65LllqRchr5ZAoCvbDJNz9Kj_L114tOhqkwkZCkrzMddG49ZHaBVW64EMicIm1D5q87f-jN2Mu5nS_Lp_5tGFr1x2Url3NCX0SnfWrwf7rU0AGqeDL80V2V33wkD/s1600/tr_graph4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG890u1CUKU0SMhBai65LllqRchr5ZAoCvbDJNz9Kj_L114tOhqkwkZCkrzMddG49ZHaBVW64EMicIm1D5q87f-jN2Mu5nS_Lp_5tGFr1x2Url3NCX0SnfWrwf7rU0AGqeDL80V2V33wkD/s320/tr_graph4.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The same graph as above, but for recorders with under 200 records</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkF7oOgs1BnFx9M9sRwwMQdlQ5J0j9rVoWbxgTm71gImbpVmjVQ0YY0sq7nUyDWWAJSzWt2y45DS0dUH9ZQoe8UNiRMcE0xLWgLMYCx1fnIeTP4jzdzLyX241Z-vAkUsAbUcr39Pc8D160/s1600/tr_graph5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjkF7oOgs1BnFx9M9sRwwMQdlQ5J0j9rVoWbxgTm71gImbpVmjVQ0YY0sq7nUyDWWAJSzWt2y45DS0dUH9ZQoe8UNiRMcE0xLWgLMYCx1fnIeTP4jzdzLyX241Z-vAkUsAbUcr39Pc8D160/s320/tr_graph5.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The same graph as above, but for recorders with under 200 records<br />I'm in the 16 records bin.</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: justify;">
This is only one dataset. I'm pretty confident that the same patterns will be seen in other atlas datasets, and, as I stated at the outset, in other citizen science datasets. It's pretty much the same pattern we see when we look at OSM contributions.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
For me the main point about this is to emphasise that there is much to learn from the long experience of Biological Recording. I think the idea that somehow we can do magical things to change the shape of the contribution curve is belied by these other data collection experiences. Indeed, the co-ordinator of the fantastically successful <a href="http://www.earthwormsoc.org.uk/earthworm-identification/recording-earthworms">National Earthworm Recording</a> scheme <a href="https://www.timeshighereducation.com/comment/6510#comment-6510">wrote recently</a> that "I speak from experience when I say that teaching someone to identify a
group does not make them record (and build up the necessary experience
to become an expert)". Therefore we should accept that it is most likely a reality which we need to recognise.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
This does not of course mean that we should cease in efforts to make OpenStreetMap look: </div>
<ul>
<li> <b>important </b>(for instance as Missing Maps & HOT have done most successfully);</li>
<li> <b>useful </b>(Richard Fairhurst's 'driven by cyclist' <a href="https://lists.openstreetmap.org/pipermail/talk-us/2015-June/014991.html">idea</a>);</li>
<li> <b>fun </b>(it is, really!);</li>
<li> <b>interesting </b>(that too); </li>
<li> or just providing <b>an excuse to get outside</b> (.</li>
</ul>
<div class="separator" style="clear: both; text-align: justify;">
OSM is all of these things; it's also <b>educative</b>, a way to <b>meet like-minded folk</b>; a way of learning <b>new skills</b> (mainly in informatics, but also observational). Nor is this pattern a reason not to strive for a more diverse community of contributors. In fact, the very likelihood that mappers "<a href="https://www.cl.cam.ac.uk/~dh475/papers/icwsm13.pdf">are made not created</a>" tells us that new contributors are as important as ever.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
OSM is different in one other important way from biological records: there is a founder effect. </div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
If I see a bird in one place today and the same species tomorrow both are useful records. In OSM if something is already mapped one can't contribute it again (although one can alter what already exists). Over time the really easy things get mapped leaving those that are harder, more tedious or just less useful. This probably doesn't matter too much for the already engaged, but it does possibly put a limit on what a newcomer might feel able to do. However, to date, there is no sign of this happening: see Simon Poole's recent <a href="http://www.openstreetmap.org/user/SimonPoole/diary/36408">diary post</a> on the subject.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<div class="separator" style="clear: both; text-align: justify;">
On a more general level, as I said at the outset of this piece, I hope to see some much more detailed and meticulous academic research in the near future. There are now hundreds of different citizen science datasets which can be examined across a range of different domains and types of data acquisition.</div>
<div class="separator" style="clear: both; text-align: justify;">
<br /></div>
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com1tag:blogger.com,1999:blog-2082422783692631078.post-27442746597973377512016-01-18T12:38:00.002+00:002016-01-18T12:38:54.485+00:00UK Open Data and Buildings in OpenStreetMapI've finally (after 8 months) got around to looking
at the OpenMap Local buildings. This new dataset was launched at the first OpenDataCamp, and I've had the <a href="http://www.cucaera.co.uk/grp/?refs=SU&map=road&vcs=true&graticule=true&nsgrids=false&zoom=9&lat=51.31580815769595&lon=-1.6900517278851757">SU</a> 100 kilometre square data on the PC since then (it's contains Southampton, where Ordnance Survey are based). I use Meridian 2 OS Open Data regularly and extensively, but these days don't make much use of the larger scale vector data.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22949197830/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Nottingham City Centre: OSM/OSGB Building Comparison"><img alt="Nottingham City Centre: OSM/OSGB Building Comparison" height="354" src="https://farm6.staticflickr.com/5624/22949197830_6c43f42980.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of Building polygons for Central Nottingham<br />
OSM has more detail and does not merge discrete buildings.<br />
<span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database right 2015. Image CC-BY-SA, the author.</span></td></tr>
</tbody></table>
I needed them for something else which
caused me to download the <a href="http://www.cucaera.co.uk/grp/?refs=SK&map=road&vcs=true&graticule=true&nsgrids=false&zoom=8&lat=53.17694393740821&lon=-2.071888715440977">SK</a> data. Co-incidentally Christian Ledermann had asked on <a href="https://lists.openstreetmap.org/pipermail/talk-gb/2015-November/018049.html">talk-gb</a> about using this data to add buildings to OpenStreetMap for <a href="https://en.wikipedia.org/wiki/Newark-on-Trent">Newark-on-Trent</a>. A little earlier the Environment Agency had released Lidar data for England, and this is also useful as input for mapping buildings.<br />
<br />
<h3>
OpenMap Local </h3>
Apart from the area I originally
needed which were in <a href="http://www.cucaera.co.uk/grp/?refs=SK41&map=road&vcs=true&graticule=true&nsgrids=false&zoom=12&lat=52.7390913199615&lon=-1.3864769678061384">SK41</a> (no buildings in OSM), I've also looked at
areas which I know much better & compared some selected areas where
we have good building coverage around Nottingham. The comparisons I made are shown visually, with my main observations summarised at the end. Note that comparisons have not been made on any systematic basis.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22648540073/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="uon_univertiy_park"><img alt="uon_univertiy_park" height="256" src="https://farm6.staticflickr.com/5642/22648540073_cc0e781001.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">University Park, University of Nottingham<br />
an area of predominantly large academic buildings.<br />
OpenStreetMap and OS OpenMap are largely in agreement: the minor differences applying to newer buildings which post-date the Bing imagery.<br />
<span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database
right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data
from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database
right 2015. Aerial Imagery via Bing, (c) as in image. Image CC-BY-SA, the author.</span></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22648540053/in/photostream/" style="margin-left: auto; margin-right: auto;" title="uon_science_city_buildings"><img alt="uon_science_city_buildings" height="256" src="https://farm1.staticflickr.com/639/22648540053_2c1591a02a.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Science City part of the University Park campus.<br />
A new large lecture theatre block is not present in OpenMap data, and the outline of the building top centre (Tower Building) is over-simplified.<br />
<span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database
right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data
from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database
right 2015. Image CC-BY-SA, the author.</span></td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22880884667/in/photostream/" style="margin-left: auto; margin-right: auto;" title="newark_buildings2"><img alt="newark_buildings2" height="256" src="https://farm6.staticflickr.com/5734/22880884667_1b6ac26674.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Central Newark. OpenMap vs Lidar.<br />
Many instances of building merging & over-simplification are apparent here, notably with the outline of the parish church.<br />
<span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database
right 2015, Lidar data
from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database
right 2015. Image CC-BY-SA, the author.</span></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22880884627/in/photostream/" style="margin-left: auto; margin-right: auto;" title="newark_buildings1"><img alt="newark_buildings1" height="256" src="https://farm6.staticflickr.com/5638/22880884627_4ac089031d.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Newark-on-Trent, residential areas, Showing inconistency in size for similar houses, and merging of terraced housing.<br />
<span style="font-size: xx-small;">Contains Ordnance Survey data (c) copyright and database
right 2015, OSM data (c) OpenStreetMap contributors 2015, Lidar data
from Environemnt Agency under OGL 3.0, (c) Crown Copyright and database
right 2015. Image CC-BY-SA, the author.</span></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
I have not made
systematic comparisons, but these are my main observations (in brackets the 1km grid square where I've noted any particular issue):
<br />
<ul>
<li><b>Best for larger buildings</b>.
The data seem much more reliable (actually matching building footprints
fairly well) for larger buildings. Even for large detached houses I
would regard the data as unreliable: on our road of 40 detached houses,
at least 16 are represented as terraces (SK5439). Similar artefacts occur in
other areas with detached houses: apparently caused when a garage is
close to both houses. Smaller houses are inherently simplified: no
better than drawing one in JOSM and then copying the outline in fact.</li>
<li><b>Building fusion</b>. This is particularly clearly seen in the city centre image, where a whole block of buildings has been simplified to a single building (centre of image), but also occurs in suburban housing (see above).</li>
<li><b>Inconsistency in geometry simplification</b>. This is most noticeable in the city centre. (SK5739). For instance compare the OSM and the OpenMap Local outlines for St Peter's Church (bottom right in map above). In OpenMap Local the church is just shown as a rectangle, whereas in practice it is more complex. Modern buildings on the Jubilee Campus of Nottingham University are generally shown with more detail.</li>
<li><b>Inconsistency in building size</b>. In SK5439 there are a very large number of houses which were identical when built. However, in the OpenMap Local they are often of different sizes. (This is also probably true of OSM, if buildings have not been created by duplication). </li>
<li><b>Voids</b>. Gaps between closely packed buildings in the city centre appear slightly arbitrary in both placing and whether such a void exists or not.</li>
<li><b>Some selection inconsistency</b> with small size buildings. Only 2 garages are shown in an area of around 500 houses. With OSM the figure is nearer 200+. (SK5439)</li>
<li><b>Demolished buildings</b>.
Whilst I would not expect the data to show the building demolished in
the past month, I would expect it to not show one demolished 2 years
ago, and I would certainly expect it not to show one demolished in 1970
(although MasterMap shows <a href="http://sk53-osm.blogspot.co.uk/2011/01/how-good-is-mastermap.html">this too</a>). (SK5439)</li>
<li><b>Better locational accuracy</b>.
If using the full transform it may be useful to take advantage of the
better locational accuracy of this data. In the main OSM buildings are
rarely more than 3 m displaced from the OS OpenMap Local. (SK5439) In general the more recently mapped buildings in Nottingham
city centre have better locational accuracy than this (SK5739).</li>
</ul>
Taken together, my use of this directly within OSM would be along the following lines :<br />
<ul>
<li><b>Selective
transfer of larger buildings</b> (schools, offices, public buildings,
factories, warehouses, larger shops) on a case-by-case basis from a
shapefile to a JOSM editing layer, or to Potlatch 2. Some minor refinement will probably
be needed (for instance a university building here has long narrow
courtyards which act as light wells which are not shown in OpenMap
Local.</li>
<li><b>Only use</b> it for <b>houses </b>and similar when shapes are <b>very
simple</b> and everything has been double checked, at the very least,
against aerial imagery. For simple shapes it's as quick to draw &
copy in JOSM anyway. A similar principle holds for more complex building shapes on
modern estates, where one building can be cloned.</li>
<li><b>Watch out </b>for demolished buildings. This requires not just checking against Bing/MapBox imagery, but some local knowledge for sense checking.</li>
</ul>
<h3>
Environment Agency Lidar Data</h3>
Another source of building data is the recently released Environment Agency Lidar data. This does not cover the whole country, and in many places may only be at 1 or 2 m resolution. It may also be quite old. However, because it does not suffer from parallax artefacts it can be used in conjunction with both aerial imagery (whether from Bing, MapBox or more local sources) and OS OpenData. I have provided examples from Nottingham, Newark, and Melton Mowbray of this data, combined with one or more of OSM buildings data, OS OpenMap or Bing aerial imagery.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://wiki.openstreetmap.org/w/images/a/a2/Melton_dsm.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://wiki.openstreetmap.org/w/images/a/a2/Melton_dsm.png" height="285" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Melton_Mowbray">Melton Mowbray</a>. EA Lidar DSM (1m) overlaid on OSM. <br />
The Lidar data was used to refine the OSM building outlines<br />
which originally were traced from OS StreetView as block-sized polygons.<br />
(see commentary)</td></tr>
</tbody></table>
Melton Mowbray illustrates many of the benefits of Lidar data. It is a fairly typical country town, with many of the buildings in the town centre ranging in age from 10 to 500 years old. Many extend back from the street in a series of outbuildings (e.g., stables) which have eventually been incorporated into the main building, but this process leaves lots of small courtyards, service yards, etc which are more or less impossible to discern on aerial imagery.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/24431405886/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Butter Cross on Market Place, Melton Mowbray"><img alt="Butter Cross on Market Place, Melton Mowbray" height="375" src="https://farm2.staticflickr.com/1640/24431405886_024bc6c5c8.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Butter Cross in Market Place, Melton Mowbray<br />
Despite the different styles & ages of the buildings, several have long ranges at the rear.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
By doing a street-level ground survey one can identify which buildings are distinct on the street front. Lidar than helps to construct a building outline which is consistent with this. I surveyed the cetre of Melton in September, and this was the first place where I used Lidar data to aid in the interpretation of aerial imagery. In this case I find it essential to have adequate street level pictures to be able to relate to the aerial imagery: most useful are the presence and distribution of chimneys: because they throw shadows they are often visible even on poor quality imagery.<br />
<br />
The Lidar data also allows one to do some other things: notably find building heights. I've done this for a 1980's estate on the edge of Maidenhead: particularly easy as the residential buildings fall into a small number of categories: bungalows, two-storey-houses & maisonettes (purpose built flats in a house-like structure.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://wiki.openstreetmap.org/w/images/3/3a/House_Heightw_from_Lidar.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://wiki.openstreetmap.org/w/images/3/3a/House_Heightw_from_Lidar.png" height="308" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A 1980s housing estate with building heights mapped from <a href="http://wiki.openstreetmap.org/wiki/English_Environment_Agency_LIDAR" title="English Environment Agency LIDAR">English Environment Agency LIDAR</a>
Open Data. Buildings fall into 3 height categories: bungalows (green:
approx 4m high), 2-storey houses of various kinds (blue: approx 6 m
high), and maisonettes (condominiums) which are about 7 m high (red).
Heights were calculated in m, so the values represent minimum heights of
the highest part of the building, which is nearly always the gable line.<br />
Outpur via Overpass Turbo, styled with MapCSS.</td></tr>
</tbody></table>
There are many other useful blog posts about using this Lidar data, both specifically for OSM, but also generally. See posts by Chris Hill (<a href="http://chris-osm.blogspot.co.uk/2015/09/more-lidar-goodness.html">"More Lidar Goodness"</a> and <a href="http://chris-osm.blogspot.co.uk/2015/09/extracting-building-heights-from-lidar.html">"Building Heights</a>") and <a href="http://wiki.openstreetmap.org/wiki/User:EdLoach/Tendring_LIDAR">Ed Loach</a> for some of the specifics, and the write-up on the <a href="http://wiki.openstreetmap.org/wiki/English_Environment_Agency_LIDAR">wiki</a>. A <a href="http://www.emu-analytics.com/blog/post.php?s=2016-01-11-2-for-the-price-of-1">nice post </a>and <a href="http://www.emu-analytics.com/buildingheights">map</a> (v. slow in my browser) showing building heights in London on OpenMap Local may also be of interest. HousePrices has processed all the Lidar data from EA and Natural Resources Wales as a <a href="https://houseprices.io/lab/lidar/map">hillshaded slippy map</a> which is useful to look at what is available. Slightly unfortunately the map is in OSGB projection (ESPG:27700) and is not shown with other slippy maps which would make it a bit easier to locate oneself.<br />
<h3>
What kind of building data should be added to OSM?</h3>
From
past experience single building outlines traced from OS StreetView, turn out to represent tens of buildings on the ground. Such simplified outlines just makes the work of
splitting the buildings properly quite a lot harder. This can be
particularly bad in town/city centres.<br />
<br />
Usually if adding detail of POIs and addresses it is important to have individual buildings mapped: this makes it much easier to correlate photos to roofline features such as chimneys, gables etc. A single very simple outline may be OK, because for more detailed mapping it should just be a question of deleting the original outline. However, the question must be asked, as to what purpose such an outline fulfils on OSM, when the source data can be readily combined with OSM data for downstream consumption.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AGranby_Street%2C_Leicester_(geograph_2296099).jpg" style="margin-left: auto; margin-right: auto;" title="Malc McDonald [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="Granby Street, Leicester (geograph 2296099)" src="https://upload.wikimedia.org/wikipedia/commons/4/4c/Granby_Street%2C_Leicester_%28geograph_2296099%29.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Granby Street, Leicester.<br />
The multiple buildings shown here are represented in OSM as single buildings for <a href="http://osm.org/go/eu5bTUIyE--">each block</a><br /> (imported from OS StreetView Open Data).<br /><a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a> © Copyright <a href="http://www.geograph.org.uk/profile/44954" title="View profile">Malc McDonald</a> and
licensed for reuse under this <a href="http://creativecommons.org/licenses/by-sa/2.0/" rel="license">Creative Commons Licence</a>.</td></tr>
</tbody></table>
I think the fundamental question about straight imports into OpenStreetMap should be "Will it make life easier or harder for subsequent mappers?".<br />
<br />
If the work involved refining a building outline takes longer than re-drawing the building then I doubt if its worth importing the building at all. This is particularly true if the outline is actually of multiple buildings. This is why large building outlines are most valuable: they are generally pretty good compared with what an initial hand-traced outline might look like, and they lend themselves better to stepwise refinement. One group of buildings I find particularly tedious to do well are schools which tend to be a sprawling mass of interconnected buildings. Starting with a decent polygon with orthogonalised angles make adding such detail much easier. The current quarterly project for UK-based mappers might be the time to test this.<br />
<br />
Of course it may be that adding buildings assists in some other mapping goal. I've already mentioned that details of buildings are very useful for addresses. However OpenMap Local lacks the detail in precisely the areas where it would be most useful (city & town centres). For suburban or inner-city housing similar polygons can be created as quickly in OSM editors (notably in JOSM, by duplicating existing buildings or using the Terracer (or even <a href="https://github.com/derickr/josm-plugin-uberterrace">UberTerracer</a>) plugins.<br />
<br />
The other thing which many people want is rendered maps largely derived from OSM, but showing more buildings. In practice, because many mappers do not have the know-how, wherewithal or time to create such a rendered view, they tend to want to import buildings. Historically, OSM tools for importing data are often much easier to use than ways to incorporate the same data <b>and </b>OSM data to render maps and make them accessible on the web. Perhaps we need to do more to help people in the latter task: which is now getting more complicated again with the move to vector tiles (at least outwith use of MapBox Studio), and TileMill's effective status of being a legacy application.<br />
<h3>
Summary </h3>
Sadly, although the new building outlines are better than what preceded them, in most cases they don't offer a decent route for iterative refinement with OpenStreetMap.<br />
<br />
This absence of a simple way to improve building outlines means that ideally people wishing to use this data would merge it with OSM data outside of OSM. I do recognise this is often too much work, or too big a learning curve for many, and consequently there will always be a desire to add buildings to OSM because many people are much more comfortable with consuming only OSM data for their purposes. <br />
<br />
Existing tools for drawing buildings in OSM are pretty powerful & getting more powerful all the time. Many of us, and I include myself in this group, are unaware of the full extent of these utilities. See bdiscoe's <a href="http://www.openstreetmap.org/user/bdiscoe/diary/37576">diary post</a> about mass adjustment of circular buildings (huts) for some insights.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-72549940885068311492015-12-02T12:01:00.002+00:002015-12-02T12:09:42.706+00:00How accurately have Townlands in Northern Ireland been mapped?From time-to-time newly released Open Data provides a nice opportunity to check OpenStreetMap for its accuracy in all its forms (see <a href="http://www.ucl.ac.uk/~ucfamha/OSM%20data%20analysis%20070808_web.pdf">Hakaly </a>(2008) for a breakdown of what this can mean).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540162/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Coastal Townlands, Cos. Derry & Antrim"><img alt="Coastal Townlands, Cos. Derry & Antrim" height="287" src="https://farm6.staticflickr.com/5789/23357540162_3898949e2a.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Coastal Townlands, Counties Derry/Londonderry and Antrim.<br />
<span style="font-size: xx-small;">Boundary lines see below. The deeper the colour of the area, the greater discrepancy in the area of the OSM polygon and the OSNI one. The pale base colour represents a divergence of under 2%. Townlands on the coast and on the UK/Ireland border seem to be most likely to diverge in size. The small cluster centre right is caused by different ways of handling townlands which cross a Civil Parish boundary (OSM & the original source GSGS 3906 split these, the OSNI data does not).</span></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
We have known for a while that both the Ordnance Survey of Northern Ireland and the Ordnance Survey of Ireland were planning OpenData releases. When they came it was all in a rush. For now the hard work starts of checking license conditions for suitability for use in OSM and other places, as well as then working out what is really useful. However, because the townland boundaries of Northern Ireland are complete, it was an ideal opportunity to look at accuracy.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvwnvt4cEUlQLUpop75xRtL5brbWR2kQek09NLKx6JLdZuxzlfz9M9RcJQT6IDXMxMb9BnY5DEMObdeThqAMLDz0PG0ks5ZeEZoTGR6bIscrDr4ESwTIKvpCrjC7W8E3Uk4zPw3cpGsUpr/s1600/IMG_9905a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvwnvt4cEUlQLUpop75xRtL5brbWR2kQek09NLKx6JLdZuxzlfz9M9RcJQT6IDXMxMb9BnY5DEMObdeThqAMLDz0PG0ks5ZeEZoTGR6bIscrDr4ESwTIKvpCrjC7W8E3Uk4zPw3cpGsUpr/s320/IMG_9905a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">View along N side of MacGilligan Peninsula towards Inishown from Umbra</td></tr>
</tbody></table>
<br />
My reasons for doing this are not just pure interest. The usefulness of the <a href="http://sk53-osm.blogspot.co.uk/2015/10/irish-vice-counties-creation-of.html">Irish Vice County</a> boundaries depends of their positional accuracy. Earlier my prediction was that such boundaries ought to be within 10 metres of their true location on the ground where they were based on townland boundaries, but this was largely based on experience with other OSM data rather than an objective statement. Thus investigating the accuracy using an independent data set provides an excellent way of testing this statement. The tests need to be done now, because (as we shall see) the nature of OSM is to fix issues spotted very quickly, and thus datasets become loosely coupled.<br />
<br />
I adopted two approaches:<br />
<ol>
<li>A straight comparison of areas (or their ratios).</li>
<li>Using a series of buffered boundaries from one source (OSNI) and seeing what proportion of the other source (OSM) was included in each buffer.</li>
</ol>
To choose which townlands to compare I followed a suggestion of Rory McCann and for each OSM townland selected the one which shared the most area in common from the OSNI data set. (I have also done it on matching names for a smaller set of data & get similar results). Note that I am comparing townland with townland, not boundary segment with boundary segment. This means that each boundary segment (other than coastal, lacustrine or riverine ones) will be included twice.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23403267576/in/photostream/" style="margin-left: auto; margin-right: auto;" title="umbra_townland_cf"><img alt="umbra_townland_cf" height="354" src="https://farm1.staticflickr.com/646/23403267576_6dc9d77ef9.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Buffering approach to investigating boundary accuracy.<br />
Demonstrated with Umbra townland in County Derry/Londonderry.<br />
This is predominantly coastal sand dunes, with a small river running along its S boundary.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540122/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Northern Ireland Townlands OSNI comparison"><img alt="Northern Ireland Townlands OSNI comparison" height="354" src="https://farm1.staticflickr.com/652/23357540122_e166446070.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Northern Ireland using the same colouring.<br />
At this scale very few boundary mismatches are apparent.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The buffering approach is based on that described by Hakaly (2008). I used buffers of 5, 10, 15 and 20 m, and then clipped the initial OSM way be each in turn. On the scale of the whole country it is clear that most boundaries match closely. This is confirmed by checking what proportion of the boundaries fall into each buffer class: over 80% are within 5m, over 90% within 10m and nearly 95% within 20m.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZIOHXZWd6uqujb1vVWHm1fxxINxFgraUmiy3C8OFWpFpMMtlDnjJfDf6Tl0lYC888CpKtM2O9tpbrEzi-4-hrc1ULfBpDMhuuHItTpKK0RuC2G-j8fth_oiZFAa6j9btY_gfXJTUTEmcv/s1600/townland_graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZIOHXZWd6uqujb1vVWHm1fxxINxFgraUmiy3C8OFWpFpMMtlDnjJfDf6Tl0lYC888CpKtM2O9tpbrEzi-4-hrc1ULfBpDMhuuHItTpKK0RuC2G-j8fth_oiZFAa6j9btY_gfXJTUTEmcv/s400/townland_graph.png" width="500" /></a></div>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Closer inspection (as with the <a href="http://www.openstreetmap.org/relation/4784274">Umbra</a>) shows much of the discrepancy to be present along the coast. This is not surprising, coastlines on OSM were originally derived automatically, and even when refined by hand are unlikely to accord with Mean High Water (MHW). Certainly, for my purposes, it is merely important that the OSM coastlines do not stray above MHW.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540202/in/photostream/" style="margin-left: auto; margin-right: auto;" title="NI Townlands, all boundaries within 5 m of OSNI"><img alt="NI Townlands, all boundaries within 5 m of OSNI" height="354" src="https://farm1.staticflickr.com/679/23357540202_01c4be999f.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">OSM townland Boundaries within 5m of OSNI data</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The analysis described so far focusses on positional accuracy. Looking at areas highlights a range of other accuracy issues.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22840038813/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="townlands_ni_cf9"><img alt="townlands_ni_cf9" height="354" src="https://farm6.staticflickr.com/5658/22840038813_8993ddb144.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Area comparison. Townlands are coloured according to absolute variance of ratio of areas from 1. <br />
The redder they are the further the ratio is from 1.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Area discrepancies of over, say 5%, may be the result of any of the following:<br />
<ul>
<li>Boundary discrepancy (such as coastlines). Mainly caused by coastlines, or difficulty of delineating some boundary feature, such as the course of the Umbra river above) </li>
<li>Erroneous interpretation of the boundary on old maps causing selection of the wrong feature. This transfers land from one townland to another, therefore these should cluster. </li>
<li>Missing townlands. When a single townland has been created without noticing one or more others inside it (<a href="http://www.openstreetmap.org/#map=14/55.0704/-6.5098&layers=N">Town Parks townland</a> at Ballymoney is an example). </li>
<li>Different treatment of townlands bisected by a Civil Parish. See caption of first image above.
Incorrect tagging. </li>
<li>Higher level administrative units having tags appropriate to a townland. I've noted two cases of this one of which was <a href="http://www.openstreetmap.org/relation/5136591">Ballyphilip </a>CP on the Ards peninsula in County Down. </li>
<li>Islands. Some offshore islands appear to be missing from the OSNI data (see The Skerries N of <a href="http://www.openstreetmap.org/#map=14/55.2215/-6.6145">Portrush</a>) </li>
</ul>
We've already caught a few examples in each of these classes through this analysis, and no doubt will find a few more. I have not yet investigated the very apparent discrepancy along the borders.<br />
<br />
To conclude, townland boundaries show exactly the kind of positional accuracy we expected (or perhaps hoped). Perhaps 1% of the total data (90-100 townlands from about 9000) may need some form of correction. I'm biased, but this seems pretty good, for a project principally relying on rectified photo-reduced maps from 1939! It's also worth remembering, that unlike road comparisons, there is no widely available sensor data (ie GPS tracks/point) to help boundary alignments.<br />
<br />
When time permits I'll extend this to include OSI Open Data too. A big thanks to both organisations for releasing their Open Data. OSNI staff have been contributors to OSM for a while: they host <a href="https://twitter.com/osni_maps/status/631832070952779778">Missing Maps lunchtime sessions</a> in their offices.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-2194736705416377552015-12-02T12:01:00.001+00:002015-12-02T12:08:29.834+00:00How accurately have Townlands in Northern Ireland been mapped?From time-to-time newly released Open Data provides a nice opportunity to check OpenStreetMap for its accuracy in all its forms (see <a href="http://www.ucl.ac.uk/~ucfamha/OSM%20data%20analysis%20070808_web.pdf">Hakaly </a>(2008) for a breakdown of what this can mean).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540162/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Coastal Townlands, Cos. Derry & Antrim"><img alt="Coastal Townlands, Cos. Derry & Antrim" height="287" src="https://farm6.staticflickr.com/5789/23357540162_3898949e2a.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Coastal Townlands, Counties Derry/Londonderry and Antrim.<br />
<span style="font-size: xx-small;">Boundary lines see below. The deeper the colour of the area, the greater discrepancy in the area of the OSM polygon and the OSNI one. The pale base colour represents a divergence of under 2%. Townlands on the coast and on the UK/Ireland border seem to be most likely to diverge in size. The small cluster centre right is caused by different ways of handling townlands which cross a Civil Parish boundary (OSM & the original source GSGS 3906 split these, the OSNI data does not).</span></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
We have known for a while that both the Ordnance Survey of Northern Ireland and the Ordnance Survey of Ireland were planning OpenData releases. When they came it was all in a rush. For now the hard work starts of checking license conditions for suitability for use in OSM and other places, as well as then working out what is really useful. However, because the townland boundaries of Northern Ireland are complete, it was an ideal opportunity to look at accuracy.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvwnvt4cEUlQLUpop75xRtL5brbWR2kQek09NLKx6JLdZuxzlfz9M9RcJQT6IDXMxMb9BnY5DEMObdeThqAMLDz0PG0ks5ZeEZoTGR6bIscrDr4ESwTIKvpCrjC7W8E3Uk4zPw3cpGsUpr/s1600/IMG_9905a.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvwnvt4cEUlQLUpop75xRtL5brbWR2kQek09NLKx6JLdZuxzlfz9M9RcJQT6IDXMxMb9BnY5DEMObdeThqAMLDz0PG0ks5ZeEZoTGR6bIscrDr4ESwTIKvpCrjC7W8E3Uk4zPw3cpGsUpr/s320/IMG_9905a.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">View along N side of MacGilligan Peninsula towards Inishown from Umbra</td></tr>
</tbody></table>
<br />
My reasons for doing this are not just pure interest. The usefulness of the <a href="http://sk53-osm.blogspot.co.uk/2015/10/irish-vice-counties-creation-of.html">Irish Vice County</a> boundaries depends of their positional accuracy. Earlier my prediction was that such boundaries ought to be within 10 metres of their true location on the ground where they were based on townland boundaries, but this was largely based on experience with other OSM data rather than an objective statement. Thus investigating the accuracy using an independent data set provides an excellent way of testing this statement. The tests need to be done now, because (as we shall see) the nature of OSM is to fix issues spotted very quickly, and thus datasets become loosely coupled.<br />
<br />
I adopted two approaches:<br />
<ol>
<li>A straight comparison of areas (or their ratios).</li>
<li>Using a series of buffered boundaries from one source (OSNI) and seeing what proportion of the other source (OSM) was included in each buffer.</li>
</ol>
To choose which townlands to compare I followed a suggestion of Rory McCann and for each OSM townland selected the one which shared the most area in common from the OSNI data set. (I have also done it on matching names for a smaller set of data & get similar results). Note that I am comparing townland with townland, not boundary segment with boundary segment. This means that each boundary segment (other than coastal, lacustrine or riverine ones) will be included twice.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23403267576/in/photostream/" style="margin-left: auto; margin-right: auto;" title="umbra_townland_cf"><img alt="umbra_townland_cf" height="354" src="https://farm1.staticflickr.com/646/23403267576_6dc9d77ef9.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Buffering approach to investigating boundary accuracy.<br />
Demonstrated with Umbra townland in County Derry/Londonderry.<br />
This is predominantly coastal sand dunes, with a small river running along its S boundary.</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540122/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Northern Ireland Townlands OSNI comparison"><img alt="Northern Ireland Townlands OSNI comparison" height="354" src="https://farm1.staticflickr.com/652/23357540122_e166446070.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Northern Ireland using the same colouring.<br />
At this scale very few boundary mismatches are apparent.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The buffering approach is based on that described by Hakaly (2008). I used buffers of 5, 10, 15 and 20 m, and then clipped the initial OSM way be each in turn. On the scale of the whole country it is clear that most boundaries match closely. This is confirmed by checking what proportion of the boundaries fall into each buffer class: over 80% are within 5m, over 90% within 10m and nearly 95% within 20m.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZIOHXZWd6uqujb1vVWHm1fxxINxFgraUmiy3C8OFWpFpMMtlDnjJfDf6Tl0lYC888CpKtM2O9tpbrEzi-4-hrc1ULfBpDMhuuHItTpKK0RuC2G-j8fth_oiZFAa6j9btY_gfXJTUTEmcv/s1600/townland_graph.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZIOHXZWd6uqujb1vVWHm1fxxINxFgraUmiy3C8OFWpFpMMtlDnjJfDf6Tl0lYC888CpKtM2O9tpbrEzi-4-hrc1ULfBpDMhuuHItTpKK0RuC2G-j8fth_oiZFAa6j9btY_gfXJTUTEmcv/s400/townland_graph.png" width="500" /></a></div>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Closer inspection (as with the <a href="http://www.openstreetmap.org/relation/4784274">Umbra</a>) shows much of the discrepancy to be present along the coast. This is not surprising, coastlines on OSM were originally derived automatically, and even when refined by hand are unlikely to accord with Mean High Water (MHW). Certainly, for my purposes, it is merely important that the OSM coastlines do not stray above MHW.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23357540202/in/photostream/" style="margin-left: auto; margin-right: auto;" title="NI Townlands, all boundaries within 5 m of OSNI"><img alt="NI Townlands, all boundaries within 5 m of OSNI" height="354" src="https://farm1.staticflickr.com/679/23357540202_01c4be999f.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">OSM townland Boundaries within 5m of OSNI data</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The analysis described so far focusses on positional accuracy. Looking at areas highlights a range of other accuracy issues.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22840038813/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="townlands_ni_cf9"><img alt="townlands_ni_cf9" height="354" src="https://farm6.staticflickr.com/5658/22840038813_8993ddb144.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Area comparison. Townlands are coloured according to absolute variance of ratio of areas from 1. <br />
The redder they are the further the ratio is from 1.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Area discrepancies of over, say 5%, may be the result of any of the following:<br />
<ul>
<li>Boundary discrepancy (such as coastlines). Mainly caused by coastlines, or difficulty of delineating some boundary feature, such as the course of the Umbra river above) </li>
<li>Erroneous interpretation of the boundary on old maps causing selection of the wrong feature. This transfers land from one townland to another, therefore these should cluster. </li>
<li>Missing townlands. When a single townland has been created without noticing one or more others inside it (<a href="http://www.openstreetmap.org/#map=14/55.0704/-6.5098&layers=N">Town Parks townland</a> at Ballymoney is an example). </li>
<li>Different treatment of townlands bisected by a Civil Parish. See caption of first image above.
Incorrect tagging. </li>
<li>Higher level administrative units having tags appropriate to a townland. I've noted two cases of this one of which was <a href="http://www.openstreetmap.org/relation/5136591">Ballyphilip </a>CP on the Ards peninsula in County Down. </li>
<li>Islands. Some offshore islands appear to be missing from the OSNI data (see The Skerries N of <a href="http://www.openstreetmap.org/#map=14/55.2215/-6.6145">Portrush</a>) </li>
</ul>
We've already caught a few examples in each of these classes through this analysis, and no doubt will find a few more. I have not yet investigated the very apparent discrepancy along the borders.<br />
<br />
To conclude, townland boundaries show exactly the kind of positional accuracy we expected (or perhaps hoped). Perhaps 1% of the total data (90-100 townlands from about 9000) may need some form of correction. I'm biased, but this seems pretty good, for a project principally relying on rectified photo-reduced maps from 1939! It's also worth remembering, that unlike road comparisons, there is no widely available sensor data (ie GPS tracks/point) to help boundary alignments.<br />
<br />
When time permits I'll extend this to include OSI Open Data too. A big thanks to both organisations for releasing their Open Data. OSNI staff have been contributors to OSM for a while: they host <a href="https://twitter.com/osni_maps/status/631832070952779778">Missing Maps lunchtime sessions</a> in their offices.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com3tag:blogger.com,1999:blog-2082422783692631078.post-22392258932018204622015-11-15T15:16:00.004+00:002015-11-15T15:16:56.105+00:00Urban Areas 4 : Derivation from OpenStreetMap using road densityAnother variation on the theme from <a href="http://sk53-osm.blogspot.com/2015/11/urban-areas-3-derivation-from-osm-using.html">the last post</a>: this time looking for some measure of road density.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481824/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="Butler Co, PA: derived Urban Areas"><img alt="Butler Co, PA: derived Urban Areas" height="354" src="https://farm6.staticflickr.com/5705/22413481824_f379de3860.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of Urban Areas derived using "block method" and gridded road density.<br />
Only grid squares with over 500 m of road included.<br />
The area shown <a href="http://www.openstreetmap.org/#map=13/40.8569/-79.9092">is around</a> Butler, Butler Co, PA</td><td class="tr-caption" style="text-align: center;"><br /></td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The easiest way is to sum road lengths in individual cells. The cells have to be quite small (say 250 metre square) to achieve the resolution desired. I've excluded link roads and motorway & trunk highway classed in this calculation.<br />
<br />
<br />
<a name='more'></a>Again I've used Butler County data from OpenSteetMap. I created a 250 m square grid in the (Google) Spherical Mercator projection, and simply summed up the road lengths in each grid square (roads were coerced to a geography type in PostGIS). Visual inspection of binned data suggested a value of between 400 and 500 metres/square gave reasonably discrete areas, and reduced artefacts along major roads. I show the result obtained above compared with the "block method" described in the previous post.<br />
<br />
An alternative way is to apply a finer grid of lines not polygons and split these grid lines each time they are intersected by a road. I've also done this for Butler County, although PostGIS doesn't provide any simple way to split a single line into multiple parts based on intersections.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481794/in/datetaken/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_density2"><img alt="butler_co_urban_density2" height="640" src="https://farm1.staticflickr.com/633/22413481794_bb0595bd0c_z.jpg" width="453" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Urban Area derivation, with results of grid intersection methods superimposed<br />
on gridded road density and block method results<br />
Area around Butler, Butler Co, PA</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
I used lines 100 m apart and in this case I filtered out any segments longer than 200 m. A second step is then to further restrict the selections to lines which intersect each other: this has the primary effect of removing dual carriageway artefacts. The former are shown as green lines, and the latter as red lines on the map above. I did not exclude motorway & trunk in this analysis, resulting in some artefacts around junctions.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22413481724/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_density1"><img alt="butler_co_urban_density1" height="500" src="https://farm1.staticflickr.com/737/22413481724_d19b0e4134.jpg" width="354" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">As above, Butler County as a whole.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
In general this produces similar results to other methods: but can miss areas where the roads are precisely aligned with the grid (a not uncommon phenomenon in the USA), suggesting that in practice grids should be rotated slightly from the cardinal directions.<br />
<br />
The two road density methods described here produce somewhat smaller polygons than the block method, but seem to be better than the latter for picking up smaller residential areas which tend to get eliminated in the latter method. Both methods may suffer for larger areas as the size of such closely spaced grids is likely to have adverse effects on performance.<br />
<br />
I think this concludes discussion of urban area derivation techniques applicable for complete road networks with unclear or inaccurate attribution, such as US Tiger data.<br />
<h4>
Appendix: Cutting a line at each intersection in PostGIS</h4>
One is so used to the power and flexibility of modern FOSS GIS tools, notably those from OSGeo, that it comes as a shock when what seems like a common operation is not supported directly. The OSM desktop editor JOSM, for instance provides a tool to split a way into multiple fragments.<br />
<br />
I'd hoped to split each x & y grid line at each point when it was crossed by a road. Filtering for only small segments would identify when roads were more frequent. The problems associated with doing this in PostGIS are reasonably well described here on <a href="http://gis.stackexchange.com/questions/115973/split-lines-at-intersection-points">GIS StackOverflow</a>. The second answer looks interesting but provides no guidance, and <span style="font-family: "Courier New",Courier,monospace;"><a href="http://postgis.net/docs/ST_Node.html">ST_NODE</a></span> must remain a mystery as there is no clear explanation of the magic it works, other than it replaces another bit of magic using <span style="font-family: "Courier New",Courier,monospace;">ST_UNION</span>. <br />
<br />
So I did something quick & dirty. I found all the intersection points between the grid lines & the road nework, taking care to retain the grid line identifiers. I then constructed lines from any intersection point to points either further east or north (using << and <<| operators) located <b>on the same line</b> & selected the shortest one. I then filtered to get those shorter than 200 m. In practice if I had known I was going to use 200 m as my filter I could have thrown out all the longer lines at that stage.<br />
<br />
<br />SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0tag:blogger.com,1999:blog-2082422783692631078.post-47954719781623401102015-11-14T15:17:00.001+00:002015-11-14T15:17:41.521+00:00Urban Areas 3 : derivation from OSM using residential blocks<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AFromCoL_8234828991.jpg" style="margin-left: auto; margin-right: auto;" title="By Zack Weinberg (Flickr Uploaded by crazypaco) [CC BY-SA 2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons"><img alt="FromCoL 8234828991" src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/FromCoL_8234828991.jpg/512px-FromCoL_8234828991.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">View S from the <a href="https://en.wikipedia.org/wiki/Cathedral_of_Learning">Cathedral of Learning</a> in <a href="https://en.wikipedia.org/wiki/Oakland_%28Pittsburgh%29">Oakland, Pittsburgh</a>, <br />showing some urban areas used as tests in this post.<br /><div style="text-align: justify;">
<span style="font-size: xx-small;">The incised valley of the Monogahela in the background contained railways and steel works. The plateau beyond has residential suburbs of Pittsburgh. To the left foreground are the woods and ravines of <a href="https://en.wikipedia.org/wiki/Schenley_Park">Schenley Park</a>, with a residential area beyond. Source: Zack Weinberg via Wikimedia Commons CC-BY-SA</span></div>
</td></tr>
</tbody></table>
<br />
One of the obvious features of the highway network for the USA on OpenStreetMap is that road density is much higher in built-up areas. I started looking at how to measure this, when I recalled a method for <a href="https://github.com/lucasmation/osm_cnefe_import">identifying city blocks</a> introduced to me by a Brazilian user of OpenStreetMap data.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22574566097/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="butler_co_urban_blocks"><img alt="butler_co_urban_blocks" height="640" src="https://farm6.staticflickr.com/5742/22574566097_c13845bc0f_z.jpg" width="453" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Residential Areas for Butler Co, Pennsylvania, identified with the block method<br />
from OpenStreetMap data. Orange line outlines Butler County.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
My idea was simple, a greater road density implies smaller areas for the polygons enclosed by a set of roads. By choosing some maximum polygon size, one should be able to pick out urban areas. <br />
<br />
The method itself is also really quite simple:<br />
<ul>
<li>Take the main road network for some area and make a union of it (which will be a MULTILINESTRING).</li>
<li>Polygonize this data, and decompose to individual polygons.</li>
</ul>
In Lucas' implementation the first step is done by municipal areas. I wanted to try the approach for a whole state without using administrative area data. I therefore once again turned to my trusty standby of using a gridded method.<br />
<br />
<a name='more'></a>First off I did this for Butler County, to test the process and iron out teething problems with the SQL (see above). The main changes I made are:<br />
<ul>
<li> to perform the initial creation of blocks using a 3 minute grid. In order to avoid losing blocks at the edges of each grid square, the grid square as a linestring needs to be included in the block generation. I achieved this by:</li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">SELECT grid_guid, geom <br /> FROM pa_roads_gridded<br /> UNION ALL <br />SELECT grid_guid, st_exteriorring(geom) <br /> FROM pa_3minute_grid</span></blockquote>
<ul>
<li>In the decomposition phase the area of each polygon is calculated, and only polygons with an area of under 100000 included (units in Spherical Mercator, actual areas are about half the size in square metres).</li>
<li>All blocks within a grid square are merged if they touch to form a number of larger polygons. (This uses the same method as before for identifying each distinct graph).</li>
<li>Finally the larger polygons which touch are merged across grid squares.</li>
</ul>
I then applied it to the whole of Pennsylvania. Initially I was too enthusiastic and tried to do the polygon merging in a single step, but soon returned to the gridded approach when I saw how slow this was in iterating to completion.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/23017578711/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh Urban Areas :derived from OSM using the block method"><img alt="Pittsburgh Urban Areas :derived from OSM using the block method" height="354" src="https://farm1.staticflickr.com/565/23017578711_36e981289e.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Urban Areas for the Pittsburgh region, derived using the 'block' method described here.<br />
Each polygon is coloured randomly to help show the boundaries.<br />
Interstates & other dual-carriageways create significant artefacts.</td></tr>
</tbody></table>
<br />
Looking at Pennsylvania as whole, and the Pittsburgh conurbation in particular, reveals quite reasonable identification of urban areas with <b>one big</b> problem. The problem is freeways, specifically interstates, but in practice it applies to any dual carriageway. These not only produce long linear extensions of the main urban areas, but actually cause areas which are not contiguous to fuse.<br />
<br />
There are a number of approaches to correcting these artefacts:<br />
<ul>
<li>Simply exclude <span style="font-family: "courier new" , "courier" , monospace;">highway=motorway</span> in the initial step. This would probably work OK for Pennsylvania, but would not, for instance, work in the UK where many <span style="font-family: "courier new" , "courier" , monospace;">highway=trunk</span> roads are dual carriageways.</li>
<li>Introduce some compactness measure for each initial polygon (e.g., ratio of perimeter to area, or ratio of area to area of an enclosing circle) , and exclude polygons over a certain threshold. This is probably the easiest to do in the short term.</li>
<li>Generalise dual carriageways to a single line feature before creating blocks. This is complex, the only solution I know of specifically for OSM data is the <a href="http://gis.stackexchange.com/questions/104359/how-to-get-a-low-level-road-network-from-openstreetmap-data?lq=1">VB6 one written</a> for <a href="https://vimeo.com/73118945">CityGuide</a>. However, this is a general problem (<a href="https://en.wikipedia.org/wiki/Topological_skeleton">skeletonization</a>) whose solution would help in a whole range of areas: for instance, better cartography for small scales; 1-dimension maps; automated address interpolation. </li>
</ul>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22995995762/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh Urban Areas: block method"><img alt="Pittsburgh Urban Areas: block method" height="354" src="https://farm1.staticflickr.com/757/22995995762_9c619d5ae9.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Urban Areas for the Pittsburgh region, derived using the modified 'block' method described above.<br />
Each polygon is coloured randomly to help show the boundaries.<br />
Compare with image above.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The second was easy to test: I excluded polygons with an <a href="http://mathworld.wolfram.com/IsoperimetricQuotient.html">isoperimetic ratio</a> of less than 0.3 and this seems to have eliminated the artefacts due to dual carriageways. As with some of my other grid-based techniques there are still some artefacts based on using a grid.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22995995842/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pennsyvlania Urban Areas : block method"><img alt="Pennsyvlania Urban Areas : block method" height="354" src="https://farm1.staticflickr.com/590/22995995842_9029e9496f.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Pennsylvania Urban Areas using modified block method.<br />
Grid artefacts are very clear at this scale.<br />
The vignetting of OSM tiles is not deliberate, but due to vagaries of the OpenLayers plugin for QGIS</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
As with all the methods described in this series of posts, this technique could do with refinement if used in anger. However, it does look to be generating relatively usable urban area polygons, albeit with some false positives, for US road data from OpenStreetMap.<br />
<br />
Some fairly simple improvements to the polygons — for instance clipping to ensure they don't overlap existing landuse type polygons — could make them suitable for reincorporation within OSM as residential areas. Two utilities may be available, the <a href="http://wiki.openstreetmap.org/wiki/Snapshot_Server">Snapshot Server</a> and <a href="http://wiki.openstreetmap.org/wiki/OSMLY">OSMLY</a>, which would allow such data to be used on a case-by-case basis when suitable by individual mappers.<br />
<br />
Another application would be to more easily reduce the number of highway=residential roads which are candidates for retagging. SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com1tag:blogger.com,1999:blog-2082422783692631078.post-17791328496716887152015-11-12T13:58:00.001+00:002015-11-13T10:37:14.794+00:00Urban Areas 2 : Derivation from OpenStreetMap using Residential Roads<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFm0s0av_tNrwK_hHqHyj7gciimYGgvcoei4XhrRDU6NSAJjN_toV6c08Qh6nlhM1ZZlMpas3LusPpE0Yff5_-2Ujgek6G0vbQy_cEytXqeHMmvPA2I8ktdOSUt1HQmZjkeVHusSUEC9WJ/s1600/IMG_4143-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFm0s0av_tNrwK_hHqHyj7gciimYGgvcoei4XhrRDU6NSAJjN_toV6c08Qh6nlhM1ZZlMpas3LusPpE0Yff5_-2Ujgek6G0vbQy_cEytXqeHMmvPA2I8ktdOSUt1HQmZjkeVHusSUEC9WJ/s320/IMG_4143-1.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Street corner, Retiro, Buenos Aires<br />
(<a href="http://www.openstreetmap.org/#map=19/-34.59284/-58.38377">Libertad/ Juncal</a>)<br />
CC-BY-SA, the author</td></tr>
</tbody></table>
Following on from my last post I have now been looking in more detail at how one might start using OpenStreetMap (OSM) to create a global dataset of Urban Areas. As OSM does not have any widely used notation for urban areas I have been looking at several ways in which other OSM data can be used to identify such areas prospectively. In this post I look at the use of residential roads (and I'm not the first <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_El_uso_de_OSM_para_la_identificaci%C3%B3n_de_Nucleos_Poblados">to do so</a>). Later posts will look at other techniques.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22438725688/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="ar_ba_urban2"><img alt="ar_ba_urban2" height="253" src="https://farm6.staticflickr.com/5712/22438725688_b97f59b27d.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Buenos Aires and hinterland, showing comparison between urban polygons<br />
derived from OSM (green) and the Natural Earth data (light brown).</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
<br />
I have chosen the following places as suitable test areas for these investigations:<br />
<ul>
<li><a href="https://en.wikipedia.org/wiki/East_Midlands"><b>East Midlands</b></a> of England. Not only my home turf, but also a well-mapped area with extensive use of landuse tags, and in excess of 99% of all residential roads. In addition Ordnance Survey Meridian 2 Open Data contains a layer corresponding to urban areas which provides an excellent control for checking results from this area.</li>
<li><a href="https://en.wikipedia.org/wiki/Pakistan"><b>Pakistan</b></a>. Not only one of the most populous countries in the world, but one of the least well mapped in OpenStreetMap. Pakistan is a likely candidate for cities which are barely mapped. I would also expect other very populous Asian countries (notably China, India and Bangladesh) which are poorly mapped to be similar to Pakistan.</li>
<li><a href="https://en.wikipedia.org/wiki/Nigeria"><b>Nigeria</b></a>. Similar criteria to Pakistan: the most populous country in Africa. The .pbf file for Nigeria is approximately 50% larger than that for Pakistan, but both are smaller than that for <a href="https://en.wikipedia.org/wiki/Lesotho">Lesotho </a>with a population of 2 million compared to 180 million (Nigeria) and 200 million (Pakistan).</li>
<li><a href="https://en.wikipedia.org/wiki/Ivory_Coast"><b>Côte d'Ivoire</b></a>. Close to Nigeria, but a place which I know has an active OSM community. Quite a number of mapping activities. (Note to Geofabrik, it's not called the Ivory Coast any more).</li>
<li><a href="https://en.wikipedia.org/wiki/Argentina"><b>Argentina</b></a>. Latin American cities are often laid out in a grid, nowhere more so than in Argentina. The prevalence of the grid system, and my believe that the urban road system is largely complete were reasons for choosing this as a Latin American example. My own experience of travelling in Argentina after SotM-14 suggests that, for the most part, urban road systems are mapped. One known gap, the newer western suburbs of Ushuaia has recently been rectified by the kind provision of aerial imagery from the Argentine National mapping agency.</li>
<li><a href="https://en.wikipedia.org/wiki/Pennsylvania"><b>Pennsylvania</b></a>. It was essential to include some US data because of the <a href="http://wiki.openstreetmap.org/wiki/TIGER">TIGER </a>import problem: all rural roads being tagged residential. Since I spent part of my childhood in Pennsylvania it is also a place I know and which I have edited (sporadically) to improve the rural road network.</li>
</ul>
Briefly I expected the following: good urban areas for the East Midlands and Argentina (i.e., better than <a href="https://en.wikipedia.org/wiki/Natural_Earth">Natural Earth</a> (NE)); middling to poor for the three developing nations (gaps relative to NE, but in some cases better precision); hopeless for Pennsylvania.<br />
<br />
<a name='more'></a>The basic process is very simple:<br />
<ol>
<li>Extract residential roads</li>
<li>Merge all roads which link together into a single multiline segment</li>
<li>Buffer each group of merged roads (I use 100 metres)</li>
<li>Buffer again by a larger amount & then again by the same amount but negated: this smooths the outline and fills any residual holes.</li>
</ol>
In practice this needs to be slightly more complex to improve performance. To this end I add the following techniques:<br />
<ul>
<li> Roads are clipped into grid squares (typically 10 km, 3 minutes or 7.5 minutes).</li>
<li>Merging is first performed within the grid</li>
<li>Some tidying up of this data is then done, notably buffering and re-clipping to the grid</li>
<li>A second round of merging is performed ignoring the grid</li>
</ul>
I'm using the postgres routine I described a couple of years ago to identify the independent graphs of roads in each grid, and in the second step. <strike>It still lacks all the performance optimisation stuff I meant to do back then!</strike> (After much messing about it turned out that it only needed indexes on the work table).<br />
<br />
<h3>
East Midlands </h3>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22479309648/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Comparison of 2 Urban Area sourced for East Midlands"><img alt="Comparison of 2 Urban Area sourced for East Midlands" height="640" src="https://farm6.staticflickr.com/5753/22479309648_a8634eae10_z.jpg" width="567" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of Urban Areas for English East Midlands</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
As expected, there is a high degree of concordance between areas identified as urban from the OSM residential road data, and the available Open Data from the Ordnance Survey. Areas not identified as urban from the OSM data, but so identified as such in the Meridian data are shown in cyan in the map above. For the most part the differences fall into two classes:<br />
<ol>
<li>Commercial, retail & industrial areas in towns and cities which are not identified through merely looking at residential roads in OSM. This initial aspect of the data is of course wholly predictable.</li>
<li>Small towns and villages in the countryside. Firstly, the goal is not to find these in these experiments. Various refinements (such as adding isolated island residential roads back at various steps in the process) have not been made (see further below). Secondly many villages will be aligned along road classes other than residential and would not be found anyway. Most of these areas not identified are under 60 ha in size.</li>
</ol>
More surprising is the very small number of false positives in the data derived from OSM. These are all erroneous classifications of rural roads as residential ones (sometimes because the same road starts in a town and runs into the countryside). For Britain, at least, this offers a useful visual QA tool for OSM data.<br />
<h3>
</h3>
<h3>
Argentina</h3>
I had also chosen Argentina as an area to validate the concept. But this was also influenced by the particular problems with Natural Earth data in this region. The huge <a href="https://en.wikipedia.org/wiki/Megalopolis_%28city_type%29">megalopolis </a>of <a href="https://en.wikipedia.org/wiki/Buenos_Aires">Buenos Aires </a>was also useful for making sure the chosen algorithms of this technique will work with the largest cities in the world.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJbABFqAT6wp61n6D-hCzpbr3jL8UvvO72ATcAa7FJ-JfecwrnQr3pPkpTY156LJrPd8QYms-TxTUdcJ8MUsxQNPIwS_wtlIDsTEtEuiAyCoemR33IJdAQoyxIfggsQntMpVLhOYxKoBUb/s1600/ar_chubut_urban.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJbABFqAT6wp61n6D-hCzpbr3jL8UvvO72ATcAa7FJ-JfecwrnQr3pPkpTY156LJrPd8QYms-TxTUdcJ8MUsxQNPIwS_wtlIDsTEtEuiAyCoemR33IJdAQoyxIfggsQntMpVLhOYxKoBUb/s320/ar_chubut_urban.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Northern <a href="https://en.wikipedia.org/wiki/Santa_Cruz_Province,_Argentina">Santa Cruz</a> & S. <a href="https://en.wikipedia.org/wiki/Chubut_Province">Chubut </a>provinces, <a href="https://en.wikipedia.org/wiki/Patagonia">Patagonia</a><br />
comparison of NE & derived OSM data. </td></tr>
</tbody></table>
Once again there is a highly satisfactory match with Natural Earth data around Buenos Aires. In <a href="https://en.wikipedia.org/wiki/Patagonia">Patagonia</a>, not only are many cities which were missed by Natural Earth identified, but there do not seem to be any obvious false positives. Indeed with some light tidying-up (clipping to coastlines, filling small holes, some smoothing) this could make a viable replacement for the NE data here.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeQYSaijLxbkSKY2yuPlE9uMoiDVwdk8hFWrFsATxwIH7ygj9PQi7zwJSMfnltdD5hAsc92QorKM_l9TlXWHVDVvlQSue4NhaoUDDekH0VCL_n0hEcTna_64of3bIkK2cQZLy2szXGE0_Q/s1600/ar_nequen_urban.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeQYSaijLxbkSKY2yuPlE9uMoiDVwdk8hFWrFsATxwIH7ygj9PQi7zwJSMfnltdD5hAsc92QorKM_l9TlXWHVDVvlQSue4NhaoUDDekH0VCL_n0hEcTna_64of3bIkK2cQZLy2szXGE0_Q/s320/ar_nequen_urban.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Neuqu%C3%A9n">Neuquén </a>city (bottom centre) and province: More false positive Urban Areas from Natural Earth compared to OSM<br />
Sources: as before</td></tr>
</tbody></table>
<h4>
Addendum</h4>
It turns out that I'm not the first to use this approach in Argentina. After I'd published this post <br />
<ahref https:="" nicolas09f9="" twitter.com="">Nicolás Álvarez</ahref> drew my attention to <a href="http://wiki.openstreetmap.org/wiki/SotM_2014_session:_El_uso_de_OSM_para_la_identificaci%C3%B3n_de_Nucleos_Poblados">a talk</a>, which I had missed, at State of the Map 2014 in Buenos Aires. The authors are :<a href="https://twitter.com/vlasvlasvlas">Vladimiro Bellini</a>, Fernando Pino, Martin Moroni
of the Ministry of Energy. Another great example of how people are doing interesting things with OSM data all over the world.<br />
<br />
I haven't looked at the details yet, but from the abstract this looks very similar to my approach with an important refinement, restricting the length of roads included to those under 2 km. Slides of this talk are available on <a href="http://es.slideshare.net/VladimirBellini/sotm-2014-session-el-uso-de-osm-para-la-identificacin-de-nucleos-poblados">Slideshare</a>. <br />
<h3>
Pakistan</h3>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdi8w6PkHoTko6pFTGGGEshg7Sw-J8OwqigHF5MFYhl9Yc7hEq7BEyy4N_MZIbj0sC5GSvOGyQV0llRVecE1t5D9J0wqhZhBuaQrx_eNDZENJWM64jgnlTlrvNaZt4y4aRy9HF9RKzpAoP/s1600/pk_urban3.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdi8w6PkHoTko6pFTGGGEshg7Sw-J8OwqigHF5MFYhl9Yc7hEq7BEyy4N_MZIbj0sC5GSvOGyQV0llRVecE1t5D9J0wqhZhBuaQrx_eNDZENJWM64jgnlTlrvNaZt4y4aRy9HF9RKzpAoP/s320/pk_urban3.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/Punjab,_Pakistan">Punjab </a>area of Pakistan: comparison with Natural Earth.<br />
<a href="https://en.wikipedia.org/wiki/Lahore">Lahore </a>is bottom right & <a href="https://en.wikipedia.org/wiki/Rawalpindi">Rawalpindi </a>top left.</td></tr>
</tbody></table>
<br />
A first cursory look at the results for Pakistan compared with Natural Earth data suggests that most major urban areas are being detected. Looking at a larger scale reveals many mid-size cities being ignored. At even larger scales the inaccuracy of the Natural Earth data itself becomes more apparent.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPhtHUl6Caeeqlh6RVKYvTfqineE2Gty-QqPGfRMGRhgprhqRBG-_X8hXhwGcdU4der0JHyd376_unWlm0Yzx9BPQnJ3fxgd_KddEntC-769FfQ12A9afhSi-jhzOy-OFVTige5eATX6Py/s1600/pk_urban2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPhtHUl6Caeeqlh6RVKYvTfqineE2Gty-QqPGfRMGRhgprhqRBG-_X8hXhwGcdU4der0JHyd376_unWlm0Yzx9BPQnJ3fxgd_KddEntC-769FfQ12A9afhSi-jhzOy-OFVTige5eATX6Py/s320/pk_urban2.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Slightly more detail with Lahore (bottom centre) & Sialkot (centre).<br />
<span style="font-size: xx-small;">Many rural as well as urban roads have the residential tag in the latter area,</span><br />
<span style="font-size: xx-small;">Note the reticulated appearance of the polygon (see below, Pennsylvania)</span></td></tr>
</tbody></table>
It seems that a significant problem exists with my algorithm for Pakistan. Even large cities may have only a handful of roads mapped as highway=residential. These either form isolated islands in the first phase or are merged into a single island after the first phase. At present I make no attempt to add back these polygons with no adjacent ones. The sensible place to do it is after the second phase of merging. To do it earlier would be to introduce too many artefacts from inappropriately tagged roads or isolated residential districts in rural areas. I have not done this for this post, but show below the difference between 1st phase (dark blue) & 2nd phase (light blue) urban polygons above & below.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLa6SnoYjHoHY7q0k0mpc-iEshxJ4IEHryyDU07F3Ekoq5bl40bv010nKRB-zHf0VexTKzk55ZFEWRbsREberMa-dMCNGueOZ8l9qaaiLtXlsLdw4UCIdWU-kPIh5mquyupdwq_6cdg7en/s1600/pk_sialkot2_urban.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLa6SnoYjHoHY7q0k0mpc-iEshxJ4IEHryyDU07F3Ekoq5bl40bv010nKRB-zHf0VexTKzk55ZFEWRbsREberMa-dMCNGueOZ8l9qaaiLtXlsLdw4UCIdWU-kPIh5mquyupdwq_6cdg7en/s320/pk_sialkot2_urban.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Higher zoom of area West of Sialkot (right).<br />
<span style="font-size: xx-small;">Many urban areas have either a tiny number of residential roads or none at all.<br />Source: Natural Earth & derived OSM polygons<br />Background Imagery: Landsat via Bing Maps</span></td></tr>
</tbody></table>
Lahore is is a megalopolis, but it has remarkably few residential streets mapped. A major factor is that in the older parts of the city, most thoroughfares in residential areas are narrow, and lined by multi-storey buildings. This makes them more or less impossible to pick out on available aerial imagery, and perhaps on any imagery likely to be available to OSM in the near future. However, my choice of buffer size to build residential roads works reasonably well even when the grid is partially mapped, as in this case.<br />
<br />
Notwithstanding this there are still many wider roads which can be mapped. Once again I have spent a little time whilst writing this post adding a few in Lahore.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AA_Busy_Street_in_Sialkot.jpg" style="margin-left: auto; margin-right: auto;" title="By Yaminjanjua at en.wikipedia (Transferred from en.wikipedia) [Public domain], via Wikimedia Commons"><img alt="A Busy Street in Sialkot" src="https://upload.wikimedia.org/wikipedia/commons/2/26/A_Busy_Street_in_Sialkot.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A busy street, Sialkot 2008<br />
CC-BY-SA via Wikimedia</td></tr>
</tbody></table>
<br />
Elsewhere, a local mapper in <a href="https://en.wikipedia.org/wiki/Sialkot">Sialkot </a>has been <a href="http://www.openstreetmap.org/#map=17/32.49392/74.53604">adding this type of detail,</a> but using the tag <a href="http://wiki.openstreetmap.org/wiki/Tag:service%3Dalley"><span style="font-family: "courier new" , "courier" , monospace;">highway=service, service=alley</span></a>. This seems an entirely reasonable choice of tags: if it is a local convention then taking account of this by adding such ways to the choice of residential roads is easy to do. (I would have to look at the use of this tag combination globally to see if there were any problems in doing this by default).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3M16gjSLyVYLDkEvzZxVgQ1-dFWCJXvEIYA9y9RYDIZukFFy4ECG8jO4-xkzlXnqPX11W-mkbh0F19UD7xwxlBhkk0qZ05N99YdkMWifEKqDYrpiLhuB8RKFeePPa6R5B6yDvDj6ntIkO/s1600/pk_res_ua.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3M16gjSLyVYLDkEvzZxVgQ1-dFWCJXvEIYA9y9RYDIZukFFy4ECG8jO4-xkzlXnqPX11W-mkbh0F19UD7xwxlBhkk0qZ05N99YdkMWifEKqDYrpiLhuB8RKFeePPa6R5B6yDvDj6ntIkO/s320/pk_res_ua.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Pakistan showing both Urban & OSM Residential areas.<br />
Note the density in <a href="https://en.wikipedia.org/wiki/Sindh">Sindh</a>, the result of a <a href="http://wiki.openstreetmap.org/wiki/Humanitarian_OSM_Team/Pakistan_Tharparkar_Drought">HOT activation</a></td></tr>
</tbody></table>
It was no surprise that OSM data for Pakistan are incomplete in many areas. I have drawn most of my examples from a particularly densely populated parts of the Punjab. In the process of examining the area it is clear that much can be achieved in a relatively short time by adding more residential areas and prominent residential roads. We should bear in mind that the <a href="http://wiki.openstreetmap.org/wiki/2010_07_Pakistan_Floods">dreadful flooding events of 2010</a> are unlikely to be isolated events. At the time map data available to relief organisations was poor, and the sheer extent of the area covered was far too large for a <a href="http://www.mapaction.org/disasterresponse/depldetail/196.html">responsive mapping campaign</a>.<br />
<br />
OSM data for Pakistan is still poor over 5 years on, we never managed to achieve the same leverage as Google Map Maker with the diaspora of people with Pakistani-heritage. But we can try again.<br />
<h3>
Nigeria</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3ARound_about_in_Ibadan-1%2C_by_Adebisi_Adewoyin.jpg" style="margin-left: auto; margin-right: auto;" title="By Adebisi Adewoyin (World66) [CC BY-SA 1.0 (http://creativecommons.org/licenses/by-sa/1.0)], via Wikimedia Commons"><img alt="Round about in Ibadan-1, by Adebisi Adewoyin" src="https://upload.wikimedia.org/wikipedia/commons/b/bf/Round_about_in_Ibadan-1%2C_by_Adebisi_Adewoyin.jpg" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Roundabout <a href="https://en.wikipedia.org/wiki/Ibadan">Ibadan</a><br />
Whilst looking for CC-BY-SA images for urban areas of Nigeria I noticed a few shots of roundabouts with distinctive sculptures in the middle. These seem <a href="http://thenationonlineng.net/imos-magnificent-roundabouts-city-gates/">quite common</a> across the country.<br />
Source: Adebisi Adewoyin via Wikimedia Commons CC-BY-SA</td></tr>
</tbody></table>
In many ways the situation in Nigeria is similar to Pakistan: at an overall level the results look reasonable, a closer look reveals that large cities are getting missed. This is most noticeable in the Southern part of the country.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22323292523/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Urban Area comparison, SW Nigera"><img alt="Urban Area comparison, SW Nigera" height="354" src="https://farm1.staticflickr.com/728/22323292523_c953269028.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of Natural Earth & derived OSM Urban Areas for SW Nigeria</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
For the area shown above at least one factor is the absence of high resolution imagery. However, some significant cities were missing in any other form than an imported GNIS node ( e.g., <a href="https://en.wikipedia.org/wiki/Owo,_Nigeria">Owu</a>). Looking at the country as a whole, it appears the North is much better mapped, with a more extensive road system and lots of minor settlements mapped as residential areas.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22323292563/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Nigeria: urban & residential areas"><img alt="Nigeria: urban & residential areas" height="256" src="https://farm6.staticflickr.com/5817/22323292563_3c605cb2a9.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Urban (NE & derived OSM) and residential areas in Nigeria</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
I have no idea why this should be, but clearly one would expect as much or more detail in other parts of the country based on available demographic data (see below). I am aware that <a href="https://en.wikipedia.org/wiki/Kano">Kano </a>seems to have the most active mappers, that there have been <a href="http://wiki.openstreetmap.org/wiki/WikiProject_Nigeria#North_Nigeria_crisis_-_HOT_Monitoring_-_mapping_goals">HOT initiatives</a> in NE Nigeria, and that eHealth Africa have been doing some mapping of residential areas (see this recent <a href="http://www.openstreetmap.org/user/olub/diary/36265">blog post</a>): it is probably the latter activity which has had the biggest effect. I dont know quite how long this has been running, but it does demonstrate how much data can be added.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5W0e69hzoUuZK_hRCK6Wgw0tDbFXvZuyZ9E1O61G6Uyzupuln8qijt30O_Uk1C1jp4pu1CkEIj3IQzecMk2_o7lCF68Uf3dG2-C3l5xCX3KfNVOP-YNaTGwH49W_Sr_8Y4ZqsY43ftVFk/s1600/ngadens.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="533" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5W0e69hzoUuZK_hRCK6Wgw0tDbFXvZuyZ9E1O61G6Uyzupuln8qijt30O_Uk1C1jp4pu1CkEIj3IQzecMk2_o7lCF68Uf3dG2-C3l5xCX3KfNVOP-YNaTGwH49W_Sr_8Y4ZqsY43ftVFk/s400/ngadens.jpg" width="412" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Nigeria Population Density, 2000<br />
Compare with density of residential mapping on OSM<br />
Source: see map CC-BY-SA</td></tr>
</tbody></table>
<h3>
Côte d'Ivoire</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://commons.wikimedia.org/wiki/File%3AAbidjanpyramid.JPG" title="By Zenman (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons"><img alt="Abidjanpyramid" src="https://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Abidjanpyramid.JPG/512px-Abidjanpyramid.JPG" width="512" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Central Abidjan, quartier Plateau (on <a href="http://www.openstreetmap.org/#map=17/5.31941/-4.01760">OSM</a>)<br />
Source: Zenman via Wikimedia Commons</td></tr>
</tbody></table>
I wanted to look at an area with some similarities to Nigeria, but with a better established mapping community to see if this makes for any significant difference. The obvious choice was Côte d'Ivoire. There always seems to be something interesting going on there: mapping parties, software training events, conferences. And it's all nicely documented on the <a href="http://www.openstreetmap.ci/">community website</a>. One of the things I particularly like is a <a href="http://umap.openstreetmap.fr/fr/map/pharma-carte_3999#12/5.3563/-4.0134">map of pharmacies in Abidjan</a> on umap: a reminder that maps help everywhere for little daily tasks or personal emergencies. Looking at Pascal Neis' maps Abidjan looks to be the place with the <a href="http://resultmaps.neis-one.org/oooc?zoom=8&lat=6.63751&lon=-4.65558&layers=B00TFFFFFT">most dedicated mappers</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtsH0zwYyC7cn2at_kTj-i1yOQOKnd3y1t-7ES7emhnxeUJaGFJV13Kr58ZjFDtEI0dsApSQZlwWvqxlBfhFo0dpnvZqKX9fAt8r_4alEoqOvpEpevVxyUlVO2wl8CbT0Gm7nQ31XlJGhr/s1600/ci_south_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtsH0zwYyC7cn2at_kTj-i1yOQOKnd3y1t-7ES7emhnxeUJaGFJV13Kr58ZjFDtEI0dsApSQZlwWvqxlBfhFo0dpnvZqKX9fAt8r_4alEoqOvpEpevVxyUlVO2wl8CbT0Gm7nQ31XlJGhr/s320/ci_south_ua.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of NE & derived OSM Urban Areas, Southern Côte d'Ivoire<br />
Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta)<br />
Imagery: Landsat via Bing Maps</td></tr>
</tbody></table>
Again on the small scale we have a pattern of significant settlements being captured from OSM and some smaller places being missed in a fairly haphazard way. We also get the impression that NE urban areas are frequently too large.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFRMUolmLaFl-EeC7sb_22uyN1HjkvF41dQHNajAjNZLP_Z8FIr4fCHEIRKxtrPot7fVFtvlqgSwmIUD4_ulSs2cz6CloIio2-AQtv-yoQnMqClKsX-VymHtTAgOZ5-h7UrXV5HNR8ekGm/s1600/ci_central_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFRMUolmLaFl-EeC7sb_22uyN1HjkvF41dQHNajAjNZLP_Z8FIr4fCHEIRKxtrPot7fVFtvlqgSwmIUD4_ulSs2cz6CloIio2-AQtv-yoQnMqClKsX-VymHtTAgOZ5-h7UrXV5HNR8ekGm/s320/ci_central_ua.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of NE & derived OSM Urban Areas, Southern Côte d'Ivoire<br />
Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta), OSM residential (blue)<br />
Imagery: Landsat via Bing Maps</td></tr>
</tbody></table>
This is confirmed by zooming in.<br />
<br />
Adding an extra layer from OSM, mapped residential landuse shows what is most different between Nigeria & the Côte d'Ivoire. Individual settlements of different sizes have been mapped in the latter as <span style="font-family: "courier new" , "courier" , monospace;">landuse=residential</span>, even if no-one has had time to add the individual roads. In Côte d'Ivoire it is detail of settlements that is missing not settlements themselves.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRMctisKhmWN4gWGQw1lIw9hZ2QHlDnq-MyStFtGyISMrB2syDJPPw6YalT0d-ZJkp9xnqP1Y9I_fHIjik22sJCvlQ-PYVJuWZSYjDBv6e3HfqamyHWHBncsllYjMIJApIuM4EUdC16T2n/s1600/ci_adidjan_ua.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRMctisKhmWN4gWGQw1lIw9hZ2QHlDnq-MyStFtGyISMrB2syDJPPw6YalT0d-ZJkp9xnqP1Y9I_fHIjik22sJCvlQ-PYVJuWZSYjDBv6e3HfqamyHWHBncsllYjMIJApIuM4EUdC16T2n/s320/ci_adidjan_ua.jpg" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Comparison of NE & derived OSM Urban Areas, Abidjan<br />
Legend: road network (OSM), NE urban (cyan), OSM derived urban (magenta)<br />
Imagery: Standard OSM layer</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
Looking at <a href="https://en.wikipedia.org/wiki/Abidjan">Abidjan </a>itself, we can see how well mapped it is, the precise correspondence of derived urban areas with landuse mapping, and, once again, that the NE polygon is too large. I do note that the derived urban area includes some large industrial landuse polygons. This suggests that <span style="font-family: "courier new" , "courier" , monospace;">highway=residential</span> has been used rather than <span style="font-family: "courier new" , "courier" , monospace;">highway=unclassified</span> in these areas.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
Unlike Nigeria & Pakistan (and the US, see below), I have not felt the need edit OSM for Côte d'Ivoire, either to add really obviously missing data, or to modify tagging. </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
Looking at these three countries together, the key point is that, unless the local OSM community is large, and mature, residential roads will not be adequate on their own to identify urban areas. In Britain, it was not until after we had official open data (Ordnance Survey) that many residential roads were added, and that was in 2010, 5 years into mapping the country. Without that external source we would probably still have significant towns & cities only partially mapped. Much of how the map Côte d'Ivoire looks now is very reminiscent of Great Britain before we had open data: great detail in places, but fairly scanty away from where most mappers lived. </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
Adding landuse is an excellent way to build up a picture of what needs to be mapped: in places like Côte d'Ivoire the combination of using both landuse & derived urban areas looks promising (more later).</div>
<h3>
Pennsylvania (and elsewhere in US)</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22832538622/in/dateposted/" style="margin-left: auto; margin-right: auto;" title="Pittsburgh area: Residential Roads on OSM"><img alt="Pittsburgh area: Residential Roads on OSM" height="354" src="https://farm6.staticflickr.com/5805/22832538622_34cb180708.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Pittsburgh area : Residential Highways & Landuse mapped on OpenStreetMap<br />
Source: (c) OpenStreetMap contributors</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
A quick look at road data for Pennsylvania shows that the network of residential roads reaches into the furthermost reaches of the state. The human eye can quickly discern the much greater density of the road network which represents truly urban locations. Finding a suitable automatic substitute will be the topic of a later post in this series.<br />
<br />
However, the reason for looking at the US was to look at the problem where residential roads have not been distinguished from other minor roads (i.e., the confusingly named "unclassified" highway tag in OSM). The first thing is just to take a look at the output from the algorithm described above.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22438725678/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="pittsburgh_ua2"><img alt="pittsburgh_ua2" height="354" src="https://farm1.staticflickr.com/616/22438725678_f628c65875.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The same area as above, with each contiguous group of residential roads given a random colour.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
Actually it's not quite as bad as I feared: rivers, railways and major roads all serve to break up the continuous network of residential roads. This effect is more pronounced in cities and large towns. The network of rural roads end up as very large continuous 'swiss cheese' polygons. In many cases these can be broken up by re-tagging only a small subset of roads from residential to unclassified (or service/track if applicable).<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22832538602/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="Kansas Residential Roads on OSM"><img alt="Kansas Residential Roads on OSM" height="256" src="https://farm6.staticflickr.com/5730/22832538602_b78d21dec6.jpg" width="500" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">All residential roads in Kansas from OSM<br />
The background is the 3 minute interval grid used in building a graph of the roads.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
I also had a quick look at Kansas (a typical mid-Western state with a gridded road system determined by township boundaries), and Oregon (mountains, a very sparsely populated interior, but lots of active mappers in the Portland & <a href="https://en.wikipedia.org/wiki/Willamette_Valley">Willamette Valley</a> areas).<br />
<br />
I'd half expected <a href="http://www.openstreetmap.org/user/ToeBee">ToeBee </a>to have done some tidying up of highway=residential in Kansas. He's certainly tidied up lots of other things, and he knows the rural roads well as a regular participant in the annual <a href="https://en.wikipedia.org/wiki/Biking_Across_Kansas">Biking Across Kansas</a> ride (many of his Mapillary pictures stem from these rides). The effect of a regular grid of roads, and no work to reclassify them from the original TIGER import is really obvious.<br />
<br />
<br />
Oregon is substantially better, The local mapping community has clearly worked to improve the Tiger data along the Willamette Valley extending south from Portland. In these cases the built-up areas of local towns and cities stand out clearly. Like Pittsburgh, Portland itself is represented by several polygons divided by rivers, railways etc. Away from the centres of population we return to the 'swiss cheese; polygons. At least in some parts of the state, many of these residential roads are nothing more than forest tracks, or old farm tracks (as in the part of the John Day Fossil Beds where the family of my great-grandfather's brother ranched until around 1975). Some are, at best, vestigial.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22869041515/in/dateposted-public/" style="margin-left: auto; margin-right: auto;" title="Urban areas derived from residential roads, Willamette Valley, OR"><img alt="Urban areas derived from residential roads, Willamette Valley, OR" height="640" src="https://farm1.staticflickr.com/681/22869041515_7fe8bf9375_z.jpg" width="453" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Willamette Valley, Oregon, showing urban areas derived from residential roads.<br />
Outlying areas can be seen to be uncorrected Tiger data, resulting in 'swiss cheese' polygons.</td></tr>
</tbody></table>
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
What this area of Oregon shows is that concerted local efforts to correct Tiger data can achieve decent results.<br />
<br />
Remember that just the process of reviewing Tiger data can lead to substantial improvements in alignment, road detail and other things. The real problem with Tiger data is its sheer abundance, which ends up being so off-putting (and boring) that rarely do mappers stick with it. Typical rural counties in Pennsylvania have 5000 or so ways tagged highway=residential, which is a hell of a lot to review, realign, check the surface type, correct other errors, etc. The sheer amount of data involved in imports often just overwhelms the capacity of local mappers to check and enhance the imported data. This factor should always be allowed for when planning imports: if the amount of data is beyond the capacity of the community to assimilate it then it often remains untouched. Even when the data is of good quality it will get dated quite quickly.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5sze6s60TMHsVPLaYcdvAgd4Jg4Ts4Uwrw73RD32JZdr-0s20sCI9dNEMWZYxBWs23n6BFY09vUrToojM8kydRpO9riRP0FTYNe_4j4NBD4nHMu__sA5bxGNxnPHuQAHcpm6b2GmmtLr6/s1600/imgus65_215c.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5sze6s60TMHsVPLaYcdvAgd4Jg4Ts4Uwrw73RD32JZdr-0s20sCI9dNEMWZYxBWs23n6BFY09vUrToojM8kydRpO9riRP0FTYNe_4j4NBD4nHMu__sA5bxGNxnPHuQAHcpm6b2GmmtLr6/s320/imgus65_215c.jpg" width="520" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A barn near Valencia, Butler Co, 16th July 1966</td></tr>
</tbody></table>
Whilst writing up this post, I've experimented with revising the classification of residential roads in Butler County, Pennsylvania. Butler is to the north of Pittsburgh, and even in the 1960s we had friends and neighbours who moved there, whilst continuing to work in the city. So although the county is still predominantly rural, there is a lot of dispersed residential settlement: either as small spur residential roads off main highways, or as non-contiguous lots along these roads. This type of settlement pattern definitely makes deciding whether residential or unclassified is the appropriate tag. One mapper has made a determined effort to map this type of landuse in a similar area just north of the Mason-Dixon Line.<br />
<br />
<div style="text-align: center;">
<iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="http://www.openstreetmap.org/export/embed.html?bbox=-77.86208152770996%2C39.7523372243077%2C-77.7662944793701%2C39.79092937807007&layer=mapnik" style="border: 1px solid black;" width="425"></iframe><br /></div>
<div style="text-align: center;">
<small><a href="http://www.openstreetmap.org/#map=14/39.7716/-77.8142">View Larger Map</a></small>
</div>
<br />
I focussed solely on changing tags, with a small amount of addition of a surface tag when this was really obvious. Equally, you can see, that I mainly looked at longer roads. Not only are they much easier to assess at lower zoom levels, but changing the tags of a few longer roads has a disproportionately useful effect on things like routing or my goal. I also steered clear of the larger towns which would have required more detailed examination of the aerial imagery.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghlBr0D23XwoUmQwLRXbLmTWj2y41lp4A_njKcSgeR4MCm33_2eiEN4uUl2mH_-5LfU0hRT4oOlEtWxFXlswu1UZ3vRnnsVUaAoHfPVXrXv1CPjGBFPru1vqBhjZ7-upSKRslEfd1Jwq1i/s1600/export%252818%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghlBr0D23XwoUmQwLRXbLmTWj2y41lp4A_njKcSgeR4MCm33_2eiEN4uUl2mH_-5LfU0hRT4oOlEtWxFXlswu1UZ3vRnnsVUaAoHfPVXrXv1CPjGBFPru1vqBhjZ7-upSKRslEfd1Jwq1i/s320/export%252818%2529.png" width="480" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Butler Co, PA: roads reassigned to unclassified.<br />
My edits whilst writing this blog.</td></tr>
</tbody></table>
This approach worked fine for my goal, and should also improve things like cycle routing. Many roads are still poorly aligned, but my overall feeling is that it is more productive to concentrate on one type of change.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a data-flickr-embed="true" href="https://www.flickr.com/photos/sk53_osm/22476112828/in/photostream/" style="margin-left: auto; margin-right: auto;" title="Butler County, Pennsylvania : Urban Areas derived from OSM"><img alt="Butler County, Pennsylvania : Urban Areas derived from OSM" height="640" src="https://farm6.staticflickr.com/5692/22476112828_ecd745ba66_z.jpg" width="453" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Derived polygons for Butler Co, PA. <br />
Pastel shades original polygons, grey edged with red, <br />
derived after reclassification of rural roads (as seen aove)</td></tr>
</tbody></table>
<br />
<script async="" charset="utf-8" src="//embedr.flickr.com/assets/client-code.js"></script>
The results of these changes compared to the PA dataset I started with are shown above.<br />
<br />
Obviously for the past several years most people making use of OSM data have not been impacted by the profusion of residential roads in rural areas (and indeed in BLM lands and National Forests). Is this because most applications are agnostic to such data in the US, or is OSM data just not used in such places? The only consumer I know which does place importance on such data is Richard Fairhurst's <a href="http://cycle.travel/">cycle.travel</a>, and he already makes use of landuse data to improve the quality of selected routes. It may be that highway=residential is so pervasive in the US that all data consumers have to work around this tagging, with the effect that there will be little incentive for regular mappers to change the tags.<br />
<h3>
Conclusions</h3>
This post has taken a rather meandering route.<br />
<ul>
<li><b>Firstly</b>, the general question: "Can useful results be obtained using a very naive approach to identify urban areas?" has been answered. It works well in areas with good mapping coverage, is a decent starting points for poorly mapped places, and only falls down when tagging practices are away from the norm.</li>
</ul>
<ul>
<li><b>Secondly</b>, the data produced can provide useful visualisations which can rapidly demonstrate areas where OSM lacks data, or existing data might be better tagged.<br /> </li>
<li><b>Thirdly</b>. As for so many things with OSM, often the simplest way to work towards the data set one wants is to add more data to OSM or improve data which already exists.</li>
<li><b>Fourthly. </b>This approach crucially depends on availability of high quality aerial or satellite imagery for at least basic urban road networks to be mapped. Areas where only landsat data are available can never be identified with this technique.</li>
</ul>
This is not the only approach I have considered. Next up will be using OSM landuse polygons directly.<br />
<br />
As a final note I'd like to express my particular appreciation of the work of local mappers in Pakistan, Nigeria & Cote d'Ivoire. Much of the detailed analysis and issues discussed above was based on that work.SK53-osmhttp://www.blogger.com/profile/05611497805471433828noreply@blogger.com0