Open geo data in The Netherlands (and beyond)

A brief introduction to the Dutch open geo data scene.

Provides answers to:
- why it is perceived as being crap
- why it is not crap
- where's the data
- how to get it
- how to make a map using TileMill / CartoDB

Go to Google Drive for working links: https://docs.google.com/presentation/d/1lgoJ4R6CGEcBKrfzs2dn5eGswsjoSpbqgntU8TuywI0/edit#slide=id.p

Transcript

1. Open Geo Data in The Netherlands (and beyond)
2. Bio and outlook @simeonnedkov Past: MSc Geomatics at TU Delft Current: PhD research at Vrije Universiteit on geo tech use in the city/people/government triad - redefine the smart city Today - Dutch/European open geo data and stores - View and get data - Display (in fancy ways) and host your own data - No D3 and OpenGeo Suite
3. ... everything you ever wanted to know about geographical information, standards and services?
4. 1. Why it sucks. 2. When it will get better. 3. What's in it for me.
5. 1. Out of sight, out of mind.. well, almost. 2. Gradually... 3. Easy access to pan-European data
6. It sucks. Verbose Unpolished and obscure docs SHP? EPSG: 28992?? GIS? WMS? WFS? Reference servers and implementations are mostly Java (and C++) Geoportals.. ugh! Lack of decent geo data sources and services Politics
7. It needs to get better And it is! Better mapping tech: OpenLayers 3, MapBox, CartoDB, Shapely, Fiona More tutorials by non-geo developers Favourable legislation: Wet Basisregistraties Publieke Dienstverlening op de Kaart (PDOK) Nationaal Georegister 2... INSPIRE
8. Why bother? Well established and powerful standards Extensive infrastructure Access to European open geo data Interoperability and ease of use Use it to make it better Encourage cities / open data initiatives to adopt standards You have no choice... :)
9. http://www.slideshare.net/jatorre/siglibre-english-17036935 Javier de la Torre (@jatorre) - Vizzuality, creators of CartoDB
10. 1. Demystify Dutch geo data stores and standards 2. Demystify some geo tools
11. Let's dive in SDIs and geoportals
12. Client Data Interact Spatial Data Infrastructures?
13. Client Data Discover View Download Spatial Data Infrastructures?
14. Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g machine readable geo services
15. Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g Geoportal
16. Client Data Discover View Download Spatial Data Infrastructures? w e b s i t e c a t a l o g Geoportal SDI
17. Geo data in theory Client Metadata - location URL - format - extent - accuracy - temporal info - coordinate system - owner - access conditions Data store I Data store III Data store II Data store IV Catalog Discover data Location of data URL Query for data Download Data - georeferenced - vector (shp) - raster (geotiff) View
18. Geo data in theory Client (Machine) searchable catalog NLR Universities Schools Governmental bodies Provinces Cities Research institutes Aerial images? NLR has them at this URL Geo extent json, svg .png, jpg
19. Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files
20. Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues
21. Geo data in practice Client NLR Universities Schools Provinces Research institutes Data store V Provinciaal GeoRegister Governmental bodies Cities Most stores CKAN flat files Politics and organisational issues The tech works!
22. Forget it!
23. But there is hope!
24. http://geo.admin.ch
25. nationaalgeoregister.nl Needs to grow... Back to reality pdok.nl national data sets robust up-to-date great use it! provinciaalgeoregister.nl provinces, no services KNMI / NLR
26. Interesting datasets BAG Basisregistraties Adressen en Gebouwen
27. Census data 2009, 2010, 2011
28. AHN Actueel Hoogtebestand Nederland - 25 m resolution
29. National cycling routes
30. More data Local announcements (live!) Topographic maps Rail/Roads/Canals/Parks CBS data Aerial and satellite images
31. PDOK.nl https://www.pdok.nl/en/products/pdok-services/overview-urls
32. Even more data INSPIRE "... an EU initiative to establish an infrastructure for spatial information in Europe ..." 1. Data from all EU member states 2. More Dutch data
33. Accessed and exchanged through geo services
34. The geo services
35. Vector vs. raster Space filling GeoTIFF Objects GeoJSON, SHP
36. Geo services View Web Map Service (WMS) Raster: images Web Map Tile Service (WMTS) Tiles Download Web Feature Service (WFS) Vector: geometry and attributes Web Coverage Service (WCS) Raw raster data
37. http://geo.serv.er/wms Server View vector and raster Download vector From data to services Tile Cache geometry sttyling http://geo.serv.er/wfs - (Spatial) filters - (Spatial) operations - Coordinate transformations vector raster values
38. AHN http://geodata.nationaalgeoregister.nl/ahn25m/wms Anatomy of WMS
39. http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetCapabilities& service=WMS& version=1.1.3 Interacting with services Request: HTTP GET / POST Response: XML, .png, .jpg, .GML, .JSON, .svg
40. Client GetCapabilities Server Capabilities document (XML) Point of entry: GetCapabilities request
41. Client Server - Description of service - Layers - Requests - GetMap - GetFeatureInfo - GetLegendGraphic Client GetCapabilities Capabilities document (XML) Server Subsequent requests .png, .jpg and text Parse
42. http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492& width=300& height=400& format=image/png GetMap - request
43. GetMap - response http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=getMap& service=WMS& version=1.1.1& layers=ahn25m& srs=EPSG:28992& bbox=13014,306243,286599,623492& width=300& height=400& format=image/png
44. GetFeatureInfo - request http://geodata.nationaalgeoregister.nl/ahn25m/wms? request=GetFeatureInfo& service=WMS& version=1.1.1& query_layers=ahn25m& info_format=text/html& x=353& y=145& layers=ahn25m& srs=EPSG:28892& bbox=197896, 429027, 205562, 433818& format=image/png& width=780& height=330&
45. GetFeatureInfo - response http://docs.geoserver.org/latest/en/user/services/wms/reference.html text/plain text/html
46. View and map
47. Viewing data: WMS PDOK kaart wizard kaart.pdok.nl QGIS - open source GIS OpenLayers 2 >> WMS made easy
48. QGIS add WMS layer QGIS add WMS layer Add WMS layer button
49. OpenLayers - WMS var map = new OpenLayers.Map('map', { maxExtent: new OpenLayers.Bounds(10000.0, 305000.0, 280000.0, 619000.0), projection: new OpenLayers.Projection("EPSG:28992") }); var wms_url = "http://geodata.nationaalgeoregister.nl/ahn25m/wms" var layer = new OpenLayers.Layer.WMS( "Actueel Hoogtebestand Nederland 25 m", wms_url, { layers: "ahn25m", format: "image/png", srs: "EPSG:28992", styles: "" }, { transitionEffect: 'resize' } ); map.addLayer(layer); map.setCenter(new OpenLayers.LonLat(445463, 156271), 1); http://bl.ocks.org/ndkv/
50. OpenLayers - Feature info var getFeatureControl = new OpenLayers.Control.WMSGetFeatureInfo({ url: proxy + wms_url, layers: [layer], queryVisible: true, eventListeners: { getfeatureinfo: function (event) { var pop = new OpenLayers.Popup.FramedCloud( "ahn25m", map.getLonLatFromPixel(event.xy), null, event.text, null, true) pop.panMapIfOutOfView = false; map.addPopup(pop); } } });
51. Waalweelde dashboard http://waalweelde.ndkv.nl
52. WMS - a note on tiling WMS: single images, no tiling, no caching WMS-C: tiling, caching - same endpoint as "pure" WMS - invoke with parameter: tiled=true - OSM and Leaflet do this automagically TMS / WMTS - used for the Basic Registry Topography service - TMS docs: http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification - WMTS docs: ...
53. Anatomy of WFS Delivers geometry Geographical Markup Language GeoJSON Anatomy of WFS BAG http://geodata.nationaalgeoregister.nl/bagviewer/wfs?
54. Client Server - Description of service - Filters - Functions - Feature types i.e. layers - Requests - GetFeature [ filter ] - DescribeFeatureType - ... Client GetCapabilities Capabilities document (XML) Server Subsequent requests .gml, .json, .shp Parse
55. http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& vesion=2.0.0& srs=EPSG:28992& bbox=70420.6,429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json GetFeature - request
56. { "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates": [ [ [ 70418.262, 429235.086 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature - response
57. http://geodata.nationaalgeoregister.nl/bagviewer/wfs? request=getFeature& service=wfs& version=2.0.0 srs=EPSG:28992& bbox=70420.6, 429242.6,70423.6,429243& typeName=bagviewer:pand& outputFormat=json& srsname=EPSG:4326 Transform coordinates
58. { "type": "Feature", "id": "pand.fid--2e3a32e5_13db0af9ceb_65e", "geometry": { "type": "Polygon", "coordinates": [ [ [ 4.159718956737263, 51.84527267018053 ], [ ... ] ] ] } "geometry_name": "geometrie", "properties": { "gid": 6571756, "identificatie": 530100002242340, "bouwjaar": 2002, "status": "Pand in gebruik", "gebruiksdoel": "woonfunctie", "oppervlakte_min": 165, "oppervlakte_max": 165, "aantal_verblijfsobjecten": 1 } } GetFeature in lon/lat
59. Get, transform, save QGIS reads WFS QGIS transforms coordinates QGIS saves to disk QGIS handles many file types
60. Add WFS layer Add WFS layer button
61. Save WFS layer as lat/lon in GeoJSON Right-click on layer -> Save As...
62. Images vs features WMS Data display Images only Predefined styling No analysis WFS Data download Access to geometries Not styled, up to you Analysis
63. WMS + WFS - Serve tiles with WMS - Get object id through GetFeatureInfo - Get object geometry through WFS GetFeature http://example.com/geoserver/wfs? service=wfs& version=2.0.0& request=GetFeature& typeName=namespace:featuretype& featureID=feature http://bagviewer.geodan.nl/index.html
64. Make me a map with TileMill (and CartoDB)
65. Make me a map 1. Extract data from WFS endpoint 2. Style with TileMill / CartoDB 3. Host on web through mapbox.com, own server and CartoDB
66. TileMill + MapProxy open-source cartography toolbox by MapBox open-source proxy for geo data - Style GIS data with CartoCSS (revolution! Google "SLD") - Bake .mbtiles - pre-rendered tile cache in SQLite - Host on mapbox.com - Cache, accelerate and transform data from existing data sources: WMS, .mbtiles and others - Disseminate through WMS, WMTS, TMS and KML - View with OpenLayers MapProxy or TileStache or Django TileMill
67. Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >> inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr
68. Data and tools CBS census data http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs - Municipalities >> inhabitants - Neighbourhoods >> inhabitants Road network http://geodata.nationaalgeoregister.nl/nwbwegen/wfs - Street geometries and names QGIS and gdal/ogr Somehow QGIS saves WFS numerical fields as text... ... GDAL/OGR to the rescue!
69. gdal/ogr Swiss army knife for querying, transforming and manipulating GIS data shapefile, csv, gml, gpx, kml, geojson, sqlite, postgresql, mysql, svg, couchdb, wfs ogrinfo - investigate data source ogr2ogr - interact with data [-f format_name] - write output in specified format [-s_srs] - source coordinate system [-t_srs] - target coordinate system [-sql] - execute sql query
70. ogrinfo investigate WFS endpoint ogrinfo -so WFS:"url" investigate gemeenten2011 layer ogrinfo -so WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
71. ogr2ogr - get - ogr2ogr -f format_name destination source get gemeenten2011 layer in native SRS as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
72. ogr2ogr - get + transform - ogr2ogr -f format_name destination source target_srs get gemeenten2011 layer in lat/lng as GeoJSON ogr2ogr -f GeoJSON gemeenten2011.geojson WFS:"url" -t_srs EPSG: 4326 gemeenten2011 url = http://geodata.nationaalgeoregister.nl/wijkenbuurten2011/wfs
73. ogr2ogr - sql ogr2ogr -f format_name destination source target_srs sql ogr2ogr -f GeoJSON gemeenten_2011_84.geojson WFS:" url" -sql "SELECT 'gemeentenaam', cast('aantal_inwoners' AS integer) AS 'aantal_inwoners' FROM 'wijkenbuurten2011:gemeenten2011'" -t_srs EPSG:4326 Load in TileMill
74. CartoDB "Web platform for creating dynamic maps [...]" - Style GIS data with CartoCSS (revolution! Google "SLD") - Store data in PostGIS (revolution! Gazillions of spatial functions) - SQL queries directly from JS (revolution! NoSQL is dead) - Real-time visualisations (revolution!) - Hosting at cartodb.com is expensive... - ... but it's on github
75. Prepare shp - ogr2ogr -f format_name destination source ogr2ogr -f "ESRI Shapefile" gemeenten2011.shp gemeenten2011.geojson - Zip gemeente2011.* and upload to CartoDB
76. Neat Thanks for your time! (Code) examples at @simeonnedkov http://ndkv.nl bl.ocks.org/ndkv waalweelde.ndkv.nl