Elastic{ON} 2018 - The State of Geo in Elasticsearch

It's everything you ever wanted to know about the latest geo capabilities in Elasticsearch and Apache Lucene — all in one session.

Learn about the data structures that enable geospatial indexing and search, get advice on field mapping strategies, and hear all about existing and upcoming geo aggregations for spatial data analysis. Plus, hear all about new spatial data structures and upcoming geo features being added to Lucene and Elasticsearch.

Nick Knize | Elasticsearch Software Engineer | Elastic
Thomas Neirynck | Software Engineer | Elastic

Transcript

1. Elastic March 1, 2018 @nknize The State of Geo in Elasticsearch Nick Knize, Elasticsearch Software Engineer Thomas Neirynck, Kibana Visualization Area Lead
2. 2 Geospatial capabilities are becoming more popular among Elasticsearch Users
3. Topics 3 Geospatial Indexing, Search, and Visualization 1 Kibana / Elastic Maps Service 2 Geo Field Mappings 3 Geo Indexing, Search, and Lucene Data Structures 4 Geo Aggregations
4. Kibana / Elastic Maps Service
5. Kibana Visualizations 5 Out-of-the-box visualizations for geodata in Elasticsearch 2 types - Coordinate Maps - Region Maps Visualize is built on top of the Elasticsearch aggregations
6. Coordinate Map Visualization 6 Shows result of geohash_grid aggregations. Shows summary of all documents that belong to a single cell. Put location of “summarized” point in the “geo-centroid” (weighted middle). This gives a better approximate location. The more zoomed in, the more precise the location. Different marker-styles (bubbles, heatmap)
7. Example 1
8. Example 2
9. Region Maps 9 “Choropleth maps” Thematic maps: color intensity correspond to magnitude of metric Shows result of terms aggregations. “Client-side” join between the result of term aggregation and a reference shape layer. - Polygons/Multipolygons (simple feature) - Documents in elasticsearch need to have field that matches a property of the
10. Request traffic Region Maps
11. Vega 1 1 Experimental feature Vega/VegaLite is a domain language in JSON to create visualizations. Vega has support for geographic projection.
12. Dashboard integration 1 2 - Use map for spatial filtering of data ... - … and have other filters applied to your map
13. Elastic Maps Service
14. Elastic Maps Service 1 4 Reference basemapping and reference data service hosted by Elastic. “Getting started” experience for mapping. (1) World base map - Base for Coordinate Map, Region Map (2) Shape layers - World countries, US States, Germany States, Canada Provinces, USA zip-codes - Number of identifier fields (name in one or more languages, and ISO-identifiers)
15. Integrating Custom Maps
16. Custom base maps 1 6 - (1) Configure global base-map in kibana.yml by using Tile Map Service URL tilemap.url: https://tiles.elastic.co/v2/default/{z}/{x}/{y} - (2) Configure visualization-specific base-map using WMS (web map service) - Requires 3rd party geo-service - Geoserverb - ArcGIS Server - MapServer - ….
17. Custom maps examples 1 7 Image Removed
18. Custom shape layers 1 8 - geojson/topojson - Configure in kibana.yml -> available in region maps UI regionmap: includeElasticMapsService: false layers: - name: "Departments of France" url: "http://my.cors.enabled.server.org/france_departements.geojson" attribution: "INRAP" fields: - name: "department" description: "Full department name" - name: "INSEE" description: "INSEE numeric identifier" - Use any web-server - Make sure is CORS enabled so Kibana can download the data (!)
19. - customization - https://www.elastic.co/blog/kibana-and-a-custom-tile-server-for-nhl-data - https://www.elastic.co/blog/custom-region-maps-in-kibana-6-0 Useful blog posts
20. Future
21. Elastic Maps Service - More base layers (satellite, contours) - Different stylesheets - On-prem deployments Kibana - Elastic Maps Service integration with Vega - No restriction on number of layers - Support for geo_shape - Visualize individual documents/custom styling - Spatial filtering Upcoming
22. Mappings Geo Field Types
23. 23 PUT crime/incidents/_mapping { “properties” : { “location” : { “type” : “geo_point”, “ignore_malformed” : true, } } } define geo_point mapping
24. POST crime/incidents { “location” : { “lat” : 41.12, “lon” : -71.34 } } 24 insert geo_point mapping POST crime/incidents { “location” : “41.12, -71.34” } POST crime/incidents { “location” : [[-71.34, 41.12], [-71.32, 41.21]] }
25. 25 define geo_shape mapping PUT police/precincts/_mapping { “properties” : { “coverage” : { “type” : “geo_shape”, “ignore_malformed” : false, “tree” : ”quadtree”, “precision” : “5m”, “distance_error_pct“ : 0.025, “orientation” : “ccw”, “points_only” : false } } }
26. 26 insert geo_shape mapping POST police/precincts/ { “coverage” : { “type” : “polygon”, “coordinates” : [[ [-73.9762134, 40.7538588], [-73.9742356, 40.7526327], [-73.9656733, 40.7516774], [-73.9763236, 40.7521246], [-73.9723788, 40.7516733], [-73.9732423, 40.7523556], [-73.9762134, 40.7538588] ]] } }
27. • Shapes are parsed using OGC and ISO standards definitions • OGC Simple Feature Access • ISO Geographic information — Spatial Schema (19107:2003) • Supports the following geo_shape types • Point, MultiPoint • LineString, MultiLineString • Polygon (with holes), MultiPolygon (with holes) • Envelope (bbox) geo_shape mapping 27 insert
28. 28 geo_point mapping Pre 5.0
29. 29 geo_point mapping 5.0+
30. 30 geo_shape mapping current
31. 31 geo_shape mapping 7.0+
32. ‹#› Geo Indexing 32
33. 33 geo_point indexing 2.x term/postings encoding term postings (doc ids) 1 1, 2, 3, 4, 5 10 1, 2, 4 11 3, 5 100 1 101 2, 4 111 3, 5 1000 2 1010 4 1011 3 1110 3 1111 5
34. 34 geo_point indexing 5.0 - “points” data structure - (Bkd-tree)
35. 35 geo_point indexing 5.0 - “points” data structure - (Bkd-tree)
36. 36 geo_point indexing performance improvements
37. 37 geo_shape indexing current - terms/postings encoding • Max tree_levels == 32 (2 bits / cell) • distance_error_pct • “slop” factor to manage transient memory usage • % of the diagonal distance (degrees) of the shape • Default == 0 if precision set (2.0) • points_only • optimization for points only shape index • short-circuits recursion
38. 38 geo_shape indexing 7.0+ - “ranges” encoding (Bkd-tree) • Dimensional Shapes represented using Minimum Bounding Ranges (MBR) ‒ Ranges (1D) - Available from 5.1+ for numerics, dates, and IP (v4 and v6) ‒ Rectangles (2D) - LatLonBoundingBox Available in Lucene 7.1+ ‒ Cubes (3D) ‒ Tesseract (4D) Quad Cells Indexed as LatLonBoundingBox
39. 39 geo_shape indexing performance - 1D Numerics
40. ‹#› Geo Search 40
41. 41 geo_point search Pre 5.0 - terms/postings encoding • Spatial Queries • BoundingBox, Distance, DistanceRange, Polygon • PRECISION_STEP controls number of query terms (must match with index) • TwoPhaseIterator • Delays boundary confirmation so other query (filters, conjunctions) can pre-filter
42. 42 geo_point search 5.0+ - “points” encoding (Bkd-tree) Leaf cell is fully within polygon (salmon) - return all docs Leaf cell crosses the boundary (gray) - two-phase check 1 2
43. 43 geo_point search 5.0+ - performance improvements
44. 44 geo_shape search capabilities • Supports the following geo_shape types ‒ Point, MultiPoint ‒ LineString, MultiLineString ‒ Polygon (with holes), MultiPolygon (with holes) ‒ Envelope (bbox) • Supports relational queries ‒ INTERSECTS, DISJOINT, WITHIN, CONTAINS
45. 45 geo_shape search current - terms/postings encoding Recursively Traverse Query terms 1 2 Collect DocIDs from Postings based on requested relation
46. 46 geo_shape search 7.0+ - “points” encoding (B-kd Tree)
47. 47 geo_shape search 7.0+ - “points” encoding (B-kd Tree)
48. 48 geo_shape search 1D numeric range performance
49. ‹#› Geo Aggregations 49
50. ‹#› 50 GeoDistance Agg { "aggs" : { “sf_rings" : { "geo_distance" : { "field" : "location", "origin" : [32.95, -96.82], "ranges" : [ { "to" : 50 }, { "from" : 50, "to" : 100 }, { "from" : 100, "to" : 300} ] } } } }
51. ‹#› 51 GeoDistance Agg
52. ‹#› 52 GeoGrid Agg { "aggs" : { “crime_cells" : { "geohash_grid" : { "field" : "location", "precision" : 8 } } } }
53. ‹#› 53 GeoGrid Agg
54. ‹#› 54 GeoCentroid Agg "query" : { "match" : { "crime" : "burglary" } }, "aggs" : { "towns" : { "terms" : { "field" : "town" }, "aggs" : { "centroid" : { "geo_centroid" : { "field" : “location" } } } } }
55. ‹#› 55 GeoCentroid Agg
56. ‹#› 56 GeoCentroid Agg
57. 57 Geo Aggregations more available, and coming soon... • matrix_stats - (Matrix Aggs) plugin ‒ kurtosis/skewness ‒ variance-covariance matrix ‒ pearson’s product correlation matrix • geo_stats - Future? ‒ Moran’s I - measuring spatial auto-correlation ‒ Getis-Ord - spatial hot spot analysis
58. Questions? 19