Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic stackにおける 地理情報の利用方法 #foss4gj

Jun Ohtani
September 16, 2017

Elastic stackにおける 地理情報の利用方法 #foss4gj

FOSS4G Tokyo 2017での発表資料です。
Geo系の紹介になります。

公式ドキュメントはこちら。https://www.elastic.co/guide
右上の虫眼鏡マークで知りたい機能を検索してください。

Jun Ohtani

September 16, 2017
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. ΞδΣϯμ • Elastic Stack ͱ͸ʁ • Elastic StackͰͷ஍ཧ৘ใͷػೳ • LogstashͰͷGeoIP

    • ElasticsearchͰͷҢ౓ܦ౓ରԠ • KibanaͰͷ஍ਤରԠ 3
  2. about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ

    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS 4
  3. Logstash architecture 9 Input Output Filter ? ? collect and

    split alter and enrich store and visualize
  4. ઃఆɿfilter 10 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  5. ύʔε 11 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
  6. Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:

    Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ 22
  7. σʔλొ࿥ 23 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
  8. Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •

    Apache License 2.0 • Elastic Stackͷ૭ͷ໾ׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 27
  9. ઃఆɿgeoip filter 31 filter { grok { match => {

    "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
  10. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 32 "clientip": "124.35.xx.xx", "clientip": "124.35.xx.xx", "geoip": { "continent_name": "Asia",

    "city_name": "Tokyo", "country_iso_code": "JP", "region_name": "Tokyo", "location": { "lon": 139.7559, "lat": 35.6845
  11. geoip filter • MaxMind GeoLite2 CityσʔλϕʔεΛϥΠϒϥϦʹόϯυϧ • IPv4ͱIPv6ͷ྆ํʹରԠ • Autonomous

    System Numberͷσʔλϕʔε΋੾Γସ͑Մೳ • MaxMindͷ঎༻σʔλϕʔε΋ར༻Մೳ • `database`ύϥϝʔλͰϑΝΠϧΛࢦఆՄೳ 33
  12. Sample data Point 36 { "location" : { "type" :

    "point", "coordinates" : [-77.03653, 38.897676] } }
  13. Sample data LineString 37 { "location" : { "type" :

    "linestring", "coordinates" : [[-77.03653, 38.897676], 
 [-77.009051, 38.889939]] } }
  14. Sample data Polygon 38 { "location" : { "type" :

    "polygon", "coordinates" : [ [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ] ] } }
  15. Sample data Geometry Collection 39 { "location" : { "type":

    "geometrycollection", "geometries": [ { "type": "point", "coordinates": [100.0, 0.0] },{ "type": "linestring", "coordinates": [ [101.0, 0.0], [102.0, 1.0] ] } ]
  16. Queries 40 • geo_shape query • geo_shapeͷݕࡧ • geo_bounding_box query

    • geo_pointͷݕࡧʢۣܗʹΑΔݕࡧʣ • geo_distance/geo_distance_range query • geo_pointͷݕࡧʢத৺఺ʴڑ཭ʹΑΔݕࡧʣ • geo_polygon query • geo_pointͷݕࡧʢϙϦΰϯʹΑΔݕࡧʣ
  17. Queries geo_shape query 41 GET /example/_search {... "filter": { "geo_shape":

    { "location": { "shape": { "type": "envelope", "coordinates" : [[13.0, 53.0], [14.0, 52.0]] }, "relation": "within" }...
  18. Queries geo_bounding_box query 42 "filter" : { "geo_bounding_box" : {

    "pin.location" : { "top_left" : { "lat" : 40.73, "lon" : -74.1 }, "bottom_right" : { "lat" : 40.01, "lon" : -71.12 }...
  19. Queries geo_distance query 43 ... "filter" : { "geo_distance" :

    { "distance" : "200km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
  20. Queries geo_distance_range query 44 ... "filter" : { "geo_distance_range" :

    { "from" : "200km", "to" : "400km", "pin.location" : { "lat" : 40, "lon" : -70 } } }...
  21. Queries geo_polygon query 45 ... "filter" : { "geo_polygon" :

    { "person.location" : { "points" : [ {"lat" : 40, "lon" : -70}, {"lat" : 30, "lon" : -80}, {"lat" : 20, "lon" : -90} ] } }...
  22. aggregation (grouping) • geo_bounds aggregation • ର৅ͷgeo_point͕શͯೖΔbounding boxΛܭࢉͯ͠ฦ٫ • geo_centroid

    aggregation • ର৅ͷgeo_point͔ΒCentroidΛܭࢉͯ͠ฦ٫ • geo_distance aggregation • ࢦఆ͞Εͨத৺఺͔Βࢦఆ͞Εͨڑ཭͝ͱͷ݅਺Λฦ٫ • geo_hash aggregation • ࢦఆ͞ΕͨprecisionͰͷgeohash͝ͱͷ݅਺Λฦ٫ 46
  23. Sorting 47 GET /_search { "sort" : [ { "_geo_distance"

    : { "pin.location" : [-70, 40], "order" : "asc", "unit" : "km", "mode" : "min", "distance_type" : "arc" } }]... }
  24. ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •

    ॻ੶ʢ೔ຊޠʣ ‒ ElasticSearchServer೔ຊޠ൛ ‒ σʔλ෼ੳج൫ߏஙೖ໳
 2017೥9݄21೔ൃച 54
  25. ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •

    Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 55
  26. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co