Elasticsearch & Kibana

Introduction Elasticsearch and Kibana.

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani 2014/Sep. Elasticsearch & Kibana
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΞδΣϯμ • Elasticsearchͷ঺հ • ElasticsearchͷσϞ • KibanaͷσϞ
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel
 Professional services: Support & development subscriptions Trainings
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited શจݕࡧͱ͸ʁ • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ ͓͍ͯɺෳ਺ͷจॻʢϑΝΠϧʣ͔Βಛఆͷจࣈ ྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ΍ʮ୯Ұ ϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻ ʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠ ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞ΕΔɻ
 ʢWikipediaΑΓʣ
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ݕࡧ৚݅ɺݕࡧࣜ
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ༻ޠ • εΩʔϚ υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ υΩϡϝϯτͷొ࿥
9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτͷొ࿥ 1 2 ΧπΦ͸αβΤͷఋ αβΤ͸ϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ ͸ ͸ ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ࿥ ୯ޠʹ෼ׂ ୯ޠ͔Βidͷ഑ྻ͕ Ҿ͚ΔΑ͏ʹ
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ৚݅ೖྗ ΧπΦɹαβΤ
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ ΧπΦ αβΤ 1 1 2 ͸ ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ৚݅ೖྗ ݕࡧ৚݅ͷύʔε
 ݕࡧΫΤϦԽ ΧπΦɹαβΤ
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 ɹ
 ɹ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 

20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ୯ޠͷ۠੾Γํ • ӳޠͷ৔߹ I am speaking Introduction Elasticsearch. 
 
 εϖʔε͕੾Ε໨ͱΘ͔Δ • ೔ຊޠͷ৔߹ ࢲ͸ೖ໳Elasticsearchʹ͍ͭͯ࿩͍ͯ͠Δɻ
 
 Ͳ͜Ͱ۠੾Ε͹Α͍ʁ
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ೔ຊޠ͸୯ޠͷ੾Ε໨͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷ Ωʔ͸ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞੒ • N-Gram NจࣈͣͭจষΛ۠੾Δ • ܗଶૉղੳ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠੾Δ
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ܗଶૉղੳ ! ! • ϝϦοτɿ ҙຯͷ͋Δ୯ޠͷ੾Ε໨
 ඼ࢺ৘ใΛݩʹ௥Ճॲཧ͕Մೳʢޠװม׵ͳͲʣ • σϝϦοτɿ ৽ޠʢະ஌ޠʣʹऑ͍ˠࣙॻϕʔεͷ৔߹ɺࣙॻʹͳ͍୯ޠ͸ݕ ग़ෆೳɻ ΧπΦ͸αβΤͷఋ ΧπΦ ͸ ͷ αβΤ ఋ
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited N-Gram ! ! • ϝϦοτɿ ະ஌ޠʹରԠՄೳ • σϝϦοτɿ ΠϯσοΫεංେԽ ඼ࢺ৘ใʹجͮ͘ॲཧ͕ෆՄೳ ΧπΦ͸αβΤͷఋ Χπ πΦ Φ͸ ͸α αβ βΤ Τͷ ͷఋ
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchͱ͸ʁ
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Πϯετʔϧ͔Βݕࡧ·Ͱ
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited μ΢ϯϩʔυͱىಈ $ wget https://download.elasticsearch.org/... $ tar -xf elasticsearch-1.3.2.tar.gz $ ./elasticsearch-1.3.2/bin/elasticsearch ... [2014-08-29 13:53:07,356][INFO ][node] [Araña] started ... Also puppet modules and RPM/DEB
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ىಈͷ֬ೝ » curl localhost:9200 { "status" : 200, "name" : "Araña", "version" : { "number" : "1.3.2", "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f", "build_timestamp" : "2014-08-13T14:29:30Z", "build_snapshot" : false, "lucene_version" : "4.9" }, "tagline" : "You Know, for Search" }
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }' σʔλొ࿥
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XPUT localhost:9200/books/book/1 -d ' { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }' σʔλߋ৽
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σʔλ࡟আ » curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ » curl —X GET localhost:9200/books/book/1 » curl —X GET localhost:9200/books/book/1/_source
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧ » curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ] } }
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited » curl -XGET ‘localhost:9200/books/book/_search' -d '{ "query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0 ... } }' ݕࡧ - Query DSL
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ෼ࢄߏ੒ɺεέʔϧ
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Basic terms • ΠϯσοΫε σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ SPOFͷղফ • γϟʔσΟϯά ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited γϟʔυͱϨϓϦΧ node 1 orders products 1 4 1 2 2 2 curl  -­‐X  PUT  localhost:9200/orders  -­‐d  '{      "settings.index.number_of_shards"  :  4      "settings.index.number_of_replicas"  :  1   }' curl  -­‐X  PUT  localhost:9200/products  -­‐d  '{      "settings.index.number_of_shards"  :  2      "settings.index.number_of_replicas"  :  0   }'
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited γϟʔυͱϨϓϦΧ node 1 orders products 1 4 1 node 2 orders products 2 2 3 4 1 2 3
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࣗಈతͳ෼ࢄ node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders products 3 4 1 3
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ϋϥελ؅ཧ • ୯ҰϚελ Ϋϥελͷঢ়ଶʢϊʔυ৘ใɺϚοϐϯάʣΛ؅ཧ • ϚϧνΩϟετʹΑΔϊʔυ୳ࡧ
 ɹ (ϢχΩϟετ΋Մೳ) • ඞཁͳઃఆ Ϋϥελ໊
 ࠷খϚελϊʔυ਺ • Tips: 3ͭͷϚελϊʔυΛ༻ҙ͠ɺσʔλΛอ࣋ ͠ͳ͍
47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΫϥελɺϊʔυͷαΠδϯά • σʔλྔͱϦΫΤετʹґଘ υΩϡϝϯτ਺͸ʁϑΟʔϧυ਺͸ʁΫΤϦͷׂ߹͸ʁ
 Facet΍AggregationɺιʔτɺΧελϜείΞͷར༻͸ʁ
 ॻ͖ࠐΈͷׂ߹͸ʁυΩϡϝϯτ͸࡟আ/ߋ৽͢Δʁ
 ࣌ؒʹґଘͨ͠σʔλʁ • 1ϊʔυɺϨϓϦΧແ͠Ͱςετ γϟʔυαΠζɺJVMώʔϓ࢖༻ྔɺGCස౓ɺγϟʔυ͓Α ͼϊʔυ਺ɺγϟʔυ͋ͨΓͷυΩϡϝϯτ਺ɺCPU࢖༻ ཰ɺσΟεΫ࢖༻཰ɺΠϯσοΫεͷύλʔϯ • Tips: 30 GB heap
48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ϋϥελ͓ΑͼΞϓϦͷߏ੒ • ϩʔΧϧσΟεΫ; SAN OK • Tips: ϨΠςϯγͷߴ͍WAN্ͷΫϥελ͸ඇਪ ঑ɻCross-zone in EC2 is OK. ! Your app ES Data 1 ES Data N ... ES Master; no data ! Your app ... ES Master; no data ES Master; no data High Speed Network ES Node Client ES Node Client
49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ΠϯσοΫεͷొ࿥ͱݕࡧͷ ಺෦ಈ࡞
50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτΛొ࿥ doc1 node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1
51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • υΩϡϝϯτͷIDͷϋογϡ஋Λܭࢉͯ͠γϟʔ υ਺ͰׂΓɺొ࿥͢ΔγϟʔυΛܾΊΔ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1
52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • γϟʔυ͕ܾ·ͬͨΒɺϓϥΠϚϦγϟʔυ͕͋ Δϊʔυ͕Ͳ͔͜Λௐ΂Δ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1
53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited υΩϡϝϯτొ࿥γʔέϯε • ϓϥΠϚϦ͔ΒϨϓϦΧ΁ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 doc1
54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݕࡧΫΤϦΛ࣮ߦ Query node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1
55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query
56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ϊʔυ͕̍ड͚औͬͨΫΤϦΛશγϟʔυʹૹ৴ • γϟʔυຖʹݕࡧ࣮ߦ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query Query Query Query Query
57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • શγϟʔυ͔Βͷ݁ՌΛϚʔδ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 Query ݁Ռ ݁Ռ ݁Ռ ݁Ռ
58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • શγϟʔυ͔Βͷ݁ՌΛϚʔδ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1 ݁Ռ
59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ݕࡧγʔέϯε • ݁ՌΛૹ৴ ݁Ռ node 1 bookstore 2 1 4 node 2 bookstore 2 3 3 4 1
60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysis & Mappings
61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͱ͸ʁ • సஔΠϯσοΫεͷ୯ޠʢTermʣΛυΩϡϝϯτ ͔Βநग़͢Δॲཧ • ϑΟʔϧυຖʹࢦఆ͞ΕͨΞφϥΠβ͕ॲཧ • ΞφϥΠβ͸CharFilterɺTokenizerɺTokenFilter͔ Βߏ੒ • ΠϯσοΫε࣌ɺݕࡧ࣌ʹ࣮ߦ
62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͷ֓ཁ Analysis υΩϡϝϯτ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ... ΠϯσοΫε
63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analysisͷ֓ཁ Analysis ݕࡧ
 Ωʔϫʔυ ΠϯσοΫε ΠϯσοΫε
 ݕࡧॲཧ ݕࡧ݁Ռ Term Id ΧπΦ 1 αβΤ 1ɺ2 ϫΧϝ 2 … ...
64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzer Analysisͷߏ੒ char_filter tokenizer char_filter token_
 filter token_
 filter Text Tokens
65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Analyzerͷઃఆ {"index":{ “analysis":{ "analyzer" : { "my_analyzer" : { "type" : "custom", "tokenizer" : “kuromoji_tokenizer”, “char_filter" : [“char_filter1”,…], “filter" : [“token_filter1”,…] } }
66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Char Filter • ೖྗจࣈྻΛจࣈ୯ҐͰॲཧ ! • ྫɿhtml_strip ೖྗɿ ! ग़ྗɿ <title>Elasticsearch is not a service of AWS</title> Elasticsearch is not a service of AWS
67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ೖྗจࣈྻΛτʔΫϯྻʹ෼ׂ • τʔΫϯ΁ͷ෼ׂ͢ΔϩδοΫ͸τʔΫφΠβʹ ґଘ • ྫɿstandard ೖྗɿ ! ग़ྗɿ Elasticsearch is not a service of AWS Elasticsearch is not a service of AWS
68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tokenizer • ྫɿkeyword ग़ྗɿ • ྫɿkuromoji_tokenizer ೖྗɿ ! ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ Elasticsearch is not a service of AWS ण͕࢘ඒຯ͔ͬͨ͠
69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • TokenizerʹΑΓग़ྗ͞ΕͨTokenྻʹରͯ͠ॲཧ • ྫɿlowercase ೖྗɿ ! ग़ྗɿ • ྫɿstop ೖྗɿ ! ग़ྗɿ Elasticsearch is not a service of AWS elasticsearch is not a service of aws Elasticsearch is not a service of AWS Elasticsearch service AWS
70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited TokenFilter • ྫɿkuromoji_baseform ೖྗɿ ! ग़ྗɿ • ྫɿkuromoji_readingform ೖྗɿ ! ग़ྗɿ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ ण࢘ ͕ ඒຯ͍͠ ͨ ण࢘ ͕ ඒຯ͔ͬ͠ ͨ sushi ga oishika ta
71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Mapping • ΠϯσοΫεͷߏ଄ʢschemaʣΛఆٛ • Schema less!͚ͩͲ… • ܗࣜΛࢦఆʢanalyze͢Δ/͠ͳ͍ɺ੔਺/গ਺ɺ೔ ෇ͳͲʣ
72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿWikipediaͷMappingͷҰ෦ "mappings": { "page": { "properties": { "link": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} } }, "text": { "type": "string"}, "title": { "type": "string", "fields": { "raw": {"type": "string", "index": "not_analyzed"} }…
73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited aggregation
74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregationͱ͸ • 1.0͔Βಋೖ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ
75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͝ͱͷϑΝηοτ » curl -XGET twitter-2014.08.22/_search -d ' GET twitter-2014.08.22/_search { "facets": { "lang_facet": { "terms": {"field": "lang"} } } }
76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͝ͱͷϑΝηοτ "facets": { "lang_facet": { "_type": "terms", "missing": 0, "total": 4521075, "other": 320228, "terms": [ { "term": "en", "count": 1471819 }, { "term": "ja", "count": 980145 },
77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͓Αͼ஍Ҭͷूܭ » curl -XGET twitter-2014.08.22/_search -d ' { "aggs": { "lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿݴޠ͓Αͼ஍Ҭͷूܭ "aggregations": { "lang": { "buckets": [{…}, { "key": "ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژ౎ࢢ෬ݟ۠, ژ౎", "doc_count":252 }, { "key": "ઍ୅ా۠, ౦ژ", "doc_count": 39 },…
79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited bucket • filter • filters (>=1.4.0) • missing • nested • reverse nested (>=1.2.0) • children • terms • significant terms (>=1.1.0) • range • date range • ipv4 range • histogram • date histogram • geo distance • geohash grid
80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited metric • min • max • sum • avg • stats • extended_stats • value count • percentiles (>=1.1.0) • percentile ranks (>=1.3) • cardinality (>=1.1.0) • geo bounds (>=1.3.0) • top_hits (>=1.3.0) • scripted metric 
 (>= 1.4.0)
81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ͦͷଞͷػೳ
82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Geo • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ • σʔλɿ Ң౓ܦ౓ɺGeoHashɺGeoShape • ݕࡧɿ geo shape queryɺgeo shape filterɺgeo distance filterɺ geo distance range filterɺgeo bounding box filterɺ geo polygon filter
83. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλొ࿥ ## Ң౓ܦ౓ { "pin" : { "location" : "41.12,-71.34" } } ! ## GeoHash { "pin" : { "location" : "drm3btev3e86" } }
84. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλొ࿥ ## GeoShape(Polygon) { "location" : { "type" : "polygon", "coordinates" : [[ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0], [100.0, 1.0], [100.0, 0.0] ]] } } ## GeoShape(Circle) { "location" : { "type" : "circle", "coordinates" : [-45.0, 45.0], "radius" : "100m" } }
85. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿGeoσʔλݕࡧ { "query": { "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [[13, 53],[14, 52]] } } } } }
86. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿڑ཭ʹΑΔϑΟϧλ { "filtered" : { "query" : {"match_all" : {}}, "filter" : { "geo_distance" : { "distance" : "200km", "pin.location" : { "lat" : 40, "lon" : -70 } } } } }
87. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Percolator • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥ • _percolate APIʹυΩϡϝϯτΛૹ৴ • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ
 Ϛονͨ͠ΫΤϦΛฦ͢
88. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿΫΤϦొ࿥ curl -XPUT 'localhost:9200/my-index/.percolator/1' -d '{ "query" : { "match" : { "message" : "bonsai tree" } } }'
89. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿσʔλνΣοΫ curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{ "doc" : { "message" : "A new bonsai tree in the office" } }'
90. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿ݁Ռ { "took" : 19, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "total" : 1, "matches" : [ { "_index" : "my-index", "_id" : "1" } ] }
91. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Snapshot/Restore • 1.0͔Βಋೖ • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ • อଘઌʢRepositoryʣ Shard FSɺS3ɺHDFSɺAzure…
92. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿϦϙδτϦ࡞੒ $ curl -XPUT 'http://localhost:9200/_snapshot/ my_backup' -d '{ "type": "fs", "settings": { "location": "/mount/backups/my_backup", "compress": true } }'
93. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫɿSnapshotͱϦετΞ ## Snapshot
 $ curl -XPUT “localhost:9200/_snapshot/my_backup/ snapshot_1?wait_for_completion=true" ! ! ## Restore $ curl -XPOST "localhost:9200/_snapshot/my_backup/ snapshot_1/_restore"
94. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ecosystem • Plugins ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ Ruby, python, php, perl, javascript, .NET Scala, clojure, go • Kibana • Logstash
95. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Hadoop࿈ܞ&Kibana • ʮαʔόʗΠϯϑϥΤϯδχΞཆ੒ಡຊ ϩάऩ ूʙՄࢹԽฤʯൃചத • Hadoop࿈ܞͷ঺հ • Kibanaͷৄࡉ
96. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
97. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Tools for sys admins
98. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited REST-based management • Elasticsearch is full of monitoring APIs Everything is returned as JSON • Humans are not the world’s best JSON parsers • What if elasticsearch had an easy to use interface from the commandline?
99. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which node is the master? $ curl "localhost:9200/_cluster/state?pretty&filter_metadata=true&! filter_routing_table=true"! {! "cluster_name" : "elasticsearch",! "master_node" : "GNf0hEXlTfaBvQXKBF300A",! "blocks" : { },! "nodes" : {! "ObdRqLHGQ6CMI5rOEstA5A" : {! "name" : "Triton",! "transport_address" : “inet[/10.0.1.11:9300]”,! "attributes" : { }! },! "4C7pKbfhTvu0slcSy_G4_w" : {! "name" : "Kid Colt",! "transport_address" : "inet[/10.0.1.12:9300]",! "attributes" : { }! },! "GNf0hEXlTfaBvQXKBF300A" : {! "name" : "Lang, Steven",! "transport_address" : "inet[/10.0.1.13:9300]",! "attributes" : { }! }! }! }
100. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Which one is the master? (v1.0) $ curl localhost:9200/_cat/master GNf0hEXlTfaBvQXKBF300A 10.0.1.13 Lang, Steven
101. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited _cat/* api • /_cat/allocation • /_cat/count • /_cat/health • /_cat/master • /_cat/aliases • /_cat/nodes • /_cat/recovery • /_cat/shards • /_cat/indices • /_cat/thread_pool
102. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Monitor your cluster with Marvel • Point in time views are a start • Marvel shows historical trends • Visualize cluster behavior, act before problems ! • Free for development, $500/year for up to 5 nodes
103. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Overview
104. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Node statistics
105. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Index statistics
106. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Cluster Pulse
107. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Sense
108. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ for Elasticsearch
 ೔ຊޠWikipedia
109. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ for Elasticsearch
 Twitter Stream
110. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ for Kibana
 Twitter Stream
111. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited σϞ for Kibana
 Git log
112. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟจݙ • Elasticsearch - The Definitive guide http://www.elasticsearch.com/guide/en/elasticsearch/guide/ current/index.html • ॻ੶ ElasticSearchServer೔ຊޠ൛
 αʔό/ΠϯϑϥΤϯδχΞ
 ɹཆ੒ಡຊɹϩάऩू
113. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Q&A • Training http://training.elasticsearch.com • Support http://www.elasticsearch.com/support/ • Github https://github.com/elasticsearch