Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchの始め方
Search
Jun Ohtani
September 02, 2016
Technology
9
36k
Elasticsearchの始め方
2016/09/02 JJUG ナイトセミナーでの発表資料。
Jun Ohtani
September 02, 2016
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.7k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
990
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
890
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.7k
What's new in Elastic Stack 6.3
johtani
2
2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.3k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
110
Intro Elastic Stack at Telemetry WG
johtani
0
210
What's new in Elastic Stack 6.1?
johtani
0
550
Other Decks in Technology
See All in Technology
Cloud Run と GitHub Template Repository による軽量なアプリケーションプラットフォーム/ #nikkei_tech_talk
nikkei_engineer_recruiting
0
110
エンジニア視点で見る、 組織で運用されるデザインシステムにするには
shunya078
1
310
Road to Single Activity
yurihondo
2
240
Developer Experienceを向上させる基盤づくりの取り組み事例集
coconala_engineer
0
150
言葉は感情の近似値である。その感情と言葉の誤差を最小化しよう ~コミュニケーションにおけるアナログ/デジタル変換の課題に立ち向かう~
nktamago
0
230
学術機関におけるID連携とOpenID Connect
fujie
0
260
リアルお遍路+SORACOM IoT
ozk009
1
140
Envoy External AuthZとgRPC Extensionを利用した「頑張らない」Microservices認証認可基盤
andoshin11
0
260
Next.js のページ遷移を全力で止める
ypresto
7
3.3k
なにもしてないのにNew Relicのデータ転送量が増えていたときに確認したこと
tk3fftk
2
230
不動産 x AIことはじめ~データの真価を拓くために
estie
0
110
プロダクトエンジニアを支えるための開発生産性向上施策
tsukakei
0
140
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
36
6.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
278
13k
The Cost Of JavaScript in 2023
addyosmani
42
5.7k
The Language of Interfaces
destraynor
153
23k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
26
1.9k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
89
16k
Faster Mobile Websites
deanohume
304
30k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
663
120k
Building Your Own Lightsaber
phodgson
101
6k
Designing Experiences People Love
moore
138
23k
Robots, Beer and Maslow
schacon
PRO
157
8.2k
What’s in a name? Adding method to the madness
productmarketing
PRO
21
3k
Transcript
‹#› 2016/09/02 Evangelist at Elastic Jun Ohtani @johtani Elasticsearch (Elastic
Stack)ͷ࢝Ίํ
‹#›
ΞδΣϯμ • Elastic stackͷհ • Elasticsearchͷ࢝Ίํ • Elasticsearchͷଓ͚ํ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Marvel, Shield, Watcher, Graph Professional services: Support & development subscriptions ‒ Trainings 4
࣭ͦͷ1 • ElasticsearchΛ͍ͬͯΔ͔Ͳ͏͔ • ฉ͍ͨ͜ͱ͢Βͳ͍ • ฉ͍ͨ͜ͱ͋Δ͕ɺ৮ͬͨ͜ͱͳ͍ • ฉ͍ͨ͜ͱ͋Δ͠ɺগ͚ͩ͠ࢼͨ͠ •
ຊ൪ڥͰӡ༻த • ฉ͍ͨ͜ͱʁͦΕͲ͜Ζ͔ɺύονʢϓϧϦΫʣॻ͖·ͬͯ͘Δ 5
࣭ͦͷ2 • ElasticsearchҎ֎ͷElastic StackΛͬͨ͜ͱ͕͋Δ • ͬͨ͜ͱͳ͍ • Logstash • BeatsγϦʔζ
• Kibana • X-Pack 6
࣭ͦͷ3 • ElasticsearchҎ֎ͷElastic Stackʹ·ͭΘΔԿ͔Λ։ൃͨ͜͠ͱ͕͋Δ • Logstash • BeatsγϦʔζ • Kibana
• Elasticsearch • X-Pack 7
8 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 9 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 10 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats + Security X-Pack Aler+ng Monitoring Repor+ng Graph
‹#› Ϣʔεέʔε1 ɹݕࡧͱͯ͠ͷElasticsearch
ϑϦʔϫʔυݕࡧ 12
ߜΓࠐΈ 13
ϋΠϥΠτ 14
ιʔτ 15
ϖʔδϯά 16
ूܭ 17
αδΣετ 18
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
20 Elasticsearch is the backbone across all of Wikimedia’s sites,
powering billions of real-time user prefix and full-text searches every day. “ ” Chad Horohoe Software Engineering
‹#› Ϣʔεέʔε2 ɹղੳͱͯ͠ͷElasticsearch
Elastic stackʹΑΔσʔλੳ 22 σʔλ Import/Parse /Export Store/Search Visualize
23 収集、リッチ化、転送 ログおよび数値指標データ センサーおよびデバイスデータ Webおよびソーシャルデータ データストアおよびストリーム 分析 Elasticsearch + 任意のデータストア
アラート Watcher+任意の通知ツール 監視 Marvel+任意の監視ツール アーカイブ Hadoop+任意のクラウドストレージプラットフ ォーム
KibanaͰՄࢹԽ 24
25 Elasticsearch, Logstash, and Kibana allow for real-time indexing, search,
and analytics for over 300 million events per day. This protects our network, services, and systems from security threats. “ ” Jeff Bryner, Security Engineer
beats
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
elasticsearch-hadoop 31 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
32 Elasticsearchͷ ࢝Ίํ
Elasticsearch in 10 seconds • ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ • JavaͷόʔδϣϯʹҙʢJava 8 update 20+ or Java 7 update 55+ʣ • ֦ு༰қʢPluginػߏͳͲʣ 33
‹#› Πϯετʔϧ͔Βىಈ·Ͱ
μϯϩʔυͱىಈʢ2016/09/02࣌ʣ 35 $ wget https://download.elastic.co/elasticsearch/… $ tar -xf elasticsearch-2.4.0.tar.gz $
cd elasticsearch-2.4.0 $ ./bin/elasticsearch
ىಈͷ֬ೝ 36 $ curl localhost:9200 { "name" : "Spinnerette", "cluster_name"
: "elasticsearch", "version" : { "number" : "2.4.0", "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55", "build_timestamp" : "2016-08-29T09:14:17Z", "build_snapshot" : false, "lucene_version" : "5.5.2" }, "tagline" : "You Know, for Search" }
RPM or Deb 37
38 Azure Market Place ˍ ARM Template • ElasticʹΑΓެࣜʹMarketplaceʹͯఏڙ •
Marketplace͔Β؆୯ద༻ • https://azure.microsoft.com/en-us/marketplace/partners/elastic/ elasticsearchelasticsearch/
39 Google Cloud Platform • Google Compute Engine্ʹΫϦοΫͰΠϯετʔϧ • https://www.elastic.co/about/partners/google-compute-engine
40 AWS? • Elastic Cloud
‹#› ݕࡧͱͯ͠ͷElasticsearch
‹#› ؆୯ͳCRUD
σʔλొ 43 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 44 curl -XPUT localhost:9200/books/book/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ 45 curl -X DELETE localhost:9200/books/book/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/book/1 curl —X GET localhost:9200/books/book/1/_source
ݕࡧ 46 curl -XGET localhost:9200/books/_search?q=elasticsearch { "took" : 2, "timed_out"
: false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 1, "max_score" : 0.076713204, "hits" : [ { "_index" : “books", "_type" : “book", "_id" : "1", "_score" : 0.076713204, "_source" : { "title" : "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : “2013-02-04", "pages" : 230 } } ]
ݕࡧ - Query DSL 47 curl -XGET ‘localhost:9200/books/book/_search' -d '{
"query": { "filtered" : { "query" : { "match": { "text" : { "query" : “To Be Or Not To Be", "cutoff_frequency" : 0.01 } } }, "filter" : { "range": { "price": { "gte": 20.0 "lte": 50.0
‹#› ࢄߏ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ 49
γϟʔυͱϨϓϦΧ 50 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ 51 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ 52 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
‹#› શจݕࡧͱʁ
શจݕࡧͱʁ • શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻ ʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯ ʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจ ॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ 54
༻ޠ • ΠϯσοΫε ݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ • υΩϡϝϯτʢจॻʣ ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ • ϑΟʔϧυ
‒ υΩϡϝϯτʹؚ·ΕΔଐੑ • ΫΤϦ ‒ ݕࡧ݅ɺݕࡧࣜ 55
༻ޠ • εΩʔϚ ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ • λʔϜʢTermʣɺτʔΫϯʢTokenʣ ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ ‒
จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ 56
υΩϡϝϯτͷొ 57 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ υΩϡϝϯτͷొ
υΩϡϝϯτͷొ 58 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ υΩϡϝϯτͷొ ୯ޠʹׂ
υΩϡϝϯτͷొ 59 1 2 ΧπΦαβΤͷఋ αβΤϫΧϝͷ࢞ 1 2 ΧπΦ αβΤ
ͷ ͷ αβΤ ϫΧϝ ఋ ࢞ ΧπΦ αβΤ 1 1 2 ͷ ࢞ ϫΧϝ 2 1 2 1 2 1 ఋ 2 υΩϡϝϯτͷొ ୯ޠʹׂ ୯ޠ͔Βidͷྻ͕ Ҿ͚ΔΑ͏ʹ
ݕࡧ 60 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ݕࡧ݅ೖྗ ΧπΦɹαβΤ
ݕࡧ 61 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 62 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 63 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 64 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 65 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
ݕࡧ 66 ΧπΦ αβΤ 1 1 2 ͷ ࢞
ϫΧϝ 2 1 2 1 2 1 ఋ 2 ΧπΦ αβΤ AND ݕࡧ݅ೖྗ ݕࡧ݅ͷύʔε ݕࡧΫΤϦԽ ΧπΦɹαβΤ
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
• ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ 67
୯ޠͷ۠Γํ • ӳޠͷ߹ I am speaking Introduction Elasticsearch.
εϖʔε͕ΕͱΘ͔Δ • ຊޠͷ߹ ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ 68
N-Gramͱܗଶૉղੳ • సஔΠϯσοΫεͷΩʔͷ࡞Γํ ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔ ओʹ࣍ͷ̎ͭͷख๏Ͱ࡞ • N-Gram ‒ NจࣈͣͭจষΛ۠Δ
• ܗଶૉղੳ ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ 69
ܗଶૉղੳ • ϝϦοτɿ ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ • σϝϦοτɿ ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆ
ೳɻ 70 ΧπΦαβΤͷఋ ΧπΦ ͷ αβΤ ఋ
N-Gram • ϝϦοτɿ ‒ ະޠʹରԠՄೳ • σϝϦοτɿ ‒ ΠϯσοΫεංେԽ ‒
ࢺใʹجͮ͘ॲཧ͕ෆՄೳ 71 ΧπΦαβΤͷఋ Χπ πΦ Φ α αβ βΤ Τͷ ͷఋ
‹#› ղੳͱͯ͠ͷElasticsearch
‹#› aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 74
ྫɿݴޠ͓ΑͼҬͷूܭ 75 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 76 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
‹#› ΫϥΠΞϯτϥΠϒϥϦ
ެࣜΫϥΠΞϯτϥΠϒϥϦ • Java • Ruby • PHP • Perl •
Python • .NET • JavaScript • Groovy 78
ΫϥΠΞϯτϥΠϒϥϦʢίϛϡχςΟʣ • Clojure • Cold Fusion • Erlang • Go
• Groovy • Haskell • Java • JavaScript • kotlin 79 • .NET • OCaml • Perl • PHP • Python • R • Ruby • Scala • Smalltalk • Vert.x
80 Elasticsearchͷ ଓ͚ํ
‹#› ڥपΓ
࣮ӡ༻͢ΔࡍʹؾΛ͚ͭΔઃఆ • ϑΝΠϧσεΫϦϓλ • 32,000͘͠64,000͕ਪ • ϝϞϦपΓ • SwapΛOff •
ώʔϓES_HEAP_SIZEͰࢦఆʢ-Xms͓Αͼ-XmxʹಉΛࢦఆʣ • ώʔϓͷϝϞϦϚγϯͷҎԼ • ωοτϫʔΫ • σϑΥϧτlocalhostͷΈ 82
‹#› ใऩू
ެࣜͷࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 84
ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html •
ॻ੶ʢຊޠʣ ‒ ElasticSearchServerຊޠ൛ αʔό/ΠϯϑϥΤϯδχΞ ɹཆಡຊɹϩάऩू 85
meetup.comʢษڧձʣ 86
͍ΖΜͳϒϩά • $shibayu36->blog; • http://blog.shibayu36.org/archive/category/elasticsearch • Wantedly Engineer Blog •
https://www.wantedly.com/companies/wantedly/post_articles/30216 • Hello Elasticsearch! • https://medium.com/hello-elasticsearch • ෆՄࢹ • http://code46.hatenablog.com/entry/2014/01/21/115620 87
‹#› ใڞ༗
ίϛϡχςΟ׆ಈ • ษڧձʢͰൃදʣ • ษڧձʢͰ࠙ձʹࢀՃʣ • ϑΥʔϥϜʹࢀՃʢͯ͠ճʣ • Issue/Pull Requestͷ࡞
• υΩϡϝϯτͷमਖ਼ͱ͔͋Γ·͢Αʂ 89
Thanks for listening! Q & A 90 We’re hiring! https://www.elastic.co/about/careers/
We’re helping! https://www.elastic.co/subscriptions http://training.elastic.co