Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - ...
Search
Jun Ohtani
July 11, 2018
0
140
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
OSC Hokkaido 2018での発表資料です。
Jun Ohtani
July 11, 2018
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1.1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.8k
What's new in Elastic Stack 6.3
johtani
2
2.2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
Intro Elastic Stack at Telemetry WG
johtani
0
250
What's new in Elastic Stack 6.1?
johtani
0
660
システムメトリクス・ログのリアルタイム解析入門 - Elastic Stackを活用して -
johtani
5
1.4k
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
61k
Mobile First: as difficult as doing things right
swwweet
224
9.9k
GraphQLとの向き合い方2022年版
quramy
49
14k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
127
53k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
Into the Great Unknown - MozCon
thekraken
40
2k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.2k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
580
For a Future-Friendly Web
brad_frost
180
9.9k
The Cost Of JavaScript in 2023
addyosmani
53
8.9k
Making Projects Easy
brettharned
117
6.4k
Transcript
!1 2018/07/06 Developer Advocate at Elastic Jun Ohtani @johtani ༷ʑͳϝτϦΫεϩάΛूΊͯγεςϜղੳ
- Elastic Stackͷೖͱ׆༻ -
!2
!3 ΞδΣϯμ • ϝτϦΫεʗϩάͱʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹͬͯΈΑ͏ • Beats - Elasticsearch
- KibanaͰղੳ • ຊ֨తʹղੳΛΔʹʁ • LogstashͰϩάϝτϦΫεΛதܧɾू • ͞Βʹ৭ʑࢼͯ͠ΈΔʹʁ
!4 about • Me, Jun Ohtani / Developer Advocate ‒
lucene-gosenίϛολʔ ‒ σʔλੳج൫ߏஙೖ ڞஶ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats Elastic APM, X-Pack, Elastic Cloud, Swiftype Professional services: Support & development subscriptions Trainings, Consulting, SaaS
!5 ͲΜͳϝτϦΫεɺ ϩάΛूΊ͍ͯ·͔͢ʁ
!6 ϝτϦΫε • CPUɺϝϞϦ༻ɺσΟεΫ༻ • ΞΫηεɺωοτϫʔΫసૹྔ • Ԡ࣌ؒ • ίωΫγϣϯ
• τϥϯβΫγϣϯɺച্ • ίϯςφͷ্ͷ֤छϝτϦΫε
!7 ϩά • ೝূϩά • γεςϜϩά • ΞϓϦέʔγϣϯϩά • Slow
log • ΞΫηεϩά • ίϯςφͷதͷϩά
!8 Ͱ͖ΕϩάͱϝτϦΫεΛ ·ͱΊͯ1ͭͷը໘Ͱ ݟ͍ͨͰ͢ΑͶʁ
!9 Elastic Stack
10 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
!11 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
ఆܕͷϝτϦΫε/ϩάղੳΛ Elastic StackͰ
!13 ϝτϦΫεɾϩάͷੳʢ؆қ൛ʣ σʔλ Import Parse/ Store/Search Visualize
!14
15 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू มͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:
ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats
The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat
Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data
Collect system and application metrics Metricbeat
lots of modules Metricbeat
tail log from file Filebeat
many modules Filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
!25 • Kubernetes module in Metricbeat ‒ CPU, memory, ωοτϫʔΫసૹྔͳͲ
• add_docker_metadataϓϩηοα ‒ Container ID, name, image, labels • add_kubernetes_metadataϓϩηοα ‒ Pod name, pod namespace, container name, pod labels Beats <3 ίϯςφ DockerKubernetesͰͷσϓϩΠΛ؆୯ʹ
!26
27 Elasticsearch Heart of the Elastic Stack ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ
։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
Elasticsearchͱʁ
!29 ϑϦʔϫʔυݕࡧ
!30 ߜΓࠐΈ
!31 ϋΠϥΠτ
!32 ιʔτ
!33 ϖʔδϯά
!34 ूܭ
!35 αδΣετ
!36 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON •
Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ
؆୯ͳCRUD
σʔλొ 38 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'
σʔλߋ৽ 39 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :
"Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'
σʔλআ !40 curl -X DELETE localhost:9200/books/doc/1 σʔλͷऔಘ curl —X GET
localhost:9200/books/doc/1 curl —X GET localhost:9200/books/doc/1/_source
ݕࡧ - Query DSL !41 curl -XGET ‘localhost:9200/books/doc/_search' -d '{
"query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'
ࢄߏɺ εέʔϧ
Basic terms • ΠϯσοΫε ‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical • ϨϓϦέʔγϣϯ •
ಡΈࠐΈͷεέʔϥϏϦςΟ্ • SPOFͷղফ • γϟʔσΟϯά • ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ !43
γϟʔυͱϨϓϦΧ !44 node 1 orders products 1 4 1 2
2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'
γϟʔυͱϨϓϦΧ !45 node 1 orders products 1 4 1 node
2 orders products 2 2 3 4 1 2 3
ࣗಈతͳࢄ !46 node 1 orders products 2 1 4 1
node 2 orders products 2 2 node 3 orders products 3 4 1 3
ͦͷଞͷػೳ
elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ ҢܦɺGeoHashɺ GeoShape… GEO
Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷՃ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,
python, php, perl, javascript, .NET • Scala, clojure, go !49
Elasticsearch - The Definitive guide http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 50 ৄ͘͠Γ͍ͨํ
!51
52 Kibana Window into the Elastic Stack ՄࢹԽͱੳ ཧۭؒ ΧελϚΠζͱ
Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stackͷ ηΩϡΞͳΞΫηεͱཧ ΧελϜAppsͷ࡞
!53 Kibana 6
!54 σϞ σʔλೖ͔ΒՄࢹԽ·Ͱ
ຊ֨తʹղੳΛߦ͏ʹʁ
!56
57 Logstash σʔλՃύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷೖ ύʔεͱಈతͳ σʔλม ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ
σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞ 200Ҏ্ͷϓϥάΠϯ
Logstash architecture !58 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 59 input { … } filter { … }
output { … }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 60
ઃఆɿfilter 61 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε !62 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter !63 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ͷύʔε 64 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …
} {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }
ઃఆɿfilter !65 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 66 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿfilter !67 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ϢʔβΤʔδΣϯτͷύʔε 68 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101
Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"
ͦͷ΄͔ʹʁ
!70 elasticsearch-hadoop - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
!71
!72 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
!73
!74 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/
index.html • ॻ੶ʢຊޠʣ ‒ σʔλੳج൫ߏஙೖ ‒ Elasticsearch࣮ફΨΠυ
!75 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co
• Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co