Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elastic Stackを利用して データから様々な気づきを見つける
Search
Jun Ohtani
February 07, 2017
Technology
0
1.1k
Elastic Stackを利用して データから様々な気づきを見つける
#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
Jun Ohtani
February 07, 2017
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1.1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.9k
What's new in Elastic Stack 6.3
johtani
2
2.2k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
140
Intro Elastic Stack at Telemetry WG
johtani
0
250
What's new in Elastic Stack 6.1?
johtani
0
660
Other Decks in Technology
See All in Technology
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
330
Windows で省エネ
murachiakira
0
160
履歴 on Rails: Bitemporal Data Modelで実現する履歴管理/history-on-rails-with-bitemporal-data-model
hypermkt
0
2.1k
KAGのLT会 #8 - 東京リージョンでGAしたAmazon Q in QuickSightを使って、報告用の資料を作ってみた
0air
0
200
stupid jj tricks
indirect
0
7.8k
Flaky Testへの現実解をGoのプロポーザルから考える | Go Conference 2025
upamune
1
400
非エンジニアのあなたもできる&もうやってる!コンテキストエンジニアリング
findy_eventslides
3
900
定期的な価値提供だけじゃない、スクラムが導くチームの共創化 / 20251004 Naoki Takahashi
shift_evolve
PRO
3
290
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
フルカイテン株式会社 エンジニア向け採用資料
fullkaiten
0
9k
Trust as Infrastructure
bcantrill
0
300
o11yで育てる、強い内製開発組織
_awache
3
110
Featured
See All Featured
Navigating Team Friction
lara
189
15k
The Invisible Side of Design
smashingmag
301
51k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
140
34k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Building an army of robots
kneath
306
46k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Transcript
‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠
σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
‹#›
ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱʁ • Ϣʔεέʔεͷհ • Elastic stackհ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack
• σϞ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱʁ
Search and analytics, it all started here More than 60%
of our customers have a search or analytics use case
7
8
Logs Logs Logs, many devices, many systems More than
40% of our customers use our products for operational log analysis
We collect more than 1.2 TB logs every day from
our infrastructure, web servers, and applications. 10
11 We handle more than 3 Billion daily events while
meeting our all of our data security requirements.
Sniff sniff sniff, find the bad actors in your data
200% YoY growth in security use cases with our products
We analyze piles of data: 13B AMP queries/day 600B emails/day
16B web requests/day 13
14 We mine and analyze 4 billion events every day
to detect security hacks and threats. 1
The Elastic Stack: A foundation to solve many use
cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL ANALYTICS LOG ANALYSIS
Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine
16
17 Enterprise search Intranet search Real-time log analytics Legal contract
repository Trade tracking application HR recruiting application
18 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 20 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats +
Ingest
22 Logstash
Logstash in 10 seconds • ϩάɾσʔλͷऩूɾཧ • ऩूɺύʔεɾՃɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) 23
Logstash architecture 24 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 25 input { … } filter { … }
output { … }
ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
ઃఆɿfilter 28 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter 30 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }
ܰྔσʔλγούʔ 33 Beats
To tail a File filebeat
To tail a File filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
Store, Search & Analytics
41 Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ 44
ߜΓࠐΈ 45
ϋΠϥΠτ 46
ιʔτ 47
ϖʔδϯά 48
ूܭ 49
αδΣετ 50
Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ 51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 54
ྫɿݴޠ͓ΑͼҬͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
elasticsearch-hadoop 57 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
The Window into the Elastic Stack
59 KibanaͰՄࢹԽ
Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •
Apache License 2.0 • Elastic Stackͷ૭ͷׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
Kibana 5 61
None
X-Pack 5.0: Extending the Elastic Stack
Security
X-Pack : Securityͷಛ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •
Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ࠪϩά 65
Alerting
X-Pack : Alertingͷಛ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ • ݅ͷઃఆ ‒
ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
Graph
Graphͷಛ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73 σϞ Demo
ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75 March 7-9, 2017 • Pier 48 • San Francisco,
CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co