Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elastic Stackを利用して データから様々な気づきを見つける
Search
Jun Ohtani
February 07, 2017
Technology
0
1.1k
Elastic Stackを利用して データから様々な気づきを見つける
#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
Jun Ohtani
February 07, 2017
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.8k
What's new in Elastic Stack 6.3
johtani
2
2.1k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.4k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
120
Intro Elastic Stack at Telemetry WG
johtani
0
240
What's new in Elastic Stack 6.1?
johtani
0
610
Other Decks in Technology
See All in Technology
30代からでも遅くない! 内製開発の世界に飛び込み、最前線で戦うLLMアプリ開発エンジニアになろう
minorun365
PRO
15
4.7k
AWSのマルチアカウント管理 ベストプラクティス最新版 2025 / Multi-Account management on AWS best practice 2025
ohmura
4
350
Porting PicoRuby to Another Microcontroller: ESP32
yuuu
4
490
watsonx.data上のベクトル・データベース Milvusを見てみよう/20250418-milvus-dojo
mayumihirano
0
140
OpsJAWS34_CloudTrailLake_for_Organizations
hiashisan
0
180
OPENLOGI Company Profile
hr01
0
63k
バクラクの認証基盤の成長と現在地 / bakuraku-authn-platform
convto
4
790
3D生成AIのための画像生成
kosukeito
2
350
AWS全冠芸人が見た世界 ~資格取得より大切なこと~
masakiokuda
5
6.5k
OpenLane-V2ベンチマークと代表的な手法
kzykmyzw
0
120
コードや知識を組み込む / Incorporating Codes and Knowledge
ks91
PRO
0
130
2025-04-24 "Manga AI Understanding & Localization" Furukawa Arata (CyberAgent, Inc)
ornew
2
300
Featured
See All Featured
Scaling GitHub
holman
459
140k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.3k
A Modern Web Designer's Workflow
chriscoyier
693
190k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.2k
GraphQLの誤解/rethinking-graphql
sonatard
71
10k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.4k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
52
2.4k
Embracing the Ebb and Flow
colly
85
4.7k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
34
2.2k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.1k
Navigating Team Friction
lara
185
15k
Transcript
‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠
σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
‹#›
ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱʁ • Ϣʔεέʔεͷհ • Elastic stackհ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack
• σϞ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱʁ
Search and analytics, it all started here More than 60%
of our customers have a search or analytics use case
7
8
Logs Logs Logs, many devices, many systems More than
40% of our customers use our products for operational log analysis
We collect more than 1.2 TB logs every day from
our infrastructure, web servers, and applications. 10
11 We handle more than 3 Billion daily events while
meeting our all of our data security requirements.
Sniff sniff sniff, find the bad actors in your data
200% YoY growth in security use cases with our products
We analyze piles of data: 13B AMP queries/day 600B emails/day
16B web requests/day 13
14 We mine and analyze 4 billion events every day
to detect security hacks and threats. 1
The Elastic Stack: A foundation to solve many use
cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL ANALYTICS LOG ANALYSIS
Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine
16
17 Enterprise search Intranet search Real-time log analytics Legal contract
repository Trade tracking application HR recruiting application
18 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 20 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats +
Ingest
22 Logstash
Logstash in 10 seconds • ϩάɾσʔλͷऩूɾཧ • ऩूɺύʔεɾՃɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) 23
Logstash architecture 24 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 25 input { … } filter { … }
output { … }
ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
ઃఆɿfilter 28 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter 30 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }
ܰྔσʔλγούʔ 33 Beats
To tail a File filebeat
To tail a File filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
Store, Search & Analytics
41 Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ 44
ߜΓࠐΈ 45
ϋΠϥΠτ 46
ιʔτ 47
ϖʔδϯά 48
ूܭ 49
αδΣετ 50
Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ 51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 54
ྫɿݴޠ͓ΑͼҬͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
elasticsearch-hadoop 57 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
The Window into the Elastic Stack
59 KibanaͰՄࢹԽ
Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •
Apache License 2.0 • Elastic Stackͷ૭ͷׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
Kibana 5 61
None
X-Pack 5.0: Extending the Elastic Stack
Security
X-Pack : Securityͷಛ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •
Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ࠪϩά 65
Alerting
X-Pack : Alertingͷಛ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ • ݅ͷઃఆ ‒
ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
Graph
Graphͷಛ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73 σϞ Demo
ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75 March 7-9, 2017 • Pier 48 • San Francisco,
CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co