Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elastic Stackを利用して データから様々な気づきを見つける
Search
Jun Ohtani
February 07, 2017
Technology
0
1.1k
Elastic Stackを利用して データから様々な気づきを見つける
#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
Jun Ohtani
February 07, 2017
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.9k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1.1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
1k
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.8k
What's new in Elastic Stack 6.3
johtani
2
2.1k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.5k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
130
Intro Elastic Stack at Telemetry WG
johtani
0
240
What's new in Elastic Stack 6.1?
johtani
0
620
Other Decks in Technology
See All in Technology
newmo の創業を支える Software Architecture と Platform Engineering
110y
5
790
VueUseから学ぶ実践TypeScript #TSKaigi #TSKaigi2025
bengo4com
3
370
勘違いから始まったProxmox on ProxmoxでGPUパススルー【JPmoxs勉強会#7】/JPmoxs7_GPU_Passthrough_on_Proxmox_on_Proxmox-A_Journey_That_Started_with_a_Misunderstanding
tsukimi_site
1
130
え!! 日本国内でGo言語のバイリンガル勉強会を!?
logica0419
2
110
撤退危機からのピボット : 4年目エンジニアがリードする TypeScript で挑む事業復活 / crisis-to-pivot-4th-year-engineer-ts-relaunch
carta_engineering
2
320
MagicPod MCPサーバー開発の裏側とAIエージェント活用の展望
magicpod
0
330
スキーマと型で拓く Full-Stack TypeScript / TSKaigi 2025
altech
2
1.6k
AI Development in .NET - Microsoft.Extensions.AI
_mertmetin
0
120
GitHub ActionsをTypeScriptで作ろう!
sansantech
PRO
2
190
生成AI時代における人間の情熱とプロダクト志向 / 20250517 Takuya Oikawa
shift_evolve
2
440
チェックツールを導入したけど使ってもらえなかった話 #GAADjp
lycorptech_jp
PRO
1
160
テスト設計チュートリアル ちびこん編 ’25
omn
1
240
Featured
See All Featured
Adopting Sorbet at Scale
ufuk
76
9.4k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.3k
Fontdeck: Realign not Redesign
paulrobertlloyd
84
5.5k
A designer walks into a library…
pauljervisheath
205
24k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Visualization
eitanlees
146
16k
Optimising Largest Contentful Paint
csswizardry
37
3.2k
Statistics for Hackers
jakevdp
799
220k
The Language of Interfaces
destraynor
158
25k
How to train your dragon (web standard)
notwaldorf
91
6k
Optimizing for Happiness
mojombo
378
70k
Transcript
‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠
σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
‹#›
ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱʁ • Ϣʔεέʔεͷհ • Elastic stackհ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack
• σϞ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱʁ
Search and analytics, it all started here More than 60%
of our customers have a search or analytics use case
7
8
Logs Logs Logs, many devices, many systems More than
40% of our customers use our products for operational log analysis
We collect more than 1.2 TB logs every day from
our infrastructure, web servers, and applications. 10
11 We handle more than 3 Billion daily events while
meeting our all of our data security requirements.
Sniff sniff sniff, find the bad actors in your data
200% YoY growth in security use cases with our products
We analyze piles of data: 13B AMP queries/day 600B emails/day
16B web requests/day 13
14 We mine and analyze 4 billion events every day
to detect security hacks and threats. 1
The Elastic Stack: A foundation to solve many use
cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL ANALYTICS LOG ANALYSIS
Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine
16
17 Enterprise search Intranet search Real-time log analytics Legal contract
repository Trade tracking application HR recruiting application
18 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 20 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats +
Ingest
22 Logstash
Logstash in 10 seconds • ϩάɾσʔλͷऩूɾཧ • ऩूɺύʔεɾՃɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) 23
Logstash architecture 24 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 25 input { … } filter { … }
output { … }
ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
ઃఆɿfilter 28 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter 30 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }
ܰྔσʔλγούʔ 33 Beats
To tail a File filebeat
To tail a File filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
Store, Search & Analytics
41 Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ 44
ߜΓࠐΈ 45
ϋΠϥΠτ 46
ιʔτ 47
ϖʔδϯά 48
ूܭ 49
αδΣετ 50
Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ 51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 54
ྫɿݴޠ͓ΑͼҬͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
elasticsearch-hadoop 57 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
The Window into the Elastic Stack
59 KibanaͰՄࢹԽ
Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •
Apache License 2.0 • Elastic Stackͷ૭ͷׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
Kibana 5 61
None
X-Pack 5.0: Extending the Elastic Stack
Security
X-Pack : Securityͷಛ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •
Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ࠪϩά 65
Alerting
X-Pack : Alertingͷಛ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ • ݅ͷઃఆ ‒
ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
Graph
Graphͷಛ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73 σϞ Demo
ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75 March 7-9, 2017 • Pier 48 • San Francisco,
CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co