Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elastic Stackを利用して データから様々な気づきを見つける
Search
Jun Ohtani
February 07, 2017
Technology
0
1.1k
Elastic Stackを利用して データから様々な気づきを見つける
#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
Jun Ohtani
February 07, 2017
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.7k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
930
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.7k
What's new in Elastic Stack 6.3
johtani
2
2.1k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.3k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
110
Intro Elastic Stack at Telemetry WG
johtani
0
220
What's new in Elastic Stack 6.1?
johtani
0
560
Other Decks in Technology
See All in Technology
フルカイテン株式会社 採用資料
fullkaiten
0
36k
生成AIの強みと弱みを理解して、生成AIがもたらすパワーをプロダクトの価値へ繋げるために実践したこと / advance-ai-generating
cyberagentdevelopers
PRO
1
180
初心者に Vue.js を 教えるには
tsukuha
5
390
ユーザーの購買行動モデリングとその分析 / dsc-purchase-analysis
cyberagentdevelopers
PRO
2
100
10分でわかるfreee エンジニア向け会社説明資料
freee
18
520k
30万人が利用するチャットをFirebase Realtime DatabaseからActionCableへ移行する方法
ryosk7
5
350
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
49k
【技術書典17】OpenFOAM(自宅で極める流体解析)2次元円柱まわりの流れ
kamakiri1225
0
210
生成AIと知識グラフの相互利用に基づく文書解析
koujikozaki
1
140
CyberAgent 生成AI Deep Dive with Amazon Web Services / genai-aws
cyberagentdevelopers
PRO
1
480
Figma Dev Modeで進化するデザインとエンジニアリングの協働 / figma-with-engineering
cyberagentdevelopers
PRO
1
430
Shift-from-React-to-Vue
calm1205
3
1.3k
Featured
See All Featured
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
22k
Practical Orchestrator
shlominoach
186
10k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
328
21k
KATA
mclloyd
29
13k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2.1k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
504
140k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.6k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
The Pragmatic Product Professional
lauravandoore
31
6.3k
Transcript
‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠
σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
‹#›
ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱʁ • Ϣʔεέʔεͷհ • Elastic stackհ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack
• σϞ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱʁ
Search and analytics, it all started here More than 60%
of our customers have a search or analytics use case
7
8
Logs Logs Logs, many devices, many systems More than
40% of our customers use our products for operational log analysis
We collect more than 1.2 TB logs every day from
our infrastructure, web servers, and applications. 10
11 We handle more than 3 Billion daily events while
meeting our all of our data security requirements.
Sniff sniff sniff, find the bad actors in your data
200% YoY growth in security use cases with our products
We analyze piles of data: 13B AMP queries/day 600B emails/day
16B web requests/day 13
14 We mine and analyze 4 billion events every day
to detect security hacks and threats. 1
The Elastic Stack: A foundation to solve many use
cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL ANALYTICS LOG ANALYSIS
Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine
16
17 Enterprise search Intranet search Real-time log analytics Legal contract
repository Trade tracking application HR recruiting application
18 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 20 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats +
Ingest
22 Logstash
Logstash in 10 seconds • ϩάɾσʔλͷऩूɾཧ • ऩूɺύʔεɾՃɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) 23
Logstash architecture 24 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 25 input { … } filter { … }
output { … }
ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
ઃఆɿfilter 28 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter 30 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }
ܰྔσʔλγούʔ 33 Beats
To tail a File filebeat
To tail a File filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
Store, Search & Analytics
41 Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ 44
ߜΓࠐΈ 45
ϋΠϥΠτ 46
ιʔτ 47
ϖʔδϯά 48
ूܭ 49
αδΣετ 50
Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ 51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 54
ྫɿݴޠ͓ΑͼҬͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
elasticsearch-hadoop 57 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
The Window into the Elastic Stack
59 KibanaͰՄࢹԽ
Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •
Apache License 2.0 • Elastic Stackͷ૭ͷׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
Kibana 5 61
None
X-Pack 5.0: Extending the Elastic Stack
Security
X-Pack : Securityͷಛ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •
Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ࠪϩά 65
Alerting
X-Pack : Alertingͷಛ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ • ݅ͷઃఆ ‒
ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
Graph
Graphͷಛ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73 σϞ Demo
ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75 March 7-9, 2017 • Pier 48 • San Francisco,
CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co