Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elastic Stackを利用して データから様々な気づきを見つける
Search
Jun Ohtani
February 07, 2017
Technology
0
1.1k
Elastic Stackを利用して データから様々な気づきを見つける
#BigDataTokyo BigData Analytics Tokyoでの発表スライドです。
Jun Ohtani
February 07, 2017
Tweet
Share
More Decks by Jun Ohtani
See All by Jun Ohtani
Elastic Stackでマイクロサービス運用を 楽にするには? / Monitoring Microservices with Elastic Stack
johtani
5
2.7k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Getting started Elastic Stack for logging/metrics
johtani
4
1k
え?SQLで入門?する ElasticsearchとElastic Stack / Getting started Elastic Stack with SQL
johtani
4
940
Elastic Stack 入門 2018.09 / Getting started Elastic Stack 2018.09
johtani
3
2.7k
What's new in Elastic Stack 6.3
johtani
2
2.1k
Elastic Stackで始めるJavaアプリのパフォーマンス監視 / Intro Elastic Stack and Elastic APM Java
johtani
5
2.4k
様々なメトリクスやログを集めてシステム解析 - Elastic Stackの入門と活用 - / Intro Elastic Stack
johtani
0
110
Intro Elastic Stack at Telemetry WG
johtani
0
220
What's new in Elastic Stack 6.1?
johtani
0
560
Other Decks in Technology
See All in Technology
データプロダクトの定義からはじめる、データコントラクト駆動なデータ基盤
chanyou0311
2
330
Adopting Jetpack Compose in Your Existing Project - GDG DevFest Bangkok 2024
akexorcist
0
110
SRE×AIOpsを始めよう!GuardDutyによるお手軽脅威検出
amixedcolor
0
170
AIチャットボット開発への生成AI活用
ryomrt
0
170
SREが投資するAIOps ~ペアーズにおけるLLM for Developerへの取り組み~
takumiogawa
1
430
CysharpのOSS群から見るModern C#の現在地
neuecc
2
3.5k
オープンソースAIとは何か? --「オープンソースAIの定義 v1.0」詳細解説
shujisado
9
1.1k
TypeScript、上達の瞬間
sadnessojisan
46
13k
IBC 2024 動画技術関連レポート / IBC 2024 Report
cyberagentdevelopers
PRO
1
110
Introduction to Works of ML Engineer in LY Corporation
lycorp_recruit_jp
0
140
Oracle Cloud Infrastructureデータベース・クラウド:各バージョンのサポート期間
oracle4engineer
PRO
28
13k
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.3k
Testing 201, or: Great Expectations
jmmastey
38
7.1k
Why Our Code Smells
bkeepers
PRO
334
57k
Measuring & Analyzing Core Web Vitals
bluesmoon
4
130
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Music & Morning Musume
bryan
46
6.2k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
229
52k
Six Lessons from altMBA
skipperchong
27
3.5k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
840
Keith and Marios Guide to Fast Websites
keithpitt
409
22k
How to Ace a Technical Interview
jacobian
276
23k
Making the Leap to Tech Lead
cromwellryan
133
8.9k
Transcript
‹#› 2017/02/07 Evangelist at Elastic Jun Ohtani @johtani Elastic StackΛར༻ͯ͠
σʔλ͔Β༷ʑͳؾ͖ͮΛݟ͚ͭΔ
‹#›
ΞδΣϯμ • ؾ͖ͮΛݟ͚ͭΔͱʁ • Ϣʔεέʔεͷհ • Elastic stackհ • BeatsɺLogstashɺElasticsearchɺKibanaɺX-Pack
• σϞ 3
about • Me, Jun Ohtani / Technical Advocate ‒ lucene-gosenίϛολʔ
‒ ElasticSearch Serverຊޠ൛ͷ༁ ‒ http://blog.johtani.info • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats X-Pack, Elastic Cloud Professional services: Support & development subscriptions ‒ Trainings & Consulting 4
༷ʑͳϢʔεέʔε 5 ؾ͖ͮΛݟ͚ͭΔ ͱʁ
Search and analytics, it all started here More than 60%
of our customers have a search or analytics use case
7
8
Logs Logs Logs, many devices, many systems More than
40% of our customers use our products for operational log analysis
We collect more than 1.2 TB logs every day from
our infrastructure, web servers, and applications. 10
11 We handle more than 3 Billion daily events while
meeting our all of our data security requirements.
Sniff sniff sniff, find the bad actors in your data
200% YoY growth in security use cases with our products
We analyze piles of data: 13B AMP queries/day 600B emails/day
16B web requests/day 13
14 We mine and analyze 4 billion events every day
to detect security hacks and threats. 1
The Elastic Stack: A foundation to solve many use
cases 75% of our customers use our products for more than one use case SEARCH SECURIT CUSTOM APPS METRICS OPERATIONAL ANALYTICS LOG ANALYSIS
Operational analytics Flight telemetry analysis Anomaly resolution Internal search engine
16
17 Enterprise search Intranet search Real-time log analytics Legal contract
repository Trade tracking application HR recruiting application
18 ElasticελοΫ
ElasticελοΫʢOpen Sourceʣ 19 Kibana Elasticsearch
Logstash Beats
ElasticελοΫ 20 Elastic Cloud
X-Pack Kibana Elasticsearch ! " Logstash Beats +
Ingest
22 Logstash
Logstash in 10 seconds • ϩάɾσʔλͷऩूɾཧ • ऩूɺύʔεɾՃɺૹग़ • ΦʔϓϯιʔεɿApache
License 2.0 • Ruby app (JRuby) 23
Logstash architecture 24 Input Output Filter ? ? collect and
split alter and enrich store and visualize
ઃఆ 25 input { … } filter { … }
output { … }
ઃఆɿinput 26 input { file { path => “/Users/johtani/sample/*_log" start_position
=> "beginning" } }
1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 27
ઃఆɿfilter 28 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
ύʔε 29 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter 30 filter { grok { match => { "message"
=> "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }
IP͔ΒҢܦͳͲ༩ 31 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,
… "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":
ઃఆɿoutput 32 output { elasticsearch { hosts => ["localhost"] index
=> “demo_access_log-%{+YYYY.MM.dd}” } }
ܰྔσʔλγούʔ 33 Beats
To tail a File filebeat
To tail a File filebeat
Capture the Packet Packetbeat
Capture the Packet Packetbeat
Welcome to 1998 winlogbeat
Now winlogbeat
Store, Search & Analytics
41 Elasticsearch
ݕࡧͱͯ͠ͷ Elasticsearch
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ 44
ߜΓࠐΈ 45
ϋΠϥΠτ 46
ιʔτ 47
ϖʔδϯά 48
ूܭ 49
αδΣετ 50
Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε:
Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ 51
ղੳͱͯ͠ͷ Elasticsearch
aggregation
Aggregationͱ • 1.0͔Βಋೖ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ • ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ
• BucketɹυΩϡϝϯτΛ͝ͱʹ݁ՌΛάϧʔϐϯά • Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ 54
ྫɿݴޠ͓ΑͼҬͷूܭ 55 curl -XGET twitter-2014.08.22/_search -d ' { "aggs": {
"lang": { "terms": {"field": "lang" }, "aggs": { "place": { "terms": { "field": “place.full_name", "size": 10 } } } } } }
ྫɿݴޠ͓ΑͼҬͷूܭ 56 "aggregations": { "lang": { "buckets": [{…}, { "key":
"ja", "doc_count": 980145, "place": { "buckets": [ { "key": "ژࢢ෬ݟ۠, ژ", "doc_count":252 }, { "key": "ઍా۠, ౦ژ", "doc_count": 39 },…
elasticsearch-hadoop 57 - • D E H • PD ecd
ER • g D • CH • Ca M DMS D FERC
The Window into the Elastic Stack
59 KibanaͰՄࢹԽ
Kibana 5 • ElasticsearchͷσʔλΛՄࢹԽ • Node.js server & JavaScript •
Apache License 2.0 • Elastic Stackͷ૭ͷׂ • ༷ʑͳGUIΛPluginͱ͍ͯެ։ • MarvelɺSenseɺTimelionͳͲ 60
Kibana 5 61
None
X-Pack 5.0: Extending the Elastic Stack
Security
X-Pack : Securityͷಛ • User Authentication ‒ LDAP/Active Directory/ϑΝΠϧϕʔε •
Authorization ‒ ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ ‒ ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ ‒ υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆՄೳʹ • ηΩϡΞͳ௨৴ ‒ ElasticsearchϊʔυؒͷSSL/TLSɺIPϑΟϧλϦϯά • ࠪϩά 65
Alerting
X-Pack : Alertingͷಛ • ΫΤϦʹΑΔWatch ‒ ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷࢹ • ݅ͷઃఆ ‒
ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ • εέδϡʔϧ ‒ ΫΤϦΛ࣮ߦ͠ɺ݅ΛνΣοΫ͢Δසͷࢦఆ • ΞΫγϣϯͷఆٛ ‒ ϝʔϧͷૹ৴ɺଞγεςϜͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ • ཤྺͷอଘ 67
Graph
Graphͷಛ • σʔλؒͷͭͳ͕ΓΛ୳ࡧ͢ΔϓϥάΠϯ • KibanaϓϥάΠϯʹΑΓGUIΛར༻ͯ͠୳ࡧՄೳ 69
Prelert
σʔλ͔Β༗ҙٛͳใΛݟ͚ͭΔํ๏ Search Aggregations Visualization Machine Learning
1SFMFSUͷςΫϊϩδʔ σʔλʹજΉߦಈϞσϧΛ ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ ݦஶʹҟͳΔ߹ʹ௨
73 σϞ Demo
ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co •
Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions 74
75 March 7-9, 2017 • Pier 48 • San Francisco,
CA • 2,500 attendees 3rd Annual Elastic User Conference Topics • Latest Roadmap • Ask Me Anything Booth • 70+ Sessions • 76 Demo Hours
ΞϯέʔτͷճΛ͓ئ͍͠·͢ bit.ly/bigdata-tokyo-elastic
Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re
helping! https://www.elastic.co/subscriptions http://training.elastic.co