Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introducing Machine Learning for the Elastic Stack
Search
Kosho Owa
May 19, 2017
Technology
2
12k
Introducing Machine Learning for the Elastic Stack
Elastic Machine Learning Seminar held on May 19th, 2017
Kosho Owa
May 19, 2017
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
290
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
300
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
670
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
88
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
150
Introducing Elastic Cloud
kosho
0
64
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
130
Elastic Stack Hands-on Workshop (EN)
kosho
1
150
Other Decks in Technology
See All in Technology
現地でMeet Upをやる場合の注意点〜反省点を添えて〜
shotashiratori
0
520
신뢰할 수 있는 AI 검색 엔진을 만들기 위한 Liner의 여정
huffon
0
340
10分でわかるfreeeのQA
freee
1
3.4k
pandasはPolarsに性能面で追いつき追い越せるのか
vaaaaanquish
4
4.6k
ネット広告に未来はあるか?「3rd Party Cookie廃止とPrivacy Sandboxの効果検証の裏側」 / third-party-cookie-privacy
cyberagentdevelopers
PRO
1
130
MAMを軸とした動画ハンドリングにおけるAI活用前提の整備と次世代ビジョン / abema-ai-mam
cyberagentdevelopers
PRO
1
110
CAMERA-Suite: 広告文生成のための評価スイート / ai-camera-suite
cyberagentdevelopers
PRO
3
270
生成AIとAWS CDKで実現! 自社ブログレビューの効率化
ymae
2
330
急成長中のWINTICKETにおける品質と開発スピードと向き合ったQA戦略と今後の展望 / winticket-autify
cyberagentdevelopers
PRO
1
160
WINTICKETアプリで実現した高可用性と高速リリースを支えるエコシステム / winticket-eco-system
cyberagentdevelopers
PRO
1
190
Automated Promptingを目指すその前に / Before we can aim for Automated Prompting
rkaga
0
110
Gradle: The Build System That Loves To Hate You
aurimas
2
150
Featured
See All Featured
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
107
49k
Producing Creativity
orderedlist
PRO
341
39k
Code Review Best Practice
trishagee
64
17k
Build The Right Thing And Hit Your Dates
maggiecrowley
32
2.4k
Fireside Chat
paigeccino
32
3k
Designing for humans not robots
tammielis
249
25k
The Invisible Side of Design
smashingmag
297
50k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
Transcript
Machine Learning for the Elastic Stack Beta in 5.4.
GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳར༻Մೳ Available in AWS
today
5 Elastic Cloud Enterprise ෳͷElastic StackڥΛࣗࡏʹ࡞ Logging as a serviceΛࣗ৫ʹల։
Public beta; Expected GA Q1 2017
ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics
Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
Operational Analytics • ΣϒαΠτͷΞΫηετϥϑΟοΫʹҟৗແ͍͔? • Ϙοτ߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩάରॲ͢Δඞཁ͕ ͋Δͷ͔? Use
Case
Security Analytics • ϚϧΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ෦ऀʹΑΔηΩϡϦςΟڴҖແ͍͔? • DNSͷϩάʹɺσʔλऔͷ͕ࠟͳ͍͔? Use Case
Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ? ▪ ଞͱҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ? ▪ ಛҟͳΠϕϯτλΠϓηϯαʔͷނোΛ͔ࣔ͢?
Use Case
10 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • σʔλෳࡶɺߴ࣍ݩɺߴʹมԽ • ਓؒͷࢹೝݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection
is not practical Where’s the anomaly?
11 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • ੩తͳᮢʹΑΔʮਖ਼ৗʯͷఆٛࠔ • ϧʔϧσʔλΠϯϑϥͷมߋʹैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts
are insufficient What’s the right threshold ?
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗͷൣғ͔Βҳͨ͠ࡍʹҟৗͱͯ͠ݕ
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗͷೖྗ͕ෆཁ • σʔλͷมԽʹै - ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽
• ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ
ҟͳΔछྨͷҟৗΛݕ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple
• ͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏϝοηʔδ Rare / unusual rates in “categories” of events
࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:
Is there unusual traffic on website ?
࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple
time series ▪ ෳͷϝτϦοΫ ▪ FieldʹΑͬͯྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
ك༗ͳඇߏϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •
ྨࣅੑʹج͍ͮͯΧςΰϦ͚ • ࣌ؒมԽʹΑΔසΛֶश • ϞσϧͱҟͳΕҟৗͱͯ͠ݕ Example: • Do my application logs contain unusual messages
X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack
$ kibana-plugin install x-pack
σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
֎෦γεςϜͱͷଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε
(.ml-anomalies-*)
Taking Action with X-Pack Alerting 24
Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in
Population: Web Server Log Rare Messages: DBMS Server Log 25
26 4JOHMF.FUSJD
27 .VMUJ.FUSJD
28 .VMUJ.FUSJD
29 0VUMJFSTJO1PQVMBUJPO
30 0VUMJFSTJO1PQVMBUJPO
31 3BSF.FTTBHFT
32 3BSF.FTTBHFT
࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ
• αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗͷσʔλΛೖ • MLδϣϒΛ࡞ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯڥ) • MLδϣϒΛ࡞ (Ϩγϐ׆༻) • AlertingͰΞΫγϣϯ