Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introducing Machine Learning for the Elastic Stack
Search
Kosho Owa
May 19, 2017
Technology
2
12k
Introducing Machine Learning for the Elastic Stack
Elastic Machine Learning Seminar held on May 19th, 2017
Kosho Owa
May 19, 2017
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
290
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
300
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
670
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
88
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
150
Introducing Elastic Cloud
kosho
0
64
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
130
Elastic Stack Hands-on Workshop (EN)
kosho
1
150
Other Decks in Technology
See All in Technology
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
TanStack Routerに移行するのかい しないのかい、どっちなんだい! / Are you going to migrate to TanStack Router or not? Which one is it?
kaminashi
0
600
Amazon CloudWatch Network Monitor のススメ
yuki_ink
1
210
個人でもIAM Identity Centerを使おう!(アクセス管理編)
ryder472
4
230
心が動くエンジニアリング ── 私が夢中になる理由
16bitidol
0
100
100 名超が参加した日経グループ横断の競技型 AWS 学習イベント「Nikkei Group AWS GameDay」の紹介/mediajaws202411
nikkei_engineer_recruiting
1
170
AWS Media Services 最新サービスアップデート 2024
eijikominami
0
200
インフラとバックエンドとフロントエンドをくまなく調べて遅いアプリを早くした件
tubone24
1
430
Adopting Jetpack Compose in Your Existing Project - GDG DevFest Bangkok 2024
akexorcist
0
110
『Firebase Dynamic Links終了に備える』 FlutterアプリでのAdjust導入とDeeplink最適化
techiro
0
130
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
3.9k
マルチプロダクトな開発組織で 「開発生産性」に向き合うために試みたこと / Improving Multi-Product Dev Productivity
sugamasao
1
310
Featured
See All Featured
Designing for Performance
lara
604
68k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.9k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Statistics for Hackers
jakevdp
796
220k
Documentation Writing (for coders)
carmenintech
65
4.4k
Unsuck your backbone
ammeep
668
57k
4 Signs Your Business is Dying
shpigford
180
21k
Building Better People: How to give real-time feedback that sticks.
wjessup
364
19k
A better future with KSS
kneath
238
17k
Raft: Consensus for Rubyists
vanstee
136
6.6k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Transcript
Machine Learning for the Elastic Stack Beta in 5.4.
GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳར༻Մೳ Available in AWS
today
5 Elastic Cloud Enterprise ෳͷElastic StackڥΛࣗࡏʹ࡞ Logging as a serviceΛࣗ৫ʹల։
Public beta; Expected GA Q1 2017
ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics
Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
Operational Analytics • ΣϒαΠτͷΞΫηετϥϑΟοΫʹҟৗແ͍͔? • Ϙοτ߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩάରॲ͢Δඞཁ͕ ͋Δͷ͔? Use
Case
Security Analytics • ϚϧΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ෦ऀʹΑΔηΩϡϦςΟڴҖແ͍͔? • DNSͷϩάʹɺσʔλऔͷ͕ࠟͳ͍͔? Use Case
Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ? ▪ ଞͱҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ? ▪ ಛҟͳΠϕϯτλΠϓηϯαʔͷނোΛ͔ࣔ͢?
Use Case
10 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • σʔλෳࡶɺߴ࣍ݩɺߴʹมԽ • ਓؒͷࢹೝݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection
is not practical Where’s the anomaly?
11 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • ੩తͳᮢʹΑΔʮਖ਼ৗʯͷఆٛࠔ • ϧʔϧσʔλΠϯϑϥͷมߋʹैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts
are insufficient What’s the right threshold ?
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗͷൣғ͔Βҳͨ͠ࡍʹҟৗͱͯ͠ݕ
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗͷೖྗ͕ෆཁ • σʔλͷมԽʹै - ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽
• ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ
ҟͳΔछྨͷҟৗΛݕ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple
• ͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏϝοηʔδ Rare / unusual rates in “categories” of events
࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:
Is there unusual traffic on website ?
࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple
time series ▪ ෳͷϝτϦοΫ ▪ FieldʹΑͬͯྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
ك༗ͳඇߏϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •
ྨࣅੑʹج͍ͮͯΧςΰϦ͚ • ࣌ؒมԽʹΑΔසΛֶश • ϞσϧͱҟͳΕҟৗͱͯ͠ݕ Example: • Do my application logs contain unusual messages
X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack
$ kibana-plugin install x-pack
σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
֎෦γεςϜͱͷଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε
(.ml-anomalies-*)
Taking Action with X-Pack Alerting 24
Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in
Population: Web Server Log Rare Messages: DBMS Server Log 25
26 4JOHMF.FUSJD
27 .VMUJ.FUSJD
28 .VMUJ.FUSJD
29 0VUMJFSTJO1PQVMBUJPO
30 0VUMJFSTJO1PQVMBUJPO
31 3BSF.FTTBHFT
32 3BSF.FTTBHFT
࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ
• αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗͷσʔλΛೖ • MLδϣϒΛ࡞ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯڥ) • MLδϣϒΛ࡞ (Ϩγϐ׆༻) • AlertingͰΞΫγϣϯ