Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
180
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.8k
The Missing Pieces of Amazon ECS (for me)
takus
2
920
マイクロサービスとSRECon / #microserv
takus
14
4.5k
Slack と Incident Command System / #slackbook
takus
2
710
Why and How SmartNews uses SaaS?
takus
2
3k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
64
18k
SmartNews と Datadog
takus
6
3.9k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.7k
Other Decks in Programming
See All in Programming
Let's Write a Train Tracking Algorithm
twocentstudios
0
220
ИИ-Агенты в каждый дом – Алексей Порядин, PythoNN
sobolevn
0
150
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
3k
Go言語の特性を活かした公式MCP SDKの設計
hond0413
1
180
CI_CD「健康診断」のススメ。現場でのボトルネック特定から、健康診断を通じた組織的な改善手法
teamlab
PRO
0
180
詳しくない分野でのVibe Codingで困ったことと学び/vibe-coding-in-unfamiliar-area
shibayu36
3
4.4k
アメ車でサンノゼを走ってきたよ!
s_shimotori
0
140
Web フロントエンドエンジニアに開かれる AI Agent プロダクト開発 - Vercel AI SDK を観察して AI Agent と仲良くなろう! #FEC余熱NIGHT
izumin5210
3
390
AI Coding Meetup #3 - 導入セッション / ai-coding-meetup-3
izumin5210
0
580
XP, Testing and ninja testing ZOZ5
m_seki
2
280
CSC305 Lecture 03
javiergs
PRO
0
230
iOSエンジニア向けの英語学習アプリを作る!
yukawashouhei
0
180
Featured
See All Featured
Side Projects
sachag
455
43k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Visualization
eitanlees
148
16k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
GraphQLとの向き合い方2022年版
quramy
49
14k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.7k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
A designer walks into a library…
pauljervisheath
209
24k
Bash Introduction
62gerente
615
210k
Unsuck your backbone
ammeep
671
58k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like