Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
170
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.8k
The Missing Pieces of Amazon ECS (for me)
takus
2
910
マイクロサービスとSRECon / #microserv
takus
14
4.5k
Slack と Incident Command System / #slackbook
takus
2
700
Why and How SmartNews uses SaaS?
takus
2
3k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
64
18k
SmartNews と Datadog
takus
6
3.9k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.7k
Other Decks in Programming
See All in Programming
テスト駆動Kaggle
isax1015
1
440
PHPでWebSocketサーバーを実装しよう2025
kubotak
0
290
AI駆動のマルチエージェントによる業務フロー自動化の設計と実践
h_okkah
0
180
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
270
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
260
ペアプロ × 生成AI 現場での実践と課題について / generative-ai-in-pair-programming
codmoninc
2
19k
ニーリーにおけるプロダクトエンジニア
nealle
0
870
VS Code Update for GitHub Copilot
74th
2
660
状態遷移図を書こう / Sequence Chart vs State Diagram
orgachem
PRO
1
130
10 Costly Database Performance Mistakes (And How To Fix Them)
andyatkinson
0
440
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
Hack Claude Code with Claude Code
choplin
5
2.3k
Featured
See All Featured
Speed Design
sergeychernyshev
32
1k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Code Reviewing Like a Champion
maltzj
524
40k
Typedesign – Prime Four
hannesfritz
42
2.7k
Music & Morning Musume
bryan
46
6.6k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Six Lessons from altMBA
skipperchong
28
3.9k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Navigating Team Friction
lara
187
15k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like