Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introduction of Skyline
Search
Takumi Sakamoto
July 12, 2013
Programming
0
170
Introduction of Skyline
Takumi Sakamoto
July 12, 2013
Tweet
Share
More Decks by Takumi Sakamoto
See All by Takumi Sakamoto
Building Data Pipelines with Apache Airflow
takus
3
2.8k
The Missing Pieces of Amazon ECS (for me)
takus
2
890
マイクロサービスとSRECon / #microserv
takus
14
4.5k
Slack と Incident Command System / #slackbook
takus
2
680
Why and How SmartNews uses SaaS?
takus
2
2.9k
スマートニュースの世界進出を支えるログ解析基盤 #jawsdays #tech
takus
64
18k
SmartNews と Datadog
takus
6
3.9k
ぜんぶ AWS でやらないワケ
takus
55
110k
Velocity Conference 2013 Report
takus
2
1.7k
Other Decks in Programming
See All in Programming
Modern Angular:Renovation for Your Applications @angularDays 2025 Munich
manfredsteyer
PRO
0
140
NestJSのコードからOpenAPIを自動生成する際の最適解を探す
astatsuya
0
190
Day0 初心者向けワークショップ実践!ソフトウェアテストの第一歩
satohiroyuki
0
410
いまさら聞けない生成AI入門: 「生成AIを高速キャッチアップ」
soh9834
12
3.7k
PHPer's Guide to Daemon Crafting Taming and Summoning
uzulla
2
1.1k
Devinのメモリ活用の学びを自社サービスにどう組み込むか?
itarutomy
0
1.7k
新卒から4年間、20年もののWebサービスと 向き合って学んだソフトウェア考古学
oguri
8
6.7k
ベクトル検索システムの気持ち
monochromegane
30
8.9k
自分のために作ったアプリが、グローバルに使われるまで / Indie App Development Lunch LT
pixyzehn
1
120
Coding Experience Cpp vs Csharp - meetup app osaka@9
harukasao
0
110
ミリしらMCP勉強会
watany
2
340
SideKiqでジョブが二重起動した事象を深堀りしました
t_hatachi
0
230
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
Six Lessons from altMBA
skipperchong
27
3.7k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.4k
Writing Fast Ruby
sferik
628
61k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
51
2.4k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Unsuck your backbone
ammeep
670
57k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.2k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
227
22k
Transcript
Introduction of Skyline @takus monitoring casual #4 2013.07.12
Velocity 2013 • Web Performance & Operations Conference • Great
talks • Great people • Great culture
My LT Talk at Velocity • 3 Popular Ops Tools
in Japan • serverspec, growthforecast, fluentd • Uploaded on youtube :-( • http://www.youtube.com/watch? v=bRYuBQyG5Sw
Some Positive Feedbacks
The Most Interesting Talk • Avoiding Performance Regression at Twitter
• fight against perf regression in an automated fashion • http://ameblo.jp/principia-ca/ entry-11561132297.html
Today’s Topic
Background • Etsy deploys their app 30+ times per day
• Optimize for quick recovery by anticipating problems, instead of fearing human error • Can’t fix what you don’t measure! • If it moves, graph it
Too Many Graphs • 250,000+ dashboards • If a graph
spikes and no one is watching, does it really spike? • There are things we do not know we don’t know.
Kale • Skyline • Detect unknown anomalies • Oculus •
Detect unknown correlations http://codeascraft.com/2013/06/11/introducing-kale/
None
Architecture Horizon Horizon Horizon Redis Analyzer Manager Analyzer Analyzer Analyzer
Assign Metrics Keys Get Metrics Keys Fetch Timeseries Metrics
Horizon • Listeners • Receiving metrics & store them to
queue • Workers • Inserting metrics to redis with Messagepack • Roombas • Purging metrics in Redis at a regular interval
Analyzer • Assigning Redis keys to each process process •
decode from Messagepack • run the detection algorithm
How to Detect Anomalies? • Consensus model • If the
majority of algorithms agree, the metric will be classified as anomalous • Use your own algorithm for each application • as long as you return a boolean, you can add any sort of algorithm you like
Basic Algorithm • A metric is anomalous if its latest
datapoint is over three standard deviations above its moving average
Anomaly?
Conclusion • Etsy monitors 250,000+ graphs • If a graph
spikes and no one is watching, does it really spike? • Skyline • detects unknown anomalies • consensus with any algorithms you like