Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
890
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
630
Other Decks in Programming
See All in Programming
NSOutlineView何もわからん:( 前編 / I Don't Understand About NSOutlineView :( Pt. 1
usagimaru
0
340
CSC509 Lecture 12
javiergs
PRO
0
160
[Do iOS '24] Ship your app on a Friday...and enjoy your weekend!
polpielladev
0
110
카카오페이는 어떻게 수천만 결제를 처리할까? 우아한 결제 분산락 노하우
kakao
PRO
0
110
Compose 1.7のTextFieldはPOBox Plusで日本語変換できない
tomoya0x00
0
200
みんなでプロポーザルを書いてみた
yuriko1211
0
280
どうして僕の作ったクラスが手続き型と言われなきゃいけないんですか
akikogoto
1
120
聞き手から登壇者へ: RubyKaigi2024 LTでの初挑戦が 教えてくれた、可能性の星
mikik0
1
140
watsonx.ai Dojo #4 生成AIを使ったアプリ開発、応用編
oniak3ibm
PRO
1
180
Flutterを言い訳にしない!アプリの使い心地改善テクニック5選🔥
kno3a87
1
210
Jakarta EE meets AI
ivargrimstad
0
190
3 Effective Rules for Using Signals in Angular
manfredsteyer
PRO
1
100
Featured
See All Featured
Visualization
eitanlees
145
15k
Designing on Purpose - Digital PM Summit 2013
jponch
115
7k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
110
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.3k
The World Runs on Bad Software
bkeepers
PRO
65
11k
The Pragmatic Product Professional
lauravandoore
31
6.3k
A Philosophy of Restraint
colly
203
16k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
Making the Leap to Tech Lead
cromwellryan
133
8.9k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.4k
Done Done
chrislema
181
16k
The Invisible Side of Design
smashingmag
298
50k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov