Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
990
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
860
Web servers - and how I created my own one
iafonov
8
1.1k
Chef - Infrastructure as code
iafonov
6
610
Other Decks in Programming
See All in Programming
2024 컴포즈 정원사
jisungbin
0
150
意外とフォントが大事だった話 / Font Issues on Internationalization
fumi23
0
110
全部見せます! クラシルリワードのSwiftTesting移行プロジェクト
uetyo
0
210
Rubyとクリエイティブコーディングの輪の広がり / The Growing Circle of Ruby and Creative Coding
chobishiba
1
270
開発を加速する共有Swift Package実践
elmetal
PRO
0
420
Debugging: All you need to know (for simultaneous interpreting)
jmatsu
2
920
僕が思い描くTypeScriptの未来を勝手に先取りする
yukukotani
9
2.4k
いつか使える ObjectSpace / Maybe useful ObjectSpace
euglena1215
2
140
今インフラ技術をイチから学び直すなら
yuhta28
1
140
LangChainでWebサイトの内容取得やGitHubソースコード取得
shukob
0
160
Google Sign-inの移行から始めるCredential Manager活用
clockvoid
0
410
エンジニア1年目で複雑なコードの改善に取り組んだ話
mtnmr
3
2k
Featured
See All Featured
Clear Off the Table
cherdarchuk
91
320k
Visualization
eitanlees
142
15k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.6k
Principles of Awesome APIs and How to Build Them.
keavy
125
16k
Scaling GitHub
holman
458
140k
Music & Morning Musume
bryan
46
6k
Thoughts on Productivity
jonyablonski
66
4.2k
GitHub's CSS Performance
jonrohan
1030
450k
RailsConf 2023
tenderlove
28
810
Unsuck your backbone
ammeep
667
57k
Designing with Data
zakiwarfel
98
5k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
123
18k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov