Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Real-time application monitoring
Search
Igor Afonov
June 19, 2012
Programming
5
1k
Real-time application monitoring
Real-time application monitoring. Coffee and code Donetsk June 2012.
Igor Afonov
June 19, 2012
Tweet
Share
More Decks by Igor Afonov
See All by Igor Afonov
Application deployment with Chef
iafonov
6
980
Web servers - and how I created my own one
iafonov
8
1.2k
Chef - Infrastructure as code
iafonov
6
670
Other Decks in Programming
See All in Programming
複雑なフォームに立ち向かう Next.js の技術選定
macchiitaka
2
240
Kiroで始めるAI-DLC
kaonash
2
630
速いWebフレームワークを作る
yusukebe
5
1.7k
スケールする組織の実現に向けた インナーソース育成術 - ISGT2025
teamlab
PRO
2
170
テストカバレッジ100%を10年続けて得られた学びと品質
mottyzzz
2
610
ぬるぬる動かせ! Riveでアニメーション実装🐾
kno3a87
1
230
複雑なドメインに挑む.pdf
yukisakai1225
5
1.2k
詳解!defer panic recover のしくみ / Understanding defer, panic, and recover
convto
0
250
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
260
Namespace and Its Future
tagomoris
6
710
Tool Catalog Agent for Bedrock AgentCore Gateway
licux
7
2.5k
AIでLINEスタンプを作ってみた
eycjur
1
230
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.8k
Designing Experiences People Love
moore
142
24k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Embracing the Ebb and Flow
colly
87
4.8k
Automating Front-end Workflow
addyosmani
1370
200k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
530
Optimizing for Happiness
mojombo
379
70k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.7k
The Art of Programming - Codeland 2020
erikaheidi
56
13k
Transcript
Real-time application monitoring Igor Afonov @iafonov
Background • SaaS application • 5 production servers (2 auxiliary
servers) • Everything managed by chef • Apache/Passenger/Rails/MySQL/Postfix
Metrics that matter • Server state • Application state
How metrics data is stored • Round-robin database • RRDTool
(C), Whisper (Python) (Minor offtopic)
Server state • Munin - storage and graphs • Monit
- alerts, basic rescue actions Shard munin-server munin-node Shard munin-node
Munin • Server pulls data from nodes • Gathers basic
server health data • A lot of custom plugins
Munin + Chef = — # Client config template munin_servers
= search(:node, "role:monitoring") <% munin_servers.sort.each do |server| -%> allow ^<%= server[:ipaddress].to_s.gsub(/\./, '\.') %>$ <% end %> # Server config template munin_nodes = search(:node, "munin:[* TO *]") <% munin_nodes.each do |system| -%> [<%= system[:hostname] %>] address <%= system[:ipaddress] %> use_node_name yes <% end %>
Application state • Subscriptions • Logins • Orders • Business
metrics • ...
Our setup Shard Monitoring Server StatsD Graphite Whisper Carbon WebApp
Shard Shard
StatsD • Lightweight proxy to graphite • 300 lines of
Javascript • Uses UDP (small size, non-blocking) • 10+ implementations • Simple protocol
StatsD • Count: counter:1|c • Measure: metric:200|ms • Gauge: value:9000|g
• Supports sampling • A lot of client-side libraries
StatsD StatsD.server = '178.22.33.88:8125' # increment counter StatsD.increment("users.new") # measure
task StatsD.measure("cron.#{task}") do task.run end # meta-programming - track subscriptions Subscription.extend StatsD::Instrument Subscription.statsd_count :subscribe, 'subscriptions'
Graphite • Python everywhere • Whisper - RRD, stores data
• Carbon - backend • Graphite - draws graphs, works with data
Graphite [stats] priority = 100 pattern = ^stats\..* retentions =
10:2160,60:10080,600:262974 (Storage schemas)
Problems • Basic UI • Complex installation and setup
telemetry.io • Fun project • SaaS application • Custom front-end
for StatsD + Graphite • Optimized for big screens (TVs) • Free • Maybe open-source
telemetry.io Client telemetry.io StatsD Graphite Whisper Carbon WebApp Custom Front-end
Client Client Client
Workflow • Get access token • Use one of available
libs or create own • Prepend token to metric name • Integrate into application StatsD.increment("#{token}.subscribers")
None
None
None
None
Implementation • Ruby on Rails • CoffeeScript • Chef (full
node bootstrap in 10 minutes)
Links • http://telemetry.io • http://code.flickr.com/blog/2008/10/27/ counting-timing/ • http://codeascraft.etsy.com/2011/02/15/ measure-anything-measure-everything/
http://iafonov.github.com/ @iafonov