Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Trending with Purpose

Trending with Purpose

Given at the BaltimorePM user group in June 2011.

Jason Dixon

May 15, 2012
Tweet

More Decks by Jason Dixon

Other Decks in Programming

Transcript

  1. "I don't care if my servers are on fire as

    long as they're making me money" Business Owner
  2. Nagios • What it does well: • Free • Extensible

    • Plugins • Configuration templates
  3. Nagios • What it does well: • Free • Extensible

    • Plugins • Configuration templates • Popular (lesser of all free evils)
  4. Nagios • What it does well: • Free • Extensible

    • Plugins • Configuration templates • Popular (lesser of all free evils) • Log metrics (“performance data”)
  5. Nagios • Where it sucks: • Interface • (Lack of)

    Scalability • Promotes bad habits
  6. Nagios • Where it sucks: • Interface • (Lack of)

    Scalability • Promotes bad habits • Acknowledgements never expire
  7. Nagios • Where it sucks: • Interface • (Lack of)

    Scalability • Promotes bad habits • Acknowledgements never expire • Configuration (over-)flexibility
  8. Nagios • Where it sucks: • Interface • (Lack of)

    Scalability • Promotes bad habits • Acknowledgements never expire • Configuration (over-)flexibility • Flapping
  9. PNP4Nagios • Basic graphing & dashboard capabilities • Retrieves Nagios

    performance data • Creates graphs with RRD • Limited introspection/correlation
  10. PNP4Nagios • Basic graphing & dashboard capabilities • Retrieves Nagios

    performance data • Creates graphs with RRD • Limited introspection/correlation • http://www.pnp4nagios.org/
  11. Graphite • Metric storage • Complex graph creation • Web

    and “CLI” interfaces • Created and released by Orbitz.com
  12. Graphite • Metric storage • Complex graph creation • Web

    and “CLI” interfaces • Created and released by Orbitz.com • http://graphite.wikidot.com/
  13. Graphite • The good stuff: • Horizontally scalable • Rapid

    graph prototyping (CLI) • Graph disparate data points
  14. Graphite • The good stuff: • Horizontally scalable • Rapid

    graph prototyping (CLI) • Graph disparate data points • Numerous formulas available
  15. Graphite • The good stuff: • Horizontally scalable • Rapid

    graph prototyping (CLI) • Graph disparate data points • Numerous formulas available • derive, transform, average, sum, etc...
  16. Graphite • The good stuff: • Horizontally scalable • Rapid

    graph prototyping (CLI) • Graph disparate data points • Numerous formulas available • derive, transform, average, sum, etc... • Share graphs with other users
  17. Graphite • The good stuff: • Horizontally scalable • Rapid

    graph prototyping (CLI) • Graph disparate data points • Numerous formulas available • derive, transform, average, sum, etc... • Share graphs with other users • Supports existing RRD databases
  18. Graphite • The not-so-good stuff: • Not a dashboard (well,

    sorta) • No hover details • Single y-axis
  19. Carbon • agent - starts other daemons, receives metrics and

    pipelines them to cache • aggregator - aggregate and transform your data before storage
  20. Carbon • agent - starts other daemons, receives metrics and

    pipelines them to cache • aggregator - aggregate and transform your data before storage • cache - caches metrics for real-time graphing, pipelines them to persister
  21. Carbon • agent - starts other daemons, receives metrics and

    pipelines them to cache • aggregator - aggregate and transform your data before storage • cache - caches metrics for real-time graphing, pipelines them to persister • persister - writes persistent data to disk
  22. Whisper • Metrics database format • Supplanted RRDtool • Accepts

    out-of-order data • Supports pipelining of data in a single operation (multiplexing)
  23. Coming Soon - Ceres • Smaller files - doesn’t pad

    missing datapoints • Doesn’t store timestamps, calculates them
  24. Coming Soon - Ceres • Smaller files - doesn’t pad

    missing datapoints • Doesn’t store timestamps, calculates them • Federates individual datapoints
  25. Graphite Web • Traditional web interface • Javascript CLI •

    Rudimentary dashboard • Django application
  26. Sending metrics to Graphite • Connect to Carbon socket (tcp/2003)

    • Send your data my $sock = IO::Socket::INET->new(‘127.0.0.1:2003’); $sock->send(“endpoint.app.metric $value $epoch\n");
  27. Sending metrics to Graphite • Connect to Carbon socket (tcp/2003)

    • Send your data my $sock = IO::Socket::INET->new(‘127.0.0.1:2003’); $sock->send(“endpoint.app.metric $value $epoch\n"); • ???
  28. Sending metrics to Graphite • Connect to Carbon socket (tcp/2003)

    • Send your data my $sock = IO::Socket::INET->new(‘127.0.0.1:2003’); $sock->send(“endpoint.app.metric $value $epoch\n"); • ??? • Profit!
  29. App/DB Profiling • How fast is our: • function foo()

    for each iteration • SQL query • 3rd party API service (e.g. payment gateway, social media)
  30. App/DB Profiling • How many times do we: • call

    function foo() • register a new user
  31. App/DB Profiling • How many times do we: • call

    function foo() • register a new user • chargeback a sale
  32. Business Profiling • How many sales did we generate this

    hour: • per sku • from the recent ad campaign
  33. Business Profiling • How many sales did we generate this

    hour: • per sku • from the recent ad campaign • from users in West Bumble, Arkansas
  34. Data Sources • Nagios • Munin • SNMP • Ganglia

    • collectd • SQL • Logs • sar • /proc • REST APIs
  35. You can’t be serious! • You’re thinking to yourself... •

    This sounds like a lot of work. • Our developers will never buy in.
  36. You can’t be serious! • You’re thinking to yourself... •

    This sounds like a lot of work. • Our developers will never buy in. • Is there an easier way?
  37. StatsD • Created and released by Etsy • "Measure Anything,

    Measure Everything" • Aggregate counters and timers
  38. StatsD • Created and released by Etsy • "Measure Anything,

    Measure Everything" • Aggregate counters and timers • Pipeline to Graphite
  39. StatsD • Created and released by Etsy • "Measure Anything,

    Measure Everything" • Aggregate counters and timers • Pipeline to Graphite • Fire-and-forget (UDP)
  40. StatsD • Created and released by Etsy • "Measure Anything,

    Measure Everything" • Aggregate counters and timers • Pipeline to Graphite • Fire-and-forget (UDP) • Perl, PHP, Python and Java clients
  41. StatsD • Created and released by Etsy • "Measure Anything,

    Measure Everything" • Aggregate counters and timers • Pipeline to Graphite • Fire-and-forget (UDP) • Perl, PHP, Python and Java clients • https://github.com/etsy/statsd
  42. StatsD • https://github.com/sivy/statsd-client use Net::StatsD::Client; my $c = Net::StatsD::Client->new(); $c->increment('endpoint.customer.app.metric');

    $c->timing('endpoint.customer.app.foo', 200); • Too much activity? Sample it! # sample 10%, StatsD will multiply it up $c->increment('endpoint.customer.app.metric', 0.1)
  43. Logster • Yet Another Etsy Project (YAEP) • Rips metrics

    from log files • https://github.com/etsy/logster
  44. Logster • Yet Another Etsy Project (YAEP) • Rips metrics

    from log files • https://github.com/etsy/logster # /usr/sbin/logster --dry-run --output=graphite \ --graphite-host=graphite.example.com:2003 \ SampleLogster /var/log/httpd/access_log