Brubeck - A high-performance statsd-compatible aggregation daemon

Transcript

1. the best way to build and ship software Brubeck a high-performance statsd-compatible aggregation daemon
2. the best way to build and ship software ! It’s all about timing… 2
3. the best way to build and ship software ! 3 Measure Everything
4. the best way to build and ship software “ ” ! IT’S NOT SHIPPED UNTIL IT’S FAST. - TED NYMAN Measure Everything 4
5. the best way to build and ship software ! 5 How did we make Brubeck fast?
6. the best way to build and ship software “ ” ! BRUBECK’S FEATURE IS THAT IT HAS NO FEATURES. IT IS A VERY SIMPLE C DAEMON WHICH USES STATE-OF-THE-ART TECHNOLOGIES SUCH AS "A WHILE LOOP" AND "MORE THAN ONE THREAD". - VICENT MARTÍ How did we make Brubeck fast? 6
7. the best way to build and ship software ! 7 What can it do?
8. the best way to build and ship software Write data to Carbon Brubeck consumes metrics via statsd, samples the metrics, and writes the results to Carbon. 8 !
9. the best way to build and ship software ! 9 … " PROTOCOLS SHARDING • Plaintext • Pickle • Consistent hashing to multiple backends Carbon Backend
10. the best way to build and ship software Rich Logging Designed to emit detailed logging information as key=value pairs. 10 !
11. the best way to build and ship software ! Rich Logging 11 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  sudo  tail  -­‐F  /var/log/application.log   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:35-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.catchall]  [38047]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.catchall  sampler=statsd  event=bad_key   key='team.resque.payload_size.job.Jobs.DeliverNotification.:1|c'  from=184.73.32.39   2015-­‐06-­‐16T15:00:38-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492038  timestamp=1434492033  drift=5   2015-­‐06-­‐16T15:00:53-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_delayed  now=1434492053  timestamp=1434492048  drift=5   2015-­‐06-­‐16T15:01:00-­‐07:00  metrics-­‐collector1-­‐cp1-­‐prd  -­‐-­‐  [config.secure]  [38045]:   instance=brubeck.metrics-­‐collector1-­‐cp1-­‐prd.secure  sampler=statsd-­‐secure   event=fail_future  now=1434492060  timestamp=1434492061
12. the best way to build and ship software Statsd Sampler Read data from statsd clients and store it in memory. 12 !
13. the best way to build and ship software ! 13 ⚡ $ CONFIGURE TYPES • Number of worker threads • Enable SO_REUSEPORT • Enable recvmsg(2) • Gauges, meters, counters, histograms, and timers Statsd Sampler
14. the best way to build and ship software Statsd-secure Sampler Just like the statsd sampler, but tamper-proof. 14 !
15. the best way to build and ship software ! 15 % & SECURITY PREVENTS • Prepends the plaintext message with a timestamp and a nonce • Generates an HMAC-SHA256 signature using a shared key • Prepends the payload with the signature • Tampering • Replay attacks • Impersonation Statsd-secure Sampler
16. the best way to build and ship software ! Statsd-secure Sampler 16 Payload Message https://github.com/github/statsd-ruby
17. the best way to build and ship software ! 17 ⚡ ' CONFIGURE CAVEATS • Drift window • Replay length • Doesn’t encrypt messages • Use a tunnel if this is a requirement • Has a 0.1% probability of dropping a valid message because of the bloom filter implementation Statsd-secure Sampler
18. the best way to build and ship software Metrics Expiration Configure the number of seconds before a metric “expires”. If a new data point hasn’t arrived for a metric in the expiration window it will be removed from memory and samples will no longer be flushed to the backend. 18 !
19. the best way to build and ship software Metrics Dump Signal Brubeck with USR2 and it will record all of the metrics in its table to disk, along with each metric’s type. 19 !
20. the best way to build and ship software HTTP API Interact with Brubeck from other tools and services. 20 !
21. the best way to build and ship software ! 21 ( ) GET /PING GET /STATS • return a short JSON payload with the current status of the daemon (just to check it's up) • get a large JSON payload with full statistics, including active endpoints and throughputs HTTP API
22. the best way to build and ship software ! 22 * + GET /METRIC/{{METRIC_NAME}} POST /EXPIRE/{{METRIC_NAME}} • get the current status of a metric, if it's being aggregated • expire a metric that is no longer being reported to stop it from being aggregated to the backend HTTP API
23. the best way to build and ship software Proclines Because the command-line is a beautiful place. 23 !
24. the best way to build and ship software ! Proclines 24 jssjr@metrics-­‐collector1-­‐cp1-­‐prd:~$  ps  auxwww  |  grep  [b]rubeck   brubeck      38043    395    0.0  802596  80092  ?                Ssl    Jun04  71997:07  brubeck  -­‐-­‐   [config.github]  [  ↑  #1  539.5gb,  #2  548.0gb,  #3  531.2gb,  #4  551.0gb,  #5  545.0gb  ]  [  ↓  : 8226  501593/s  ]   brubeck      38045    2.9    0.0  542148  10176  ?                Ssl    Jun04  535:13  brubeck  -­‐-­‐   [config.secure]  [  ↑  #1  67.1gb,  #2  71.0gb,  #3  64.1gb,  #4  69.8gb,  #5  67.1gb  ]  [  ↓  :9126   1131/s  ]   burbeck      38047    112    0.0  793308  94252  ?                Ssl    Jun04  20540:24  brubeck  -­‐-­‐   [config.catchall]  [  ↑  #1  232.0gb,  #2  234.2gb,  #3  232.2gb,  #4  234.2gb,  #5  235.5gb  ]  [  ↓  : 8126  47292/s  ]
25. the best way to build and ship software Limitations Not everything is actually compatible. Sorry. 25 !
26. the best way to build and ship software ! 26 x - DOESN’T SUPPORT DIFFERENCES • Multi-line messages • Statsd sample timing (foo:1|ms|@0.1) • The set type • No stats and stats_count namespace split • Histograms are slightly different, let me show you… Limitations
27. the best way to build and ship software ! Histogram Differences 27 Esty Statsd GitHub Brubeck
28. the best way to build and ship software Thanks! Scott Sanders Infrastructure Engineer @ GitHub https://github.com/jssjr https://twitter.com/scott_sanders