Graphite 101

A bitly tech talk on how we do internal app metrics with Graphite

Transcript

1. GRAPHITE 101 What we count, How we count it, and Why Jehiah - bitly Tech Talk - Jan 16th 2014
2. Why do we look at Numbers and Metrics every day? Detect Trends in usage (desired or undesired) Make Product Decisions Quantify Effects of Changes Ensure Health of Systems
3. 3 Basic Metric Types How Graphite Works Building a Chart Gotchas WHAT WE WILL COVER
4. 3 BASIC METRIC TYPES
5. COUNTER Something you can discretely count via 1+1 You would measure entrances to the bitly office by counting +=1 each time someone enters. You can break the counting into parts (by time and by hour) and sum() to get the total.
6. GAUGE A gauge is something you want to keep track of the absolute value over time (like the temperature outside). If you count from multiple locations or multiple segments you max()/min()/avg() to aggregate.
7. PERCENTILES When you want to know the distribution of a set of values. (What was the fastest/slowest/average time x took). Stored as pre-calculated percentiles, and averages for all the values in a time period (minute). 100th percentile is a max, 0th percentile is a min. 50th percentile is the median To combine values you avg()/min()/max() percentiles depending on desired output
8. WHAT ARE WE MEASURING?
9. 434 servers 14,000 HTTP Requests/second (1B/day) 3,250 Link Clicks/second (223M/day) 67,000 Link Metrics/second (4B/day) 59,515 Graphite Metrics every minute
10. GRAPHITE
11. App statsdaemon UDP key:3|c Graphite TCP */60sec key 8 {ts} key:5|c App statsdaemon UDP temp:3|g Graphite TCP */60sec temp 5 {ts} temp:5|g App statsdaemon UDP d:7|ms Graphite TCP */60sec d.count 3 {ts} d:5|ms d:23|ms d.upper_100 23 {ts} d.median 7 {ts}
12. • app.{app_name}.%h.{metric_name} • staging.app.{app_name}.%h.{metric_name} • stream_metrics.%h.{data_type}.{metric_name} • audit.daily.{data_type}.{metric_name} • server.%h.{metric_name} • nsq.%h.{topic}.{channel}.{metric_name} Organizing Metrics
14. BUILDING A GRAPHITE CHART
15. stream_metrics.streammetrics01_ec2.decodes \ .device_type.iphone
16. stream_metrics.*.decodes.device_type.iphone
17. sumSeries( stream_metrics.*.decodes.device_type.iphone )
18. summarize( sumSeries( stream_metrics.*.decodes.device_type.iphone ) ,"1h", "avg")
19. scale( summarize( sumSeries( stream_metrics.*.decodes.device_type.iphone ) ,"1h", "avg") , 60)
20. scale( summarize( groupByNode( stream_metrics.*.decodes.device_type.* , 4, "sumSeries") ,"1h", "avg") , 60)
21. GRAPHITE FUNCTIONS sumSeries(…) summarize(…,"1d","avg") scale(…, n) asPercent(…, …) alias(…, "key") maxSeries(…) groupByNode(…, n) highestAverage(…, n) timeShift(…, "7d") movingAverage(…,"1d") http://bit.ly/graphite-functions
22. Percentile Example (API Response Time)
23. GOTCHAS • At less than 1 datapoint per hour, per host metrics are inaccurate (nulls and zero’s don’t average the same way) • 95th percentile counted across five hosts is the 99th percentile. • Percentiles combine in low volume situations (when you have fewer datapoints than percentiles)
24. Questions? @Jehiah