Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amit Saha - Counter, Gauge, Upper 90 - Oh my!

Amit Saha - Counter, Gauge, Upper 90 - Oh my!

Setting up application monitoring is often an afterthought, and in the speaker's opinion can be a bit overwhelming to get started with. What is a `metric`? What is a `gauge`? What is a `counter`? What's that `upper 90` metric you have up on your `dashboard`? And what *all* metrics should I monitor?

This talk aims to get you started on the monitoring journey in Python. In addition to clearing up some of the jargon, we will look at `statsd` and `prometheus` monitoring systems and how to integrate our applications with these.

Without the numbers, we are really flying blind!

https://us.pycon.org/2018/schedule/presentation/133/

PyCon 2018

May 11, 2018
Tweet

More Decks by PyCon 2018

Other Decks in Programming

Transcript

  1. Counter, Gauge, Upper 90 - Oh my! let’s learn enough

    to worry think about metrics Amit Saha @echorand
  2. “When an ostrich is afraid, it will bury its head

    in the ground, assuming that because it cannot see, it cannot be seen” http://drpaulose.com/spirituality/ostrich-mentality
  3. Currently DevOps Engineer at RateSetter Australia Author of “Doing Math

    with Python” and various technical articles Fedora Scientific creator/maintainer About me
  4. Metric The measure/value of a quantity at a given point

    of time Source: matplotlib examples showcase
  5. Gauge A metric whose value can go up or down

    arbitrarily - usually with a floor and ceiling
  6. Demo 1: What did we see? Lots of metrics generated,

    hence we need to summarize the data
  7. Demo 1: What did we see? No characteristics in the

    metrics - which endpoint? What response status?
  8. Mean and Median Mean Mean of 5, 8, 3 =

    (5+8+3)/3 = 5.33.... Median: a better average Median of 5, 8, 3 is 5
  9. Percentile and Upper X The percentile is a measure which

    gives us a measure below which a certain, k percentage of the numbers lie. Most monitoring systems refer to it as upper_X where X is the percentile.
  10. Quantile A quantile gives us another way to find a

    number at a specific position in a set of numbers 0.xy quantile => xy percentile
  11. Why do we need characteristics? What was the latency for

    a specific instance of the application?
  12. Why do we need characteristics? What were the number of

    HTTP 500s for a specific endpoint?
  13. Examples of metric characteristics System identifier (IP address, Container ID,

    AWS instance ID..) HTTP Endpoint name HTTP Method HTTP response status RPC Method Name ..
  14. Demo 2: What did we see? We saw how we

    can add characteristics to metrics
  15. Demo 2: What did we see? We have a multi-column

    CSV file - what does it look similar to?
  16. Summary: Monitoring your applications 1. Your application calculates the metrics

    (Middleware) 2. A monitoring system stores these (CSV files) 3. Human/machine queries the monitoring system (Pandas)
  17. What application metrics should I calculate? Network servers: Request latency,

    Queue size (if any), Exceptions, Waiting time, Worker usage Batch jobs: Last run, latency Consumers: Latency Recommended: The four golden signals
  18. Monitoring Systems Self hosted/maintained - statsd, prometheus Third party SaaS

    - https://www.outlyer.com/features/ - https://docs.datadoghq.com/developers/dogstatsd/ - https://honeycomb.io/docs/
  19. Key statsd concepts Application push metrics to statsd server (usually

    over UDP) A metric key is of the form webapp1.<key1>.<key2>...<keyN>.latency Each dot separated part of the key is a metric characteristic/dimension
  20. Key prometheus concepts Each metric can be associated with multiple

    labels which are the characteristics of the metric Internally, each metric and label combination is a separate metric
  21. Native prometheus exporting in Python has certain gotchas I recommend

    using the statsd exporter I have written about this topic elsewhere
  22. We should talk about them, learn as we go -

    may be from first principles And once we have learned enough ...
  23. Thanks You for choosing my talk! PyCon committee for the

    opportunity! My previous employer and team at Freelancer.com My employer - RateSetter Australia for funding my conference visit! Sydney Python Meetup group for the opportunity to deliver a version of this talk Nick Coghlan for feedback and lending a travel adapter :)