Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lies, Damn Lies, and Metrics (Strange Loop 2016)

André Arko
September 17, 2016

Lies, Damn Lies, and Metrics (Strange Loop 2016)

Metrics are great, and measuring things can provide tremendously useful insights. But there's a problem: metrics lie to you. Metrics just report the numbers that were measured. Analyzing those numbers is up to us, and that analysis can go wrong in so, so many ways. Learn how to arm yourself against human intuition, interpreter pauses, routing, instrumentation lag, and other issues. Don't get so caught up in instrumenting that you lose sight of why metrics exist! Make sure your metrics are telling you actionable information, instead of just accurate numbers.

André Arko

September 17, 2016
Tweet

More Decks by André Arko

Other Decks in Technology

Transcript

  1. “Normal” 5 -5 -4 -3 -2 -1 0 1 2

    3 4 0 0.1 0.2 0.3 0.4
  2. “Normal” 5 -5 -4 -3 -2 -1 0 1 2

    3 4 0 0.1 0.2 0.3 0.4
  3. Real Life 5 -5 -4 -3 -2 -1 0 1

    2 3 4 0 0.1 0.2 0.3 0.4
  4. The problem with averages: If you put one hand in

    a bucket of ice and the other in a bucket of hot coals, on average, you’re comfortable. Erik Michaels-Ober @sferik
  5. 10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  6. 10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  7. 10 0 1 2 3 4 5 6 7 8

    9 250 0 50 100 150 200
  8. 10 0 1 2 3 4 5 6 7 8

    9 1000 0 250 500 750
  9. Average of X Average of Y Variance of X Variance

    of Y Correlation of X and Y Linear regression All four data sets have the same