Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Correlation: The Next Frontier

Correlation: The Next Frontier

My talk from #monitorama 2013 with ideas about how to apply different types of correlation to our data

Aaron Quint

March 28, 2013
Tweet

More Decks by Aaron Quint

Other Decks in Technology

Transcript

  1. A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations
  2. A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations • Beginner Distributed Systems Maintainer
  3. 0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time
  4. 0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time
  5. 1 data = [ 2 [100, 0.7], 3 [125, 0.5],

    4 [150, 1], 5 [300, 2.1], 6 [500, 3.4], 7 [900, 6] 8 ] 9 10 x, y = data.transpose 11 n = data.size 12 x_mean = x.reduce(:+) / n 13 y_mean = y.reduce(:+) / n 14 x_stddev = Math.sqrt(x.inject {|sum, i| sum + (i - x_mean)**2 } / (n - 1).to_f) 15 y_stddev = Math.sqrt(y.inject {|sum, i| sum + (i - y_mean)**2 } / (n - 1).to_f) 16 z_x = x.collect {|i| (i - x_mean) / x_stddev } 17 z_y = y.collect {|i| (i - y_mean) / y_stddev } 18 pearsons = z_x.zip(z_y).collect {|x| x[0] * x[1] }.reduce(:+) / n 19 # => 0.9265763490538744
  6. PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages?
  7. PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages? • Could we pull it into a graphite function? (Hackathon anyone?)
  8. LIMITS OF Mathematical correlation • Requires known inputs and assumptions

    • Suggestion of correlation, not proof • Needs a large amount of knowledge of the data set to make decisions
  9. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise
  10. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events
  11. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events • Good for trends but not as much for events
  12. Each person uses their unique knowledge of the situation to

    point out unique data points. RASHoMONING
  13. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people
  14. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge”
  15. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge” • HUMANS (Hence Rashomoning)