Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bring the Noise: Continuously Deploying Under a...

Bring the Noise: Continuously Deploying Under a Hailstorm of Metrics

This talk was given at Velocity '13 in Santa Clara, and an abbreviated version was given at BACON '13 in London. It offers an overview of Etsy's Kale stack.

BACON Video: devslovebacon.com/conferences/bacon-2013/talks/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metrics

Velocity video forthcoming.

Abe Stanway

June 18, 2013
Tweet

More Decks by Abe Stanway

Other Decks in Programming

Transcript

  1. Abe Stanway @jonlives BRING THE NOISE! MAKING SENSE OF A

    HAILSTORM OF METRICS Jon Cowie @abestanway
  2. Ninety minutes is a long time. - motivations - skyline

    - oculus - demo! - questions This talk: ~10 ~25 ~30 ~10 ~15
  3. Ninety minutes is a long time. - motivations - skyline

    - oculus - demo! - questions This talk: ~10 ~25 ~30 ~10 ~15 But we have some sweet stuff to show you.
  4. 1.5 billion page views $117 million of goods sold 950

    thousand users (in december ‘12)
  5. de • ploy /diˈploi/ Verb To release your code for

    the world to see, hopefully without breaking the Internet
  6. [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20]

    [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60] [1358731200, 20] [1358731200, 20]
  7. Graphite’s relay agent original graphite backup graphite [statsd.numStats, [1365603422, 82345]]

    pickles [statsd.numStats, [1365603432, 80611]] [statsd.numStats, [1365603412, 73421]]
  8. Graphite’s relay agent original graphite skyline [statsd.numStats, [1365603422, 82345]] pickles

    [statsd.numStats, [1365603432, 80611]] [statsd.numStats, [1365603412, 73421]]
  9. Basic algorithm: “A metric is anomalous if its latest datapoint

    is over three standard deviations above its moving average.”
  10. “[[975, 1365528530], [643, 1365528540], [750, 1365528550], [992, 1365528560], [580, 1365528570],

    [586, 1365528580], [649, 1365528590], [548, 1365528600], [901, 1365528610], [633, 1365528620]]” Use raw timeseries instead of raw graphs
  11. “975 643 643 750 992 992 992 580” “sharpdecrement flat

    increment sharpincrement flat flat shapdecrement” Shape Description Alphabet
  12. “975 643 643 750 992 992 992 580” “sharpdecrement flat

    increment sharpincrement flat flat shapdecrement” Shape Description Alphabet “24 4 4 11 25 25 25 0 1” (normalization step)
  13. Elasticsearch Details Phrase search for first pass scores across shape

    description fingerprints Custom FastDTW and euclidian distance plugins to score across the remaining filtered timeseries
  14. First pass query :match => { :fingerprint => { :query

    => “sdec inc sinc sdec inc”, :type => "phrase", :slop => 20 } } shape description fingerprint
  15. Refinement query {:custom_score => { :query => <first_pass_query>, :script =>

    "oculus_dtw", :params => { :query_value => “10 20 20 10 30”, :query_field => "values.untouched", }, } raw timeseries
  16. Special thanks to: Dr. Neil Gunther, PerfDynamics Dr. Brian Whitman,

    Echonest Burc Arpat, Facebook Seth Walker, Etsy Rafe Colburn, Etsy Mike Rembetsy, Etsy John Allspaw, Etsy