Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Observing and Understanding Behavior in Complex...

Observing and Understanding Behavior in Complex Systems

Theo Schlossnagle

November 06, 2013
Tweet

More Decks by Theo Schlossnagle

Other Decks in Technology

Transcript

  1. THE MANY FACES OF THEO FUN WITH BEARDS AND HAIR

    FUCK IT ALL VENDETTA SCARY DETERMINED CAREFREE NO-FLY
 ZONE
  2. Complex? Classic Complex Chaotic damped, driven system whose total energy

    exceeds the thresholds according to classical mechanics but does not meet the thresholds for chaotic systems. Operator Happiness 100
  3. Chaotic Systems are sensitive to initial conditions, must be topologically

    mixing, its periodic orbits must be dense. ! to apply this directly
 … we must have an evolution function
 … we do not ✓ ✓ analogously ✓ with bugs
  4. Where from here? Our systems are complex. Bugs make them

    chaotic at times. Decoupling makes them very hard to observe
 … let alone understand. ! because determining causality by observation
 is not possible.
  5. Data over logs The transaction is “lost” in an asynchronous

    system Determination of causal effects cross-txn is impossible The best we can do is: Observe Analyze Correlate
  6. Some Tenets Most people suck at monitoring. They monitor all

    the wrong things (somewhat bad) The don’t monitor the important things (awful)
  7. Do not collect rates of things Rates are like trees

    making sounds falling in the forest. Direct measurement of rates leads to data loss
 and ultimately ignorance.
  8. Prefer high level telemetry 1. Business drivers via KPIs, 2.

    Team KPIs, 3. Staff KPIs, 4. ... then telemetry from everything else.
  9. Methodology I’m going to focus on methodology
 that can be

    applied across whatever toolset you have.
  10. Pull vs. Push Anyone who says one is better than

    the other is...
 WRONG. They both have their uses.
  11. Reasons for pull 1. Synthesized observation is desirable. 2. Observable

    activity is infrequent. 3. Alterations in observation frequency are useful.
  12. Reasons for push Direct observation is desirable. Discrete observed actions

    are useful. Discrete observed actions are frequent.
  13. Protocol Soup SNMP(v1,v2,v3) both push(trap) and pull(query) collectd(v4,v5) push only

    statsd push only JMX, JDBC, ICMP, DHCP, NTP, SSH, TCP, UDP, barf.
  14. High-volume Data Occasionally, data velocity is beyond what’s reasonable for

    individual HTTP PUT/POST for each observation. 1. You can fall back to UDP (try statsd) 2. I prefer to batch them and continue to use REST
  15. The real question is: “what?” What should I be monitoring?

    This is the best question you can ask yourself. Before you start. While you’re implementing. After you’re done.
  16. The industry answer: MONITOR ALL THE THINGS! I’ll tell you

    this too, in fact. But we have put the cart ahead of the horse.
  17. Question? If I could monitor one thing, what would it

    be?
 
 hint: CPU utilization on your web server ain’t it.
  18. Answer: It depends on your business. If you don’t know

    the answer to this,
 I suggest you stop worrying about monitoring
 and start worrying about WTF your company does.
  19. Monitoring purchases. So, we should monitor how many purchases were

    made and ensure it is within acceptable levels. ! Not so fast.
  20. Actually. We want to make sure customers
 can purchase from

    the site and
 are purchasing from the site. This semantic different is critically important. And choosing which comes down to velocity.
  21. What is this velocity thing? Displacement / time
 (i.e. purchases/second

    or $/second) BUT WAIT! You said:
 “Do not collect rates of things.” Correct...
 collect the displacement,
 visualize and alert on the rate.
  22. So which? High velocity w/ predictably smooth trends:
 velocity is

    more important Low velocity or uneven arrival rates:
 measuring capability is more important
  23. To rephrase If you have sufficient real data,
 observing that

    data works best; otherwise, you must
 synthesize data and monitor that.
  24. More demonstrable (in a short session) I’ve got a web

    site that my customers need to visit. The business understands that we need to serve customers with at least a basic level of QoS:
 no page loads over 4s
  25. A first attempt curl http://surge.omniti.com/ extract the HTTP response code

    if 200, we’re super good! ! Admittedly not so good.
  26. A wealth of data. Synthesizing an HTTPS GET could provide:

    SSL Subject, validity, expiration HTTP code, Headers and Content Timings on TCP connection, first byte, full payload
  27. Still, this is highly imperfect. Don’t get me wrong, they

    are useful.
 We use them all over the place... they are cheap. But, ideally, you want to load the page closer to the way a user does (all assets, javascript, etc.) Enter phantomjs
  28. var page = require('webpage').create(); page.viewportSize = { width: 1024, height:

    768 }; ! page.onError = function(err) { stats.errors++; }; page.onInitialized = function() { start = new Date(); }; page.onLoadStarted = function() { stats.load_started = new Date() - start; }; page.onLoadFinished = function() { stats.load_finished = new Date() - start; }; page.onResourceRequested = function() { stats.res++; }; page.onResourceError = function(err) { stats.res_errors++; }; page.onUrlChanged = function() { stats.url_redirects++; }; ! page.open('http://surge.omniti.com/', function(status) { stats.status = status; stats.duration = new Date() - start; console.log(JSON.stringify(stats)); phantom.exit(); });
  29. var start, stats = { status: null , errors: 0

    , load_started: null , load_finished: null , resources: 0 , resource_errors: 0 , url_redirects: 0 };
  30. Now for the passive stuff Some examples are Google Analytics,

    Omniture, etc. Statsd (out-of-the-box) and Metrics
 are mediocre approach. If we have a lot of observable data N,
 N̅ isn’t so useful,
 , |N|, q(0.5), q(0.95), q(0.99), q(0), q(1), add a lot.
  31. Still... we can do better. N̅, , |N|, q(0,0.5,0.95,0.99,1) is

    8 statistical aggregates Let’s look at API latencies...
 say we do 1000/s,
 that’s 60k/minute. Over a minute of time, 60k points to 8 represents...
 a lot of information loss.
  32. Histograms 101 This. This is a histogram. It shows the

    frequency of
 values within a population. Height represents frequency
  33. Histograms 101 This. This is a histogram. It shows the

    frequency of
 values within a population. Now, height and color
 represents frequency
  34. This. This is a histogram. It shows the frequency of


    values within a population. Now, only color
 represents frequency Histograms 101
  35. This. This is a histogram. It shows the frequency of


    values within a population. Now, only color
 represents frequency Histograms ➠ time series at a single time interval
  36. Observability I don’t want to launch into a tutorial on

    DTrace
 despite the fact that you can simple spin up an OmniOS AMI in Amazon and have it now. Instead let’s talk about what shouldn’t happen.
  37. The production questions: I wonder if that queue is backed

    up... Performance like that should only happen if our binary tree is badly imbalanced (replace with countless other pathologically bad precipitates of failure); I wonder if it is... It’s almost like some requests are super slow; I wonder if they are. STOP WONDERING.
  38. Instrument your software Instrument your software and systems 
 and

    stop the wonder Do it for the kids This is simple with DTrace & a bit more work otherwise Avoiding work is not an excuse for ignorance
  39. A tour through our Sauna We have this software that

    stores data...
 happens to store all data visualized in Circonus. We have to get data into the system. We have to get data out of the system. I don’t wonder... here’s why.
  40. Data tenet. Do not collect data twice. That which you

    collect for visualization
 should be the same data on which you alert.