Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Chaos To Build Resilient Systems

Using Chaos To Build Resilient Systems

https://qconnewyork.com/ny2018/speakers/tammy-butow

There are those of us that are motivated to build resilient systems, improve uptime, move fast and keep systems reliable. Then there are those of us who feel overwhelmed by our to-do lists and the features or projects we feel we need to get out the door.

The world needs more resilient systems because the world needs engineers in this for the long haul. We can create a better future for ourselves, those who come after us, our customers and our wider teams by focusing on building resilient systems. How do we make it easier for everyone to build resilient systems?

It is not easy to build resilient systems, but that doesn’t mean we shouldn’t try. Engineers love a technical challenge. In this talk I will explain how focusing on the detection, mitigation, resolution and prevention of incidents is a great place to start. I will share my experiences using chaos engineering to build resilient systems... even when you can’t build your systems from scratch.

Tammy Bryant Butow

June 29, 2018
Tweet

More Decks by Tammy Bryant Butow

Other Decks in Technology

Transcript

  1. TAMMY BÜTOW Principal SRE, Gremlin Causing chaos in prod since

    2009. Previously SRE Manager @ Dropbox leading Databases, Block Storage and Code Workflows for 500 million users and 800 engineers. @tammybütow % @tammybütow #QCONNYC
  2. GREMLIN • We are practitioners of Chaos Engineering • We

    build software that helps engineers build resilient systems in a safe, secure and simple way. • We offer 11 ways to inject chaos for your Chaos Engineering experiments (e.g. host/container packet loss and shutdown) @tammybütow #QCONNYC
  3. • A resilient system is a highly available and durable

    system. • A resilient system can maintain an acceptable level of service in the face of failure. • A resilient system can weather the storm (a misconfiguration, a large scale natural disaster or controlled chaos engineering). @tammybütow #QCONNYC Let’s Define A Resilient System:
  4. It would be silly to give an Olympic pole-vaulter a

    broom and ban them from practicing! @tammybütow #QCONNYC
  5. “Thoughtful planned experiments designed to reveal the weaknesses in our

    systems” - Kolton Andrus, Gremlin CEO @tammybütow, Gremlin @tammybütow #QCONNYC
  6. Inject something harmful in order to build an immunity. @tammybütow,

    Gremlin @tammybütow #QCONNYC Think of it like a vaccination:
  7. Eventually systems will break in many undesired ways. Break them

    first on purpose with controlled chaos! @tammybütow #QCONNYC
  8. DOGFOODING • Using your own product. • For us that

    means using Gremlin for our Chaos Engineering experiments. • Failure Fridays @tammybütow #QCONNYC
  9. Failure Fridays are dedicated time for teams to collaboratively focus

    on using Chaos Engineering practices to reveal weaknesses in your services. @tammybütow #QCONNYC
  10. WHY DO DISTRIBUTED SYSTEMS NEED CHAOS? • Unusual hard to

    debug failures are common • Systems & companies scale rapidly and Chaos Engineering helps you learn along the way @tammybütow #QCONNYC
  11. FULL-STACK CHAOS ENGINEERING • You can inject chaos at any

    layer. • API, App, Cache, Database, OS, Host, Network, Power & more. @tammybütow #QCONNYC
  12. Are you confident that your metrics and alerting are as

    good as they should be? #pagerpain @tammybütow #QCONNYC
  13. Are you confident your customers are getting as good an

    experience as they should be? #customerpain @tammybütow #QCONNYC
  14. Are you losing money due to downtime and broken features?

    #businesspain @tammybütow #QCONNYC
  15. HOW TO RUN A CHAOS ENGINEERING EXPERIMENT • Form a

    hypothesis • Consider blast radius • Run experiment • Measure results • Find & fix issues or scale ⚡ @tammybütow #QCONNYC
  16. The 3 Prerequisites for Chaos Engineering @tammybütow, Gremlin @tammybütow #QCONNYC

    1. Monitoring & Observability 2. On-Call & Incident Management 3. Know Your Cost of Downtime Per Hour
  17. HOW TO CHOOSE A CHAOS EXPERIMENT • Identify top 5

    critical systems • Choose 1 system • Whiteboard the system • Select attack: resource/ state/network • Determine scope ⚡ @tammybütow #QCONNYC
  18. WHAT SHOULD WE MEASURE? • Availability — 500s • Service

    specific KPIs • System metrics: CPU, IO, Disk • Customer complaints @tammybütow #QCONNYC
  19. EXAMPLE SYSTEM: KUBERNETES RETAIL STORE User Primary: kube-01 Node: kube-02

    Node: kube-03 Node: kube-04 @tammybütow #QCONNYC
  20. We can increase CPU, Disk, IO & Memory consumption to

    ensure monitoring is setup to catch problems. Important to catch issues before they turn into high severity incidents (unable to purchase new product!) and downtime for customers. RESOURCE CHAOS @tammybütow #QCONNYC
  21. Ways to create process chaos on purpose: PROCESS CHAOS •

    Kill one process • Loop kill a process • Spawn new processes • Fork bomb @tammybütow #QCONNYC
  22. WHAT ARE OTHER WAYS YOU CAN TURN OFF A SERVER?

    WHAT IF YOU WANT TO TURN OFF EVERY SERVER WHEN IT’S ONE WEEK OLD? @tammybütow #QCONNYC
  23. THE MANY WAYS TO KILL CONTAINERS • Kill self •

    Kill a container from the host • Use one container to kill another • Use one container to kills several containers • Use several containers to kill several @tammybütow #QCONNYC
  24. The average lifespan of a container is 2.5 days And

    they fail in many unexpected ways. @tammybütow #QCONNYC
  25. We can combine different types of chaos engineering experiments to

    reproduce complicated outages. Reproducing outages gives you confidence you can handle it if/when it happens again. @tammybütow #QCONNYC
  26. Let’s go back in time to look at some of

    the worst outage stories that kicked off the introduction of chaos engineering. @tammybütow #QCONNYC
  27. DROPBOX’S WORST OUTAGE EVER Some master-replica pairs were impacted which

    resulted in the site going down. https://blogs.dropbox.com/tech/2014/01/outage-post-mortem/ @tammybütow #QCONNYC
  28. UBER’S DATABASE OUTAGE 1.Master log replication to S3 failed 2.Logs

    backed up on the primary 3.Alerts fired to engineer but they are ignored 4.Disk fills up on database primary 5.Engineer deletes unarchived WAL files 6.Error in config prevents promotion — Matt Ranney, Uber, 2015 @tammybütow #QCONNYC
  29. THERE ARE MANY MORE OUTAGES YOU CAN READ ABOUT HERE:

    https://github.com/danluu/post-mortems @tammybütow #QCONNYC