Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chaos Engineering - DevOps @ Nike Day

Chaos Engineering - DevOps @ Nike Day

A talk for the DevOps @ Nike Day.

Learn more about Chaos Engineering: gremlin.com/community
Join the Chaos Engineering Slack: gremlin.com/slack

Tammy Bryant Butow

June 18, 2018
Tweet

More Decks by Tammy Bryant Butow

Other Decks in Technology

Transcript

  1. “People constantly updated testing documents, and every day we made

    changes in the moment” @tammybütow DEVOPS @ NIKE DAY
  2. People Put Nike Hyperadapt 1.0 To The Test: • On

    The Court - 20 Basketball Research Athletes • Pounding Pavement - running at Nike campus • In The Gym - Nike Employee Training Classes • Walking And Working - 150 Employees @tammybütow DEVOPS @ NIKE DAY
  3. You can use Chaos Engineering to ensure your systems are

    as resilient as your sneakers. @tammybütow, Gremlin @tammybütow DEVOPS @ NIKE DAY
  4. TAMMY BÜTOW Principal SRE, Gremlin Causing chaos in prod since

    2009 @tammybütow @tammybütow DEVOPS @ NIKE DAY !
  5. GREMLIN • We are practitioners of Chaos Engineering • We

    build software that helps engineers build resilient systems • We offer 11 ways to inject chaos for your Chaos Engineering experiments @tammybütow DEVOPS @ NIKE DAY
  6. It would be silly to give an Olympic pole-vaulter a

    broom and ban them from practicing! @tammybütow DEVOPS @ NIKE DAY
  7. “Thoughtful planned experiments designed to reveal the weaknesses in our

    systems” - Kolton Andrus, Gremlin CEO @tammybütow, Gremlin @tammybütow DEVOPS @ NIKE DAY
  8. Eventually systems will break in many undesired ways. Break them

    first on purpose with controlled chaos! @tammybütow DEVOPS @ NIKE DAY
  9. DOGFOODING • Using your own product. • For us that

    means using Gremlin for our Chaos Engineering experiments. • Failure Fridays @tammybütow DEVOPS @ NIKE DAY
  10. Failure Fridays are dedicated time for teams to collaboratively focus

    on using Chaos Engineering practices to reveal weaknesses in your services. @tammybütow DEVOPS @ NIKE DAY
  11. WHY DO DISTRIBUTED SYSTEMS NEED CHAOS? • Unusual hard to

    debug failures are common • Systems & companies scale rapidly and Chaos Engineering helps you learn along the way @tammybütow DEVOPS @ NIKE DAY
  12. FULL-STACK CHAOS ENGINEERING • You can inject chaos at any

    layer. • API, App, Cache, Database, OS, Host, Network, Power & more. @tammybütow DEVOPS @ NIKE DAY
  13. Are you confident that your metrics and alerting are as

    good as they should be? @tammybütow DEVOPS @ NIKE DAY #pagerpain
  14. Are you confident your customers are getting as good an

    experience as they should be? @tammybütow DEVOPS @ NIKE DAY #customerpain
  15. Are you losing money due to downtime and broken features?

    @tammybütow DEVOPS @ NIKE DAY #businesspain
  16. HOW TO RUN A CHAOS ENGINEERING EXPERIMENT • Form a

    hypothesis • Consider blast radius • Run experiment • Measure results • Find & fix issues or scale @tammybütow DEVOPS @ NIKE DAY ⚡
  17. HOW TO CHOOSE A CHAOS EXPERIMENT • Identify top 5

    critical systems • Choose 1 system • Whiteboard the system • Select attack: resource/ state/network • Determine scope @tammybütow DEVOPS @ NIKE DAY ⚡
  18. WHAT SHOULD WE MEASURE? • Availability — 500s • Service

    specific KPIs • System metrics: CPU, IO, Disk • Customer complaints @tammybütow DEVOPS @ NIKE DAY
  19. EXAMPLE SYSTEM: KUBERNETES RETAIL STORE @tammybütow DEVOPS @ NIKE DAY

    User Primary: kube-01 Node: kube-02 Node: kube-03 Node: kube-04
  20. @tammybütow DEVOPS @ NIKE DAY We can increase CPU, Disk,

    IO & Memory consumption to ensure monitoring is setup to catch problems. Important to catch issues before they turn into high severity incidents (unable to purchase new product!) and downtime for customers. RESOURCE CHAOS
  21. @tammybütow DEVOPS @ NIKE DAY Ways to create process chaos

    on purpose: PROCESS CHAOS • Kill one process • Loop kill a process • Spawn new processes • Fork bomb
  22. WHAT ARE OTHER WAYS YOU CAN TURN OFF A SERVER?

    WHAT IF YOU WANT TO TURN OFF EVERY SERVER WHEN IT’S ONE WEEK OLD? @tammybütow DEVOPS @ NIKE DAY
  23. @tammybütow DEVOPS @ NIKE DAY THE MANY WAYS TO KILL

    CONTAINERS • Kill self • Kill a container from the host • Use one container to kill another • Use one container to kills several containers • Use several containers to kill several
  24. The average lifespan of a container is 2.5 days And

    they fail in many unexpected ways. @tammybütow DEVOPS @ NIKE DAY
  25. We can combine different types of chaos engineering experiments to

    reproduce complicated outages. Reproducing outages gives you confidence you can handle it if/when it happens again. @tammybütow DEVOPS @ NIKE DAY
  26. Let’s go back in time to look at some of

    the worst outage stories that kicked off the introduction of chaos engineering. @tammybütow DEVOPS @ NIKE DAY
  27. DROPBOX’S WORST OUTAGE EVER @tammybütow DEVOPS @ NIKE DAY Some

    master-replica pairs were impacted which resulted in the site going down. https://blogs.dropbox.com/tech/2014/01/outage-post-mortem/
  28. UBER’S DATABASE OUTAGE @tammybütow DEVOPS @ NIKE DAY 1.Master log

    replication to S3 failed 2.Logs backed up on the primary 3.Alerts fired to engineer but they are ignored 4.Disk fills up on database primary 5.Engineer deletes unarchived WAL files 6.Error in config prevents promotion — Matt Ranney, Uber, 2015
  29. THERE ARE MANY MORE OUTAGES YOU CAN READ ABOUT HERE:

    https://github.com/danluu/post-mortems @tammybütow DEVOPS @ NIKE DAY