Chaos management during a major incident

This is a short talk that I gave about Incident Response and how PagerDuty does it's incident response.
I gave two versions of this talk one at dotScale 2017 in Paris and the other at Full Stack Fest, Barcelona

Transcript

1. Aish Raj Dahal, So.ware Engineer, PagerDuty Chaos management during a major incident
2. sudo rm -rf /
3. Failure free operations require experience with failure
 - Richard Cook
4. A story about failure
5. Chapter I: This is fine
6. Almost every engineer I knew was on the call
7. Someone dialed in from a Bar
8. Most people were doing the same task
9. We had no clue where to start
10. Chapter II: Dark and Stormy night
11. “It was a dark and stormy night; the rain fell in torrents — except at occasional intervals, when it was checked by a violent gust of wind…..” - Almost every clueless person on the call
12. Nobody to coordinate
13. Chapter III: The exec-swoop
14. “Can you send me a spreadsheet with a list of affected customers ?”
15. Chapter IV: The morning aOer
16. What’s wrong with the picture ?
17. Define, Prepare, Measure
18. Failures should be unique, if not automate the response.
20. Single responsibility principle
21. Role: The Subject Matter Expert
22. Role: The Incident Commander
23. Notify that this is a major incident
24. Verify that all SMEs are present
25. Divide and conquer
26. Communicate effectively
27. Avoid the bystander effect
28. “Please say yes, if you think it is a good idea to do so.”
29. “Is there any strong objection to that ?”
30. Role: The Deputy
31. Assist the Incident Commander
32. Get all Subject Matter Experts up to speed about what’s happening
33. Liaise with stakeholders
34. Role: The Scribe
35. Documents the timeline of an incident as it progresses
36. Role: Customer Liaison
37. Notify customers about the incident
38. Keep the Incident Commander apprised of any relevant customer information
39. Transfer of command if necessary
40. Blameless post-mortem
41. You can’t fire you way to reliability
42. Review
43. happens, prepare for it
44. Develop on-call empathy
45. People are your most valuable asset. Don’t burn them out doing something that can be automated.
46. aishrajdahal
47. Appendix • The Big Red Button from Flickr by włodi CC SA • Upside down from Flickr by Akimasa Harada (CC SA) • Geography from Flicker (CC SA) • Gene Kranz’s image on Public Domain • That’s all folks image on the Public Domain
48. References • Cook, Richard I. "How complex systems fail." Cognitive Technologies Laboratory, University of Chicago. Chicago IL (1998). • Incident Response, PagerDuty Incident Response Docs, https:/ /response.pagerduty.com/ • Allspaw, John. "Fault injection in production." Communications of the ACM 55.10 (2012): 48-52. • Krishnan, Kripa. "Weathering the Unexpected." Commun. ACM 55.11 (2012): 48-52. • Limoncelli, Tom, et al. "Resilience engineering: learning to embrace failure." Communications of the ACM 55.11 (2012): 40-47.