SRE NEXT 2022: Sensible Incident Management for Software Startups

Transcript

1. SRE NEXT 2022 Sensible Incident Management for So4ware Startups Takayuki Watanabe @Launchable, Inc.
2. Who? Name: Takayuki Watanabe Affiliation: Launchable, Inc. Role: Software Engineer Sns: Blog: blog.takanabe.tokyo GitHub: takanabe Twitter: @takanabe_w Interests: - Developer Productivity - Site Reliability Engineering - Sustainability Engineering SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 2
3. Your takeaways You can understand: • Incident management has a life cycle. • Incident response roles and structures exist to embody 3T mental models. • Choosing strategies and tools makes incident managements at startups sensible. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 3
4. Out of scope • Fundamental SRE terminology (e.g. SLO, SLI, Error budget, Postmortem) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 4
5. Disclaimer • This session refers a lot of exis0ng incident management and SRE prac0ces. • But contains a lot of opinionated ideas and philosophy as well. • So, the ideas might contradict to some people's. • Let's discuss on TwiAer using #srenext with @takanabe_w SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 5
6. Today's agenda • About Launchable • Does a startup need incident management? • Dissect incident management prac8ces. • 3T mental models and life cycles • How can we improve incident management? • Choosing right strategies and tools SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 6
7. Chapter 1: About Launchable SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 7
8. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 8
9. What is Launchable? A SaaS accelera)ng so.ware development cycles. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 9
10. What is Launchable? Current focus is machine learning based test selec0ons by: • Predic(ng a meaningful subset of tests. • Iden(fying flaky tests. • Visualizing test trends with metrics. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 10
11. What is Launchable? e.g. Reordering tests based on likelihood of failures. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 11
12. Our team size • Launchable is a startup • 2 CEOs + 15 employees • So3ware engineer (7 people) • Product manager • Marke>ng • Sales • etc... SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 12
13. Phases and the number of so0ware engineers Note: the numbers are es/mated by the presenter based on previous experiences. • Phase 0: Founding ~ 4 so3ware engineers • Phase 1: 5 ~ 10 so3ware engineers • Phase 2: 11 ~ so3ware engineers SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 13
14. My SRE NEXT 2022 is about ... SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 14
15. Incident management at so#ware startups SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 15
16. Does a startup need incident management? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 16
17. Yes, it's obvious if products have customers. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 17
18. Do you have enough engineering members? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 18
19. No! but... SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 19
20. Learning from previous careers1 • I've worked at various sizes and stages. • Company A: +300,000 people • Company B: +400 people (Joined when they had +300 people) • Company C: +150 people (Joined when they only had less than 10 people) • Product developments are always the highest priority concerns. • OperaHon improvement != Product development velocity degradaHon. • We will never have enough engineering members to improve opera;ons. Never. 1 SRE NEXT 2020: Designing fault-tolerant microservices with SRE and circuit breaker centric architecture SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 20
21. Are speed and quality trade-off? • I personally don't think so 2 3. • I believe sensible incident management accelerates our development velocity. 3 A Philosophy of So.ware Deisgn, Chapter 3: Working Code Isnt' Enough, pp. 13 - 18. 2 mar&nFowler.com: Is High Quality So;ware Worth the Cost? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 21
22. Can we reframe the original ques3on? • We want to reframe "Does a startup need incident management?" to: • Which incident management processes won't change even for rapid developments? • Which processes should we improve? • Let's dissect incident management prac=ces in the industry. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 22
23. Chapter 2: Dissect incident management prac/ces SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 23
24. What is incident management? Incident management • High level and overall process for handling incidents in an organiza5on. Incident response • Part of incident management for actual technical steps including detec5on, repor5ng, mi5ga5on, and recovery during incidents. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 24
25. Exis%ng prac%ces SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 25
26. e.g. Terminology SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 26
27. Examples of terminology 4 • CAN Reports • Deputy • Execu3ve Swoop • Grenade Thrower • Incident Commander (IC) • Resolver • Severity • Scribe • Subject Ma4er Expert (SME) 4 h$ps:/ /response.pagerduty.com/training/glossary/ SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 27
28. e.g. Roles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 28
29. Examples of roles at Google 5 6 6 Anatomy of an Incident Google’s Approach to Incident Management for Produc;on Services, Chapter 4: Mi;ga;on and Recovery, pp. 31-32. 5 Google SRE Workbook, Chapter 9: Incident Response SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 29
30. Examples of roles at PagerDuty 7 8 8 Google SRE Workbook, Chapter 9: Incident Response 7 PagerDuty Incident Response Documenta6on, Different Roles - SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 30
31. Too much! SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 31
32. Can we translate these prac.ces into more higher level concepts? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 32
33. Chapter 3: 3T mental models and life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 33
34. Examples of roles at PagerDuty SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 34
35. Command role • Responsibility is managing incident responses to align in organiza5ons. • Understand ongoing opera5ons • Understand who is doing what • Delegate sub-commander responsibility to others if necessary. • Make incident response tangible. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 35
36. Liason role • Responsibility is smooth repor1ng and communica1ons. • For both internally and externally. • Make incident response transparent. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 36
37. Opera&on role • Responsibility is actual technical ac2vi2es to solve issues. • Focus on triage, analysis, mi2ga2on and recovery. • Communica2on with rest of organiza2ons is not a primary concern. • In many cases, operators produce root causes of incidents but don't blame them. • Nobody wants to cause incidents. • All par2cipants focus on assigned roles based on chain of trust. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 37
38. 3T mental models for incident response The incident response roles embody 3T mental models. • Transparency • Keep informa-on of incident responses reachable for everybody. • Tangibility • Manage status of incidents. • Manage who handles what. • Trust • Believe everybody makes best efforts during incidents. • Don't blame anybody because nobody wants to cause incidents. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 38
39. High level view of incident management cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 39
40. High level view of incident management cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 40
41. High level view of incident management cycles Examples: • Incident management policy • Documenta3on • Repor3ng mechanism • Observability • Aler.ng policy • Incident response training SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 41
42. High level view of incident management cycles Examples: • Aler&ng • Triage • Root-cause analysis • Escala'ons • Opening war rooms SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 42
43. High level view of incident management cycles Examples: • Rollback deployment (mi3ga3on) • Kill slow queries (mi3ga3on) • Fix bug (recovery) • Add index to tables (recovery) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 43
44. High level view of incident management cycles Examples: • Addi%onal triage • Prepara%on for postmortems • Postmortems • Handle ac*on items raised at postmortems SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 44
45. Postmortem vs FtS h"ps:/ /twi"er.com/takanabe_w/status/1510943694467186699 SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 45
46. Chapter 4: How can we improve incident management? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 46
47. Where should we invest our 0me? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 47
48. Where should we invest our 0me? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 48
49. Key %mes of incident response SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 49
50. Key %mes of incident response • Time to detect (TTD) • Time to engagement (TTE) • Time to fix (TTF) • Time to repair/recovery (TTR) • Time between failures (TBF) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 50
51. Time to detect (TTD) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 51
52. Time to engagement (TTE) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 52
53. Time to fix (TTF) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 53
54. Time to recovery (TTR) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 54
55. Time between failures (TBF) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 55
56. Which &me do we want to improve? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 56
57. TTD at Launchable Current status • We've already had several detec0on mechanisms using Datadog and Sentry. Solu%on • Introduc*on of SLO and Error Budget makes our aler*ng criteria more clear. • But don't forget "Law of diminishing returns" to make decisions. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 57
58. TTE at Launchable Current status • Easy enough to no.ce during office hours at Slack channels. • We don't have on-call rota.ons ATM, which makes TTE uncontrollable. Solu%on • Apply follow-the-sun strategy to cover wide-range hours. • Introducing on-call rota:ons and pager. • But we don't feel it's necessary now. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 58
59. TTF at Launchable Current status • We don't have enough observability mechanisms • Depending on each developer's debug skill • During this window, developers cannot spend .me on product developments. Solu%on • Introducing more team-shared observability dashboards. • Introducing more observability mechanism to drill down root causes. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 59
60. Which &me do we want to improve? TTF improvement brings us high returns with small efforts. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 60
61. Which do we op+mize? MTTR vs MTBF • Short MTTR and long MTBF are the best • Short MTTR but short MTBF = Incidents frequently occur but are recovered quickly. • Long MTTR but long MTBF = Incidents don't occur frequently but once occur, they aren't recovered soon. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 61
62. Startups should focus on MTTR improvement • There is no evolu.on without high cadence itera.ons at startups. • TTD and TTE are difficult to improve for us. • Reducing TTF results in reducing MTTR. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 62
63. Do we have other +mes we haven't ar+culated? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 63
64. Hidden key )mes of incident life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 64
65. Hidden key )mes of incident life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 65
66. Hidden key )mes of incident life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 66
67. Hidden key )mes of incident life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 67
68. Hidden key )mes of incident life cycles SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 68
69. Hidden key )mes of incident life cycles Don't underes,mate the ,mes we spend as post-incident ac,vi,es. • Time to (addi,onal )triage (TTT) • Time to learn (TTL) • Time to improvement (TTI) • Time to prepara,on (TTP) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 69
70. Power ques*on: Which process do you hate? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 70
71. Which process do you hate? I personally don't want to spend 0me on the following processes. • Addi$onal triages to dig root causes. • Prepara$on for learning ( != I don't like joining postmortem sessions). • Maintainance of incident management processes. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 71
72. Why addi(onal triages? • Startups don't have enough observability mechanisms. • We some:mes cannot find root causes (this is acceptable). • We tend to spend a lot of :me here in that situa:on. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 72
73. Why prepara)on for learning? • Prepara'on for team-wise learning sessions take 'me. • Documen'ng for Postmortems. • Copy & paste dances to create 'meline sca=ered various places. • Timeline needs to consider 'me-zones. • There is a gravity which prevent people from announcing incident casually. • For starups, the most important ac'vi'es are learning as a team. • If TTL is long, people cannot announce incidents casually. • As a result, postmortems ruin short MTTR with high cadence learning itera'ons. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 73
74. Why maintenance of incident management processes? • Maintainance of incident management processes contains: • Upda.ng incident management policy. • Improving incident management structures. • Upda.ng documents. • Training people to align with the updates. • Characteris.cally, incidents don't occur frequently, • Too tough to memorize incident response processes for everybody. • In urgent situa.on, people don't read documents. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 74
75. Can we reduce TTI? • It's depending on ac0on items coming from postmortems. • No teams can handle all ac0on items we discussed during postmortems. • Common an0-pa<ern: people create too many ac0on items and assign without priori0es. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 75
76. Approach for unbalanced ac0on items • Think of engineering members' capacity • Priori7ze and classify the work9 10 10 Anatomy of an incident management, Chapter 5 9 Postmortem Ac,on Items: Plan the Work and Work the Plan, USENIX SRECon 2017 SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 76
77. Importance / Size / Urgency (ISU) Matrix • Assignee's confidence is also valuable to declare. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 77
78. ISU Matrix on GitHub Projects SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 78
79. My focus is reduc-on of TTT, TTL, and TTP SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 79
80. Chapter 5: Choosing right strategies and tools SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 80
81. Phases and the number of so0ware engineers Note: the numbers are es/mated by the presenter based on previous experiences. • Phase 0: Founding ~ 4 so3ware engineers • Phase 1: 5 ~ 10 so3ware engineers • Phase 2: 11 ~ so3ware engineers SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 81
82. Let's reframe the original ques3on again • Reframe "Does a startup need incident management?" • At startups, how can we: • Build an incident management structure enforcing the 3T mental models? • Improve the ":mes" of the incident management life cycle? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 82
83. Evolu&on of incident management at Launchable Improvement target Ac/ons from phase 0 to 1 Ac/ons from phase 1 to 2 Transparency - Encourage push communica3on - Encourage pull communica3on - Create war rooms - Share status pages Tangibility - Automate parts of incident response flow - Automate en3re incident response flow - Introduce incident lead role Trust - Introduce blameless culture - Split lead and opera3on roles for complex incidents Time to Engagement (TTE) - Automate incident announcements - Automate en3re incident response flow - Introduce on-call rota3ons - Expand follow-the-sun coverages Time to Fix (TTF) - Introduce observability - Improve observability Time to Triage (TTT) - Introduce observability - Improve observability Time to Learn (TTL) - Introduce postmortem template - Generate postmortem Time to Prepara3on (TTP) - Create incident management policies - Enforce incident management policies - Self-service incident response trainings SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 83
84. Phase 0: Founding ~ 4 so2ware engineers SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 84
85. No strategy • Product does not have customers. • We don't need incident responses. • Build incident management structure based on product growth. • All members do everything if necessary. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 85
86. Incident management system SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 86
87. Phase 1: 5 ~ 10 so-ware engineers SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 87
88. Environmental changes from phase 0 to 1 • When products have customers, we need an incident management. • The more so8ware engineers join, the more incidents happen. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 88
89. Strategy Make everything simple and easy to follow SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 89
90. Incident management changes from phase 0 to 1 Improvement target Ac/ons from phase 0 to 1 Ac/ons from phase 1 to 2 Transparency - Encourage push communica/on - Encourage pull communica3on - Create war rooms - Share status pages Tangibility - Automate parts of incident response flow - Automate en3re incident response flow - Introduce incident lead role Trust - Introduce blameless culture - Split lead and opera3on roles for complex incidents Time to Engagement (TTE) - Automate incident announcements - Automate en3re incident response flow - Introduce on-call rota3ons - Expand follow-the-sun coverages Time to Fix (TTF) - Introduce observability - Improve observability Time to Triage (TTT) - Introduce observability - Improve observability Time to Learn (TTL) - Introduce postmortem template - Generate postmortem Time to Prepara3on (TTP) - Create incident management policies - Enforce incident management policies - Self-service incident response trainings SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 90
91. Incident management system (phase 0 to 1) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 91
92. Incident management system (phase 0 to 1) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 92
93. Founda'on of incident management policies • We maintain policies on Confluence. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 93
94. Founda'on of incident management policies • We maintain policies on Confluence. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 94
95. Automa'on of incident escala'ons • We escalate incidents using Slack Workflow. • We handle incidents in Slack channel and Google Meet. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 95
96. Automa'on of incident escala'ons • We escalate incidents using Slack Workflow. • We handle incidents in Slack channel and Google Meet. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 96
97. Automa'on of incident escala'ons • We escalate incidents using Slack Workflow. • We handle incidents in Slack channel and Google Meet. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 97
98. Introduc)on of postmortem • We keep all postmortems on Confluence. • We create a new postmortem page using a Confluence template feature. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 98
99. Introduc)on of postmortem • We keep all postmortems on Confluence. • We create a new postmortem page using a Confluence template feature. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 99
100. Very simple and easy to follow SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 100
101. Postmortems as strong fact-based data • Even we cannot solve root causes, you can use the postmortems as data. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 101
102. Can we improve the incident management? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 102
103. e.g. Does this what human should take care? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 103
104. e.g. Does this what human should take care? • We don't have solid policy but policy does not scale. • Employees are living in Japan and US. • Sharing all informa>on on Slack is easy to miss. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 104
105. e.g. Do we need roles? SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 105
106. Phase 2: 11 ~ ?? so-ware engineers SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 106
107. Environmental changes from Phase 1 to 2 • Our products have more customers. • The more so3ware engineers join, the more incidents happen. • Increase of employees and >me-zone gaps make sync and push-style communica>ons tough. • In the first place, Launchable encourages async and wriEen communica>ons. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 107
108. Strategy • Enforce incident management policies by so4ware not by documents. • Involve appropriate people based on pull-style communica8ons. • Use the current tool chains in the company. • Too many new tools degrade teams' performance. • Use Slack as interac?ve communica?on places to keep flow info. • Use Confluence to keep stock info (non-urgent communica?ons). SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 108
109. Incident management system (phase 1 to 2) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 109
110. Incident management system (phase 1 to 2) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 110
111. Incident management system (phase 1 to 2) SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 111
112. SaaS: incident.io • h#ps:/ /incident.io/ SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 112
113. SaaS: Blameless • h#ps:/ /www.blameless.com/product/incident-resolu8on SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 113
114. SaaS: Datadog Incident • h#ps:/ /www.datadoghq.com/blog/incident-response-with-datadog/ SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 114
115. SaaS: Grafana Incident • h#ps:/ /go2.grafana.com/incident-beta-interest.html SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 115
116. OSS: monzo/response OSS version of incident.io • h#ps:/ /github.com/monzo/response • h#ps:/ /monzo.com/blog/2019/07/08/how-we-respond-to-incidents/ SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 116
117. in-house tool: Slack App + Web App • It's not so difficult to implement Slack App and Web App for this purpose. • But... I want to use my ?me for other stuff. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 117
118. SaaS vs OSS vs in-house tool • We want to maximize developers' disposal 5me for product developments. • We don't want to increase cogni5ve loads. • OSS and in-house tool needs code and document maintenance. • OSS and in-house tool needs evangelical ac5vi5es for this type of tools. • Use SaaS if money allows (Buy, Not Build) • Salaries for so@ware engineers are way more expensive than SaaS cost. • SaaS improves their features as their business. • SaaS maintains documents as product features. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 118
119. incident.io covers wide improvement targets Improvement target Ac/ons from phase 0 to 1 Ac/ons from phase 1 to 2 Transparency - Encourage push communica3on - Encourage pull communica/on - Create war rooms - Share status pages Tangibility - Automate parts of incident response flow - Automate en/re incident response flow - Introduce incident lead role Trust - Introduce blameless culture - Split lead and opera/on roles for complex incidents Time to Engagement (TTE) - Automate incident announcements - Automate en/re incident response flow - Introduce on-call rota3ons - Expand follow-the-sun coverages Time to Fix (TTF) - Introduce observability - Improve observability Time to Triage (TTT) - Introduce observability - Improve observability Time to Learn (TTL) - Introduce postmortem template - Generate postmortem Time to Prepara3on (TTP) - Create incident management policies - Enforce incident management policies - Self-service incident response trainings SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 119
120. Central channel for all incidents incident.io can share all incidents in the specified Slack channel. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 120
121. Dedicated war rooms (Slack channel) incident.io handles all tasks we want to complete for incident response ini4aliza4ons. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 121
122. Dedicated war rooms (Slack channel) incident.io can assist incident responses. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 122
123. Dedicated Slack channel (closing incident) At the end of incident responses, incident.io tells us what we need to be done next. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 123
124. Status updates at central channel incident.io automa-cally syncs the latest status of incidents at the central channel. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 124
125. Postmortem genera,on incident.io can collect ,melines from war rooms and generates postmortems. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 125
126. Postmortem genera,on We can generate a postmortem documents using incident.io. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 126
127. Postmortem genera,on We can collect *melines from dedicated Slack channels. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 127
128. Postmortem genera,on SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 128
129. Self-training mode incident.io has a mode to walk though dummy incident responses on Slack. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 129
130. Introduc)on of lead role • We need communica-on leads when incidents are complex • However, for most of incident, a single person can be responsible for opera-ons and communica-ons. • So, adding a lead role only is prudent so we don't make incident managements overly complex. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 130
131. We have more rooms to improve! SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 131
132. Recap • Incident management has a life cycle. • Prepara6on -> Detec6on -> Recovery -> Post-incident ac6ons -> Prepara6on • Incident response roles and structures exist to embody 3T. • Transparency • Tangibility • Trust • Choosing strategy and tools makes incident managements at startups sensible. SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 132
133. Thanks SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 133
134. References SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 134
135. Incident management 1. Atlassian: Understanding incident response roles and responsibili5es, h8ps:/ /www.atlassian.com/incident-management/incident- response/roles-responsibili5es 2. PagerDuty Incident Response Training, h8ps:/ /response.pagerduty.com/training/overview/. 3. Anatomy of an Incident, Ayelet Sachto, Adrienne Walcer, and Jessie Yang, 2022. 4. US Federal Emergency Management Agency, Emergency Management Ins5tute ICS Resource Center, h8ps:/ /training.fema.gov/ emiweb/is/icsresource/. 5. The Na5onal Ins5tute of Standards and Technology SP 800-61, Computer Security Incident Handling Guide, h8p:/ /dx.doi.org/ 10.6028/NIST.SP.800-61r2. 6. Introduc5on: Incident Response overview, Gov UK Na5onal Cyber Security Centre, h8ps:/ /www.ncsc.gov.uk/collec5on/incident- management/incident-response 7. Incident Review and Postmortem Best Prac5ces, h8ps:/ /newsle8er.pragma5cengineer.com/p/incident-review-best-prac5ces 8. Incident Review Prac5ces [The Pragma5c Engineer Newsle8er], h8ps:/ /docs.google.com/spreadsheets/d/1GPINipdf- l2H05iKOUbpkrqwlZ61ZCJDnwY5iE8LtRM/edit#gid=0 SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 135
136. SRE 1. Google SRE book Chapter 14 - Managing Incidents, h>ps:/ /sre.google/sre-book/managing-incidents/ 2. Postmortem AcEon Items: Plan the Work and Work the Plan, Sue Lueder and Betsy Beyer (Google), USENIX SRECon 2017, h>ps:/ /www.usenix.org/conference/srecon17americas/program/presentaEon/lueder. 3. Google SRE book Chapter 15 - Postmortem Culture: Learning from Failure, h>ps:/ /sre.google/sre-book/postmortem-culture/. 4. Postmortem Metadata Index, h>ps:/ /postmortems.app/. 5. The Art of SLOs, Google Site Reliability Engineering, h>ps:/ /sre.google/resources/pracEces-and-processes/art-of-slos/ 6. danluu/post-mortems: A collecEon of postmortems, h>ps:/ /github.com/danluu/post-mortems. 7. Great Incident Review Examples, The PragmaEc Engineer, h>ps:/ /blog.pragmaEcengineer.com/postmortem-best-pracEces/#great- incident-review-examples SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 136
137. DevOps performance metrics 1. Accelerate: The Science of Lean So4ware and DevOps: Building and Scaling High Performing Technology OrganizaDons, 2018. 2. GoogleCloudPlaKorm/fourkeys, hNps:/ /github.com/GoogleCloudPlaKorm/fourkeys 3. Are you an Elite DevOps performer? Find out with the Four Keys Project, Google Cloud, hNps:/ /cloud.google.com/blog/products/ devops-sre/using-the-four-keys-to-measure-your-devops-performance 4. DORA DevOps Quick Check., hNps:/ /www.devops-research.com/quickcheck.html SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 137
138. SaaS and OSS 1. Datadog, h,ps:/ /www.datadoghq.com/blog/incident-response-with-datadog/ 2. incident.io, h,ps:/ /incident.io/ 3. jeli, h,ps:/ /www.jeli.io/ 4. monzo/response, h,ps:/ /monzo.com/blog/2019/07/08/how-we-respond-to-incidents 5. Etsy/morgue, h,ps:/ /github.com/etsy/morgue SRE NEXT 2022 / presented by Takayuki Watanabe@Launchable, Inc. (May 15, 2022) 138