Martyrs On Film: learning to hate the #oncallselfie

Transcript

1. Martyrs on Film
2. Hi! I’m Alice I like systems and Twitter and tea.
3. Hi! I’m Alice And not getting paged.
6. Benefits of On-Call • Hones troubleshooting skills • Forces you to identify the weak points in your systems • Teaches you what is and isn’t production-ready
7. Team bonding
15. PagerDuty
18. PagerDuty
19. PagerDuty New Year’s Eve
23. PagerDuty New Year’s Eve
24. PagerDuty New Year’s Eve S3 Outage
29. Me
34. A totally normal on-call routine • Don’t leave house except to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
39. Be heroes
40. Prepare for battle
41. Naomi Orwin Writer
42. “Action scenes stop the plot.” - Naomi Orwin
43. Pages stop the plot of your career
45. Miserable on-call professionals • Have terrible work/life balance • Are supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
47. Red flag: too few owning too much
48. Centralia infrastructure
52. Red flag: bandaids
54. • Bump thresholds • Snooze pages • Delays
55. Red flag: no visibility
56. Systems visibility
57. Team visibility
58. Too many pages
59. Average # of weekly pages during WORST on-call
60. Average # of weekly pages during WORST on-call
61. Average # of weekly pages during WORST on-call
62. Average # of weekly pages during WORST on-call
64. Average # of weekly pages during BEST on-call
65. Average # of weekly pages during BEST on-call
66. Average # of weekly pages during BEST on-call
67. How do we get there?
68. Notification cleanup
69. Actionable alerts
70. Actionable Alerts • Something breaks • Customers notice • I am the best person to fix it • I need to fix it immediately
71. Cluster alerts
73. Devs on-call
75. “If a developer is good, being ‘on call’ just means having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
76. “Fixed this for him! ‘Put your developers on call. You’ll be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
77. “If your org has change control working properly, if code breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
78. The right tool
79. Work together
81. Start small
83. Your people will burn out before your company does
87. Where does that leave #oncallselfie?
99. “Why are you getting paged so much?”
100. Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you