Fixing Your Noisy Pager in 500 Easy Steps

You're not sure when it happened, but your pager suddenly seems noisy. You've started dreading your on-call shifts before they begin. You breathe a sigh of relief every time you sleep without interruption. Sound familiar?

Noisy on-call rotas sneak up on us one page at a time - an edge case in a new feature, an alert with too many false positives, processes that get stuck and need restarting. Each of these is easy to tolerate alone, but they quickly add up, leaving you swamped in alert noise and tired from missed sleep.

In this talk we'll explore techniques for digging ourselves out of the hole. We'll look at how to demonstrate the scale of the issue to our colleagues, what to do when the list of problems seems insurmountable, and how to get started with automated remediation in a low-risk way - I promise it's less scary than it sounds.

Transcript

1. Fixing your noisy pager in 500 easy steps
5. This is fi xable
6. Hi
7. sinjo.dev
8. sinjo.dev
9. Infra Engineer
11. Pages per month (2023–24) Aug Sep Oct Nov Dec Jan Feb Mar 0 500 1,000 1,500 Daytime Evening Night ...
12. Pages per month (2023–24) Aug Sep Oct Nov Dec Jan Feb Mar 0 Daytime Evening Night 500 1,000 1,500
13. How did we get here?
14. Pages per month (2023–24) Aug Sep Oct Nov Dec Jan Feb Mar 0 Daytime Evening Night 500 1,000 1,500
15. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan 500 1,000 1,500
16. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan 500 1,000 1,500
17. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan 500 1,000 1,500
18. Problem: convincing yourselves you have a problem
19. We didn’t have those graphs
20. “Not much to report.” “Nothing interesting from my shift.”
21. If nothing interesting happened, why did a computer wake you up every night?
22. The way out of this situation is data
23. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan 500 1,000 1,500
24. So how do we fi x it?
25. 1. Group alerts by name 2. Sort by frequency 3. Categorise each alert
26. 1. Group alerts by name 2. Sort by frequency 3. Categorise each alert
27. 1. Group alerts by name 2. Sort by frequency 3. Categorise each alert
28. Alert Name Count HttpErrorRateHigh 37 TooManyUnhealthyReplicas 21 AutoscalerMaxedOut 5 Alert frequency
29. Categorise how?
30. Alerts that are mostly right vs Alerts that are mostly wrong
31. Alerts that are mostly right vs Alerts that are mostly wrong
32. Easier socially Harder technically
33. Buy in for fi xing bugs and improving automation
34. Alerts that are mostly right vs Alerts that are mostly wrong
35. Easier technically Harder socially
36. Changing people’s minds is harder than changing a couple of lines of PromQL
37. Background noise Alerts that were once useful
38. Compelling reasons - Pager fatigue: we miss real issues - Tiredness: people can’t do their best work - Learned helplessness: we don’t believe we can improve things
39. Compelling reasons - Pager fatigue: we miss real issues - Tiredness: people can’t do their best work - Learned helplessness: we don’t believe we can improve things
40. Compelling reasons - Pager fatigue: we miss real issues - Tiredness: people can’t do their best work - Learned helplessness: we don’t believe we can improve things
41. Compelling reasons - Pager fatigue: we miss real issues - Tiredness: people can’t do their best work - Learned helplessness: we don’t believe we can improve things
42. 2 choices
43. Make it more precise or Delete the alert
44. Make it more precise or Delete the alert
45. Bonus Alerts that are right, but in an annoying way
46. Excessive urgency ‑ Send to Slack or create a ticket
47. Excessive urgency ‑ Send to Slack or create a ticket
48. Flappy alerts ‑ Calculate rate over longer window
49. Flappy alerts ‑ Calculate rate over longer window
50. Pager storms ‑ Use alert grouping/ inhibition
51. Pager storms ‑ Use alert grouping/ inhibition
52. Thorny case: Real problems in software outside your control
53. Inside your company, across team boundaries
54. Usually fi xed by another team? ‑ Currently owned by the wrong team
55. Usually fi xed by another team? ‑ Currently owned by the wrong team
56. Open source and third-party software
57. Work with the maintainers!
58. https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1951
59. In the meantime… protect your team
60. Automated remediation 101
61. Restarting the software fi xes the software (temporarily)
62. Restarting the software fi xes the software (temporarily)
63. Problem shape - Recurring problem: happens regularly - Reliable detection: highly correlated alert - Mechanical fi x: on-caller follows runbook
64. Problem shape - Recurring problem: happens regularly - Reliable detection: highly correlated alert - Mechanical fi x: on-caller follows runbook
65. Problem shape - Recurring problem: happens regularly - Reliable detection: highly correlated alert - Mechanical fi x: on-caller follows runbook
66. Problem shape - Recurring problem: happens regularly - Reliable detection: highly correlated alert - Mechanical fi x: on-caller follows runbook
67. Waking someone up to apply a mechanical fi x is a terrible use of their time
68. Mechanical work is what computers are great at!
69. We wrote a tool: auto-repair
70. Write a non-paging alert that goes o ff before your paging one
71. auto-repair (simpli fi ed) alerts = get(“prom:9090/api/v1/alerts") issues = filter_fixable(alerts) for i in issues do // for most issues, restart process apply_fix(i) end
72. auto-repair (simpli fi ed) alerts = get("prom:9090/api/v1/alerts") issues = filter_fixable(alerts) for i in issues do // for most issues, restart process apply_fix(i) end
73. auto-repair (simpli fi ed) alerts = get("prom:9090/api/v1/alerts") issues = filter_fixable(alerts) for i in issues do // for most issues, restart process apply_fix(i) end
74. auto-repair (simpli fi ed) alerts = get("prom:9090/api/v1/alerts") issues = filter_fixable(alerts) for i in issues do // for most issues, restart process apply_fix(i) end
75. It’s that simple (kinda)
76. It’s that simple (kinda)
77. Runaway automation
78. What the tool doesn’t do is more important than what it does do
79. 3 limitations Don’t restart: - Too many processes with the same issue - The same instance repeatedly - Processes that have already paged
80. Don’t restart: - Too many processes with the same issue - The same instance repeatedly - Processes that have already paged 3 limitations
81. Don’t restart: - Too many processes with the same issue - The same instance repeatedly - Processes that have already paged 3 limitations
82. Don’t restart: - Too many processes with the same issue - The same instance repeatedly - Processes that have already paged 3 limitations
83. This prevents high ones to low tens of pages per week
84. (yes, we still fi le bugs)
85. What did we learn ?
86. You need long-term buy-in
87. Talk about how it impacts customers
88. Embrace hacky fi xes that help you survive
89. Dumb ideas that work aren’t dumb
90. Good things happen if you make it a habit
91. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan 500 1,000 1,500
92. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan Apr Jul 500 1,000 1,500
93. Pages per month (2023–24) Jan 0 Daytime Evening Night Apr Jul Oct Jan Apr Jul 500 1,000 1,500
94. Thank you ✌❤ @planetscaledata sinjo.dev
95. Image credits • Analog Alarm Clock in Morning Sunlight - Ruslan Sikunov - https:// www.pexels.com/photo/analog-alarm-clock-in-morning-sunlight-19188894/
97. Questions? ✌❤ @planetscaledata sinjo.dev