Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Martyrs On Film: learning to hate the #oncallse...
Search
Alice Goldfuss
May 22, 2017
Technology
5
1.1k
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
1.1k
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
1.1k
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.7k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
1k
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
1.2k
Docker in a Flash
alicegoldfuss
2
520
Other Decks in Technology
See All in Technology
少人数でも回る! DevinとPlaybookで支える運用改善
ishikawa_pro
4
1.8k
データエンジニアがクラシルでやりたいことの現在地
gappy50
3
730
AI によるドキュメント処理を加速するためのOCR 結果の永続化と再利用戦略
tomoaki25
0
160
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
2
280
VLMサービスを用いた請求書データ化検証 / SaaSxML_Session_1
sansan_randd
0
130
TypeScript 上達の道
ysknsid25
23
4.9k
東京海上日動におけるセキュアな開発プロセスの取り組み
miyabit
0
200
[MIRU2025]Preference Optimization for Multimodal Large Language Models for Image Captioning Tasks
keio_smilab
PRO
0
120
CSPヘッダー導入で実現するWebサイトの多層防御:今すぐ試せる設定例と運用知見
llamakko
1
270
Claude Codeが働くAI中心の業務システム構築の挑戦―AIエージェント中心の働き方を目指して
os1ma
4
150
AI工学特論: MLOps・継続的評価
asei
10
2k
【CEDEC2025】大規模言語モデルを活用したゲーム内会話パートのスクリプト作成支援への取り組み
cygames
PRO
1
420
Featured
See All Featured
How GitHub (no longer) Works
holman
314
140k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Designing for Performance
lara
610
69k
KATA
mclloyd
30
14k
Producing Creativity
orderedlist
PRO
346
40k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
The Invisible Side of Design
smashingmag
301
51k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
1k
Designing for humans not robots
tammielis
253
25k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
How to Ace a Technical Interview
jacobian
278
23k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None