Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Martyrs On Film: learning to hate the #oncallse...
Search
Alice Goldfuss
May 22, 2017
Technology
5
1.1k
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
1k
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
1k
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.6k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
910
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
1k
Docker in a Flash
alicegoldfuss
2
450
Other Decks in Technology
See All in Technology
プロダクトエンジニアが活躍する環境を作りたくて 事業責任者になった話 ~プロダクトエンジニアの行き着く先~
gimupop
1
480
GitHub Universe: Evaluating RAG apps in GitHub Actions
pamelafox
0
180
ガバメントクラウド先行事業中間報告を読み解く
sugiim
1
1.4k
スプリントゴールにチームの状態も設定する背景とその効果 / Team state in sprint goals why and impact
kakehashi
2
100
チームを主語にしてみる / Making "Team" the Subject
ar_tama
4
310
独自ツール開発でスタジオ撮影をDX!「VLS(Virtual LED Studio)」 / dx-studio-vls
cyberagentdevelopers
PRO
1
180
カメラを用いた店内計測におけるオプトインの仕組みの実現 / ai-optin-camera
cyberagentdevelopers
PRO
1
120
Oracle Cloud Infrastructureデータベース・クラウド:各バージョンのサポート期間
oracle4engineer
PRO
27
12k
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
49k
コンテンツを支える 若手ゲームクリエイターの アートディレクションの事例紹介 / cagamefi-game
cyberagentdevelopers
PRO
1
130
大規模データ基盤チームのオンプレTiDB運用への挑戦 / dpu-tidb
cyberagentdevelopers
PRO
1
110
顧客が本当に必要だったもの - パフォーマンス改善編 / Make what is needed
soudai
24
6.8k
Featured
See All Featured
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
Six Lessons from altMBA
skipperchong
26
3.5k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
Docker and Python
trallard
40
3.1k
The Invisible Side of Design
smashingmag
297
50k
GraphQLとの向き合い方2022年版
quramy
43
13k
The Art of Programming - Codeland 2020
erikaheidi
51
13k
Build The Right Thing And Hit Your Dates
maggiecrowley
32
2.4k
RailsConf 2023
tenderlove
29
880
Navigating Team Friction
lara
183
14k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None