Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
72
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
410
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
140
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
210
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
250
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
150
3AM, a survey.
Avatar for Eric Sigler esigler
0
270
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
190
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
Gaussian Splattingの実用化 - 映像制作への展開
Avatar for GPU UNITE gpuunite_official
0
200
AI全盛の今だからこそ、あえてもう一度振り返るAPIの基礎
Avatar for Makky12 smt7174
3
140
TypeScriptはどのようにどこまで推論できるのか ─ とにかく as は禁止で
Avatar for ypresto ypresto
0
200
Cortex(Code) を ML モデルの 精度改善サイクルに組み込む.pdf
Avatar for Yuki Fushinuki oimo23
0
240
ワールドカフェ再び、そしてゴール・ルール・ロール・ツール / World Café Revisited, and the Goals-Rules-Roles-Tools
Avatar for Kenji Saito ks91
PRO
0
180
How to learn AWS Well-Architected with AWS BuilderCards: Security Edition
Avatar for Kosuke ENOMOTO coosuke
PRO
0
180
Claude Code で使える DuckDB Skills を試してみた / DuckDB Skills and Claude Code
Avatar for MasahiroKawahara masahirokawahara
1
1.3k
AWSアップデートから考える継続的な運用改善
Avatar for Toru_Kubota toru_kubota
2
310
AI時代に、 データアナリストがデータエンジニアに異動して
Avatar for jackojacko_ jackojacko_
0
1.1k
20260515 ⾃分のアカウントとプライバシーを守る認証と認可の話〜利⽤者向け〜
Avatar for OpenID Foundation Japan oidfj
0
790
サイボウズ、プラットフォームエンジニアリング始めるってよ ― プラットフォームチームの事業貢献と組織アラインメントの強化
Avatar for Shin'ya Ueoka ueokande
0
120
PdM・Eng・QAで進めるAI駆動開発の現在地/aidd-with-pdm-eng-qa
Avatar for butatamasoba / Shota Kusaba / 草場翔太 shota_kusaba
0
260

Featured

See All Featured
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
370
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
220
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.7k
Exploring the relationship between traditional SERPs and Gen AI search
Avatar for Ray Grieselhuber raygrieselhuber
PRO
2
4k
Lightning talk: Run Django tests with GitHub Actions
Avatar for Sarah Abderemane sabderemane
0
180
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
2
1k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
360
30k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
122
21k
WCS-LA-2024
Avatar for Leonardo Collado-Torres lcolladotor
0
590
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
304
21k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler