Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
65

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
400
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
130
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
190
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
240
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
140
3AM, a survey.
Avatar for Eric Sigler esigler
0
250
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
180
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
決済サービスを支えるElastic Cloud - Elastic Cloudの導入と推進、決済サービスのObservability
Avatar for suzukij suzukij
2
600
複数クラスタ運用と検索の高度化：ビズリーチにおけるElastic活用事例 / ElasticON Tokyo2026
Avatar for Visional Engineering ＆ Design visional_engineering_and_design
0
130
DevOpsエージェントで実現する!！ AWS Well-Architected（W-A） を実現するシステム設計 / 20260307 Masaki Okuda
Avatar for SHIFT EVOLVE shift_evolve
PRO
3
560
Claude Codeが爆速進化してプラグイン追従がつらいので半自動化した話 ver.2
Avatar for ryu rfdnxbro
0
500
20260311 ビジネスSWG活動報告（デジタルアイデンティティ人材育成推進WG Ph2 活動報告会）
Avatar for OpenID Foundation Japan oidfj
0
260
猫でもわかるKiro CLI(AI 駆動開発への道編)
Avatar for Hiroo Katoh kentapapa
0
100
JAWS DAYS 2026 ExaWizards_20260307
Avatar for ExaWizards exawizards
0
410
OCHaCafe S11 #2 コンテナ時代の次の一手：Wasm 最前線
Avatar for oracle4engineer oracle4engineer
PRO
1
100
SaaSからAIへの過渡期の中で現在、組織内で起こっている変化 / SaaS to AI Paradigm Shift
Avatar for AEON aeonpeople
0
120
2026-03-11 JAWS-UG 茨城 #12 改めてALBを便利に使う
Avatar for SUZUKI Masashi masasuzu
2
350
スクリプトの先へ！AIエージェントと組み合わせる モバイルE2Eテスト
Avatar for Ryo WATANABE error96num
0
150
PMBOK第8版は第7版から何が変わったのか（PMBOK第8版概要解説） / 20260304 Takeshi Watarai
Avatar for SHIFT EVOLVE shift_evolve
PRO
0
200

Featured

See All Featured
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.8k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
1.9k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
Avatar for Vitaly Friedman smashingmag
254
22k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
1.8k
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
300
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
32k
Winning Ecommerce Organic Search in an AI Era - #searchnstuff2025
Avatar for Aleyda Solis aleyda
1
1.9k
Chasing Engaging Ingredients in Design
Avatar for Sebastian Deterding codingconduct
0
140
Between Models and Reality
Avatar for mayunak mayunak
2
230
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
120
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
4
250

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler