Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
71
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
410
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
140
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
200
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
250
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
150
3AM, a survey.
Avatar for Eric Sigler esigler
0
260
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
190
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
M5Stack CoreS3とZephyr(RTOS)で Edge AIっぽいことしてみた
Avatar for misoji engineer iotengineer22
0
210
エージェントスキルを作って自分のインプットに役立てよう
Avatar for Yuta Matsumura tsubakimoto_s
0
360
コミュニティ・勉強会を作るのは目的じゃない
Avatar for uutan1108 ohmori_yusuke
0
210
レビューしきれない？それは「全て人力でのレビュー」だからではないでしょうか
Avatar for amixedcolor amixedcolor
0
330
2026年、知っておくべき最新 サーバレスTips10選/serverless-10-tips
Avatar for Serverless Operations slsops
13
5.2k
MLOps導入のための組織作りの第一歩
Avatar for Daisuke Akagawa (Akasan) akasan
0
330
国内外の生成AIセキュリティの最新動向 & AIガードレール製品「chakoshi」のご紹介 / Latest Trends in Generative AI Security (Domestic & International) & Introduction to AI Guardrail Product "chakoshi"
Avatar for NTT docomo Business nttcom
2
850
AI와 협업하는 조직으로의 여정
Avatar for Arawn Park arawn
0
430
「SaaSの次の時代」に重要性を増すステークホルダーマネジメントの要諦 ～解像度を圧倒的に高めPdMの価値を最大化させる方法～
Avatar for KAKEHASHI kakehashi
PRO
2
860
ぼくがかんがえたさいきょうのあうとぷっと
Avatar for Yuuki Yamashita yama3133
0
190
ハーネスエンジニアリングの概要と設計思想
Avatar for sergicalsix sergicalsix
9
5k
AI時代における技術的負債への取り組み
Avatar for Tadashi Shigeoka codenote
1
1.5k

Featured

See All Featured
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
260
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
3.3k
Design in an AI World
Avatar for tapps tapps
1
200
Music & Morning Musume
Avatar for Bryan Veloso bryan
47
7.2k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
26k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
The Anti-SEO Checklist Checklist. Pubcon Cyber Week
Avatar for Ryan Jones ryanjones
0
120
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
274
21k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
58k
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.1k
Claude Code のすすめ
Avatar for schroneko schroneko
67
220k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler