Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
77
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
410
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
140
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
160
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
230
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
280
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
160
3AM, a survey.
Avatar for Eric Sigler esigler
0
290
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
210
Engineering for Engineers
Avatar for Eric Sigler esigler
0
120

Other Decks in Technology

See All in Technology
SmartHR Engineering Team Deck
Avatar for SmartHR, Inc. smarthr
1
3.1k
LLM Internals: 언어 모델의 계보와 알고리즘 진화 (2023~2026)
Avatar for Jeongkyu Shin inureyes
PRO
1
800
AIに狂うスタートアップが、あえて「人との協働」に全振りした新卒エンジニア研修 / New Graduate Engineer Training at a Startup Accelerating AI Adoption
Avatar for akira ohno ohnoeight
0
160
まちスペース®とデジタルツインと「まちづくり」
Avatar for hiro_OGI hiro_ogi
0
130
名刺メーカーDevグループ 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
1.2k
【書籍出版記念】 10周回って、エージェント開発は RAGがすべてだった。〜RAGの歴史と開発現場で見えた実践知〜
Avatar for 為藤アキラ akiratameto
3
420
DatadogのBits Chatが開発組織にもたらしたもの / What Bits Chat Has Brought Us
Avatar for SMS tech sms_tech
1
290
LanceDB入門
Avatar for Tomoko Uchida mocobeta
9
640
【GCC2026】鉄拳8でのVFX開発事例
Avatar for Bandai Namco Studios Inc. bandainamcostudios
PRO
0
120
Kiro入門｜仕様駆動開発で変わるAI時代の開発スタイル
Avatar for 工藤皓夢 cmkudo
0
260
自宅NWにISR4331を導入してみた話
Avatar for okaits#7534 okaits
0
120
ホームラボ紹介
Avatar for 世良泰明 y_sera15
0
220

Featured

See All Featured
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
350
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
63
45k
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
Avatar for Mine Cetinkaya-Rundel minecr
1
350
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
Avatar for soudai sone soudai
PRO
68
57k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
Done Done
Avatar for chrislema chrislema
186
16k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
3k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
270
14k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1.5k
Scaling GitHub
Avatar for Zach Holman holman
464
140k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
6.1k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler