Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
0
64

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017
Tweet

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
400
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
120
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
130
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
190
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
240
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
140
3AM, a survey.
Avatar for Eric Sigler esigler
0
250
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
180
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
私たち準委任PdEは2つのプロダクトに挑戦する ～ソフトウェア、開発支援という”二重”のプロダクトエンジニアリングの実践～ / 20260212 Naoki Takahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
3
350
Exadata Fleet Update
Avatar for oracle4engineer oracle4engineer
PRO
0
1.2k
個人的3D Gaussian Splattingニュースをご紹介 / sharing 3d gaussian splatting news
Avatar for にー兄さん drumath2237
0
250
Claude Code で画面の仕様書を作ろう
Avatar for ZOZO Developers zozotech
PRO
0
310
Amazon Rekognitionで 「信玄餅きなこ問題」を解決する
Avatar for usanchuu usanchuu
1
410
【Claude Code】Plugins作成から始まったファインディの開発フロー改革
Avatar for starfish719 starfish719
0
260
フルスタックGoでスコア改ざんを防いだ話
Avatar for ponyo877 ponyo877
0
470
xDS を活用したサービスディスカバリーで実現するブランチ別 QA 環境の構築手法
Avatar for Kenta Takahashi knwoop
1
160
(技術的には)社内システムもOKなブラウザエージェントを作ってみた！
Avatar for Har1101 har1101
1
460
Agent Skills 入門
Avatar for puku0x puku0x
0
660
Agent Skils
Avatar for ディップ株式会社 dip_tech
PRO
0
200
Generative UI を試そう！A2-UIでAIエージェントにダッシュボードを作らせてみた
Avatar for Kamoshika kamoshika
1
220

Featured

See All Featured
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.1k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
5.3k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
620
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
28
2.5k
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
0
150
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
370
Believing is Seeing
Avatar for Spiro Bolos oripsolob
1
65
A Guide to Academic Writing Using Generative AI - A Workshop
Avatar for Kenji Saito ks91
PRO
0
210
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.1k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
290

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler