Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
74
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
410
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
130
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
150
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
220
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
270
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
150
3AM, a survey.
Avatar for Eric Sigler esigler
0
280
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
200
Engineering for Engineers
Avatar for Eric Sigler esigler
0
110

Other Decks in Technology

See All in Technology
Text-to-SQLをAgentCoreで実現し、生成されるSQLの精度を定量的に評価する
Avatar for やくも yakumo
2
450
Fabricをフル活用する AI Agent Hub -製造業特化AIエージェントの設計
Avatar for AIxIoTビジネス共創ラボ iotcomjpadmin
0
190
AI・ロボティクスと自動化社会 / AI, Robotics, and the Automated Society
Avatar for Kenji Saito ks91
PRO
0
150
初めてのDatabricks勉強会
Avatar for Takaaki Yayoi taka_aki
2
230
AIをフル活用してオンコール機能のプロトタイプを2日で作った話 / Building an AI-Powered On-Call Prototype in Just Two Days
Avatar for Narimichi Takamura nari_ex
0
170
product engineering with qa
Avatar for Nealle nealle
0
140
アラート調査向けAIエージェントの本番導入とその後/AI Agents for Alert Investigation: Production Deployment and After
Avatar for SadayoshiTada taddy_919
1
320
Claude Codeとハーネスについて考えてみる
Avatar for Oikon oikon48
13
5.6k
“ID沼入口” - 基本とセキュリティから始める、考え続けるためのID管理技術勉強会 告知&イントロ
Avatar for ritou ritou
0
380
技術・能力を向上する原理原則 #きのこセッションa #きのこ2026
Avatar for bash0C7 bash0c7
0
200
AIエージェントとPhysical AIが拓く製造業の変革（ハノーバーメッセリキャップ）
Avatar for AIxIoTビジネス共創ラボ iotcomjpadmin
0
210
Docker Desktop不要の時代が来る？ WSL標準の「wslc」で Linuxコンテナを動かしてみた.
Avatar for ueponx ueponx
0
580

Featured

See All Featured
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.3k
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
1k
The Mindset for Success: Future Career Progression
Avatar for Greg Gifford greggifford
PRO
0
370
New Earth Scene 8
Avatar for Sydney Stone popppiees
3
2.4k
More Than Pixels: Becoming A User Experience Designer
Avatar for Michelle Schulp Hunt marktimemedia
3
460
Future Trends and Review - Lecture 12 - Web Technologies (1019888BNR)
Avatar for Beat Signer signer
PRO
0
3.6k
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
Avatar for Tech SEO Connect techseoconnect
PRO
0
200
Navigating the moral maze — ethical principles for Al-driven product design
Avatar for Skipper Chong Warson skipperchong
2
400
Abbi's Birthday
Avatar for Violet Bock coloredviolet
3
8.4k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
287
14k
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
580
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
4
340

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler