Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Four years of breaking things in production, on...

Avatar for Eric Sigler Eric Sigler
November 09, 2017
Technology
88
0

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Avatar for Eric Sigler

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler
Instrumenting The Rest Of The Company: Hunting For Metrics
Avatar for Eric Sigler esigler
0
420
A Brief Introduction To DevOps
Avatar for Eric Sigler esigler
0
140
Humans are terrible compilers: A User's Guide
Avatar for Eric Sigler esigler
0
160
Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.
Avatar for Eric Sigler esigler
0
240
"Is there any strong objection?"
Avatar for Eric Sigler esigler
0
290
Fear, Uncertainty, and Continuous Deployment
Avatar for Eric Sigler esigler
1
160
3AM, a survey.
Avatar for Eric Sigler esigler
0
300
Strategies For Being On Call & Keeping Your Sanity At The Same Time
Avatar for Eric Sigler esigler
0
210
Engineering for Engineers
Avatar for Eric Sigler esigler
0
130

Other Decks in Technology

See All in Technology
CloudWatch Logsのコスト最適化：どこまでAIに任せてどこから人間が承認すべきか
Avatar for Yuuki Yamashita yama3133
0
110
宣言型部分更新は何を解決しどこへ向かうのか
Avatar for did0es shuta13
1
490
Railsで挑む大量データ永続化 〜600億件を永続化するための設計とボトルネックとの戦い〜
Avatar for Shoichi Ochi occhi
1
880
React改造してみた
Avatar for Mooncake Sugar tsukina_7mochi
0
140
スキマバイトという雇用形態は存在しないので、雇用期間1日のアルバイト契約を事業者とワーカーが結ぶ手続きを支えるのがタイミーなんです
Avatar for しが あきとし(akitoshiga) akitoshiga
0
210
Benchmarking Disaggregated Serving on vLLM + Dynamo with a Real Coding-Agent Workload
Avatar for NTT docomo Business nttcom
1
200
1つのリクエストを追って理解する - llm-dのP/D分離
Avatar for akhino akhino
1
170
開発を高速化。AIコーディングエージェントとCodeRabbitでループエンジニアリング
Avatar for Atsushi Nakatsugawa moongift
PRO
0
100
20261017福岡devIO 「AIは"良いダッシュボード"を作れるか？
Avatar for 武田 jtakeda
0
190
S3 Filesで挑む！ 八戸せんべい汁の食べ頃問題
Avatar for usanchuu usanchuu
0
120
AIがコードを書く時代に、エンジニアは何で自分の価値を証明するのか / In an era where AI writes code how do engineers prove their value
Avatar for CARTA Engineering carta_engineering
3
1.1k
ブラウザとサーバーから考えるAI時代のフロントエンド監視
Avatar for Shogo Fukami shogo4131
3
1k

Featured

See All Featured
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
301
52k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8.4k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
Avatar for Rebecca Miller-Webster rmw
35
3.9k
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
460
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
2
630
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
409
67k
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
2
1.8k
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
500
The Limits of Empathy - UXLibs8
Avatar for Cassini Nazir cassininazir
2
720
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.9k
Scaling GitHub
Avatar for Zach Holman holman
464
140k

Transcript

  1. Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of

    breaking things in production, on purpose.

  2. @esigler Obligatory disclaimer: This is what works for us. Take

    away ideas, not dogmas.

  3. @esigler

  4. @esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016

    2017

  5. @esigler 2013 2014 2015 2016 2017

  6. None

  7. @esigler 2014: Expanding Scope 2013 2014 2015 2016 2017

  8. @esigler 2013 2014 2015 2016 2017

  9. @esigler 2015: Automation 2013 2014 2015 2016 2017

  10. @esigler 2013 2014 2015 2016 2017

  11. @esigler 2013 2014 2015 2016 2017

  12. @esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017

  13. @esigler 2013 2014 2015 2016 2017

  14. @esigler Also 2016: Putting It All Together 2013 2014 2015

    2016 2017

  15. @esigler 2013 2014 2015 2016 2017

  16. @esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017

  17. @esigler 2013 2014 2015 2016 2017

  18. @esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections

    resulting in a public postmortem: 3

  19. @esigler Simulated full AZ failures: 4 Simulated full Region failures:

    3 Simulated partial Disaster Recovery: 2

  20. @esigler Tickets created from Failure Friday: over 225 Distinct services

    that had faults injected: 49

  21. @esigler

  22. @esigler Optimized for learning first, tooling second Built the toolchain

    to enable other teams Distributed chaos engineering knowledge

  23. @esigler