Four years of breaking things in production, on...

November 09, 2017

52

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

Tweet

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

330

A Brief Introduction To DevOps

0

99

Humans are terrible compilers: A User's Guide

0

110

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

160

"Is there any strong objection?"

0

210

Fear, Uncertainty, and Continuous Deployment

1

110

0

210

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

160

Engineering for Engineers

0

84

Other Decks in Technology

See All in Technology

【LT】ソフトウェア産業は進化しているのか？ #Agilejapan

0

100

日経電子版のStoreKit2フルリニューアル

1

150

オープンソースAIとは何か？ --「オープンソースAIの定義 v1.0」詳細解説

10

1.3k

DynamoDB でスロットリングが発生したとき/when_throttling_occurs_in_dynamodb_short

0

270

TypeScriptの次なる大進化なるか！？　条件型を返り値とする関数の型推論

2

1.7k

『Firebase Dynamic Links終了に備える』 FlutterアプリでのAdjust導入とDeeplink最適化

0

170

RubyのWebアプリケーションを50倍速くする方法 / How to Make a Ruby Web Application 50 Times Faster

3

950

SREが投資するAIOps ~ペアーズにおけるLLM for Developerへの取り組み~

2

510

FlutterアプリにおけるSLI/SLOを用いたユーザー体験の可視化と計測基盤構築

0

120

プロダクト活用度で見えた真実ホリゾンタルSaaSでの顧客解像度の高め方

0

210

SRE×AIOpsを始めよう！GuardDutyによるお手軽脅威検出

0

200

アジャイルチームがらしさを発揮するための目標づくり / Making the goal and enabling the team

3

160

Featured

See All Featured

Imperfection Machines: The Place of Print at Facebook

265

13k

XXLCSS - How to scale CSS and keep your sanity

246

1.3M

sergeychernyshev

25

620

Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything

26

2.1k

Easily Structure & Communicate Ideas using Wireframe

191

16k

[RailsConf 2023] Rails as a piece of cake

52

4.9k

34

3k

"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)

226

22k

Refactoring Trust on Your Teams (GOTO; Chicago 2020)

31

2.7k

Performance Is Good for Brains [We Love Speed 2024]

6

430

Build The Right Thing And Hit Your Dates

33

2.4k

Designing for humans not robots

250

25k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler