Four years of breaking things in production, on...

November 09, 2017

55

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

Tweet

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

360

A Brief Introduction To DevOps

0

100

Humans are terrible compilers: A User's Guide

0

110

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

160

"Is there any strong objection?"

0

210

Fear, Uncertainty, and Continuous Deployment

1

120

0

220

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

160

Engineering for Engineers

0

87

Other Decks in Technology

See All in Technology

ビジネスとデザインとエンジニアリングを繋ぐために一人のエンジニアは何ができるか / What can a single engineer do to connect business, design, and engineering?

2

770

OpsJAWS34_CloudTrailLake_for_Organizations

0

190

Conquering PDFs: document understanding beyond plain text

1

360

Porting PicoRuby to Another Microcontroller: ESP32

4

500

日経電子版 for Android の技術的課題と取り組み(令和最新版)/android-20250423

nikkei_engineer_recruiting

1

560

ドキュメント管理の理想と現実

1

270

OpenLane-V2ベンチマークと代表的な手法

0

120

白金鉱業Meetup_Vol.18_AIエージェント時代のUI/UX設計

1

250

生成AIによるCloud Native基盤構築の可能性と実践的ガードレールの敷設について

7

1.3k

LiteXとオレオレCPUで作る自作SoC奮闘記

0

880

【Λ（らむだ）】最近のアプデ情報 / RPALT20250422

0

130

AIにおけるソフトウェアテスト_ver1.00

1

290

Featured

See All Featured

Art, The Web, and Tiny UX

298

20k

Mobile First: as difficult as doing things right

223

9.6k

Adopting Sorbet at Scale

76

9.3k

30

1.1k

ReactJS: Keep Simple. Everything can be a component!

667

120k

Music & Morning Musume

47

6.5k

Improving Core Web Vitals using Speculation Rules API

sergeychernyshev

13

810

Creating an realtime collaboration tool: Agile Flush - .NET Oxford

30

2k

How GitHub (no longer) Works

314

140k

453

42k

Distributed Sagas: A Protocol for Coordinating Microservices

331

21k

How To Stay Up To Date on Web Technology

790

250k

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler