Four years of breaking things in production, on...

November 09, 2017

80

Four years of breaking things in production, on purpose.

Presented at Chaos Day Twin Cities, November 2017.

Eric Sigler

November 09, 2017

More Decks by Eric Sigler

See All by Eric Sigler

Instrumenting The Rest Of The Company: Hunting For Metrics

0

410

A Brief Introduction To DevOps

0

140

Humans are terrible compilers: A User's Guide

0

160

Do You Know If Your Service Is Working Properly? A Guide To Being Paranoid.

0

230

"Is there any strong objection?"

0

290

Fear, Uncertainty, and Continuous Deployment

1

160

0

290

Strategies For Being On Call & Keeping Your Sanity At The Same Time

0

210

Engineering for Engineers

0

120

Other Decks in Technology

See All in Technology

Webとヘルスデータ

1

200

AIで仕事のやり方を変える

2

980

Sigmaユーザーのための有用リソース一挙公開 & Sigmaで使えるMCP #sigma_ucj /useful-resources-for-sigma-computing-users-and-mcps-with-sigma

0

150

プロダクトエンジニアに必要な「いい感じ」に作る能力〜たくさん作れる時代に、どこまで作るかの決め方〜

jnishime_dresscode

2

1.4k

Azure Cost Management の FOCUS コストデータを迷わず読むための“3つの軸”

0

150

データエンジニアリングワークショップ：Auto LoaderとSparkで学ぶデータパイプライン構築

databricksjapan

PRO

0

210

Oracle Cloud Infrastructure IaaS 新機能アップデート 2026/6 - 2026/8

oracle4engineer

PRO

0

120

Beyond the Hype: Practical AI for Your Oracle Database with MCP

1

280

Multica × 長期記憶：40個のミニプロジェクト管理

1

200

AI活用の現在地、ちゃんと見えてますか？/XPfest-2026

visional_engineering_and_design

0

160

動画配信アプリでの Engage SDK 導入 — TVer Android が Play ストアにコンテンツを届けるまで

PRO

0

140

スクラムで身についていた動き方を、XPで捉え直してみた

PRO

1

300

Featured

See All Featured

Making the Leap to Tech Lead

135

10k

Understanding Cognitive Biases in Performance Measurement

32

3k

Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)

PRO

0

450

Imperfection Machines: The Place of Print at Facebook

270

14k

Stop Working from a Prison Cell

274

21k

The Impact of AI in SEO - AI Overviews June 2024 Edition

6

1.2k

Exploring anti-patterns in Rails

3

490

Designing for humans not robots

254

26k

jQuery: Nuts, Bolts and Bling

66

8.6k

How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.

0

190

JavaScript: Past, Present, and Future - NDC Porto 2020

52

6.1k

A brief & incomplete history of  UX Design for the World Wide Web: 1989–2019

2

500

Transcript

Eric Sigler, Head of DevOps, PagerDuty @esigler Four years of
breaking things in production, on purpose.
@esigler Obligatory disclaimer: This is what works for us. Take
away ideas, not dogmas.
@esigler
@esigler 2013: Every Friday, 1 hour. 2013 2014 2015 2016
2017
@esigler 2013 2014 2015 2016 2017
None
@esigler 2014: Expanding Scope 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2015: Automation 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2016: Adding In Randomness 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Also 2016: Putting It All Together 2013 2014 2015
2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler 2017: Distributing Knowledge 2013 2014 2015 2016 2017
@esigler 2013 2014 2015 2016 2017
@esigler Failure Friday sessions: 133 Faults injected: 708 Fault injections
resulting in a public postmortem: 3
@esigler Simulated full AZ failures: 4 Simulated full Region failures:
3 Simulated partial Disaster Recovery: 2
@esigler Tickets created from Failure Friday: over 225 Distinct services
that had faults injected: 49
@esigler
@esigler Optimized for learning first, tooling second Built the toolchain
to enable other teams Distributed chaos engineering knowledge
@esigler