Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critica...
Search
Caitie McCaffrey
June 14, 2016
Technology
8
470
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
340
Argus Papers We Love
caitiem20
13
1.2k
The Verification of a Distributed System
caitiem20
22
2.3k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
820
The Verification of a Distributed System
caitiem20
12
780
The Verification of a Distributed System
caitiem20
6
780
A Brief History of Distributed Programming: RPC
caitiem20
31
6.7k
Building Scalable Stateful Services
caitiem20
12
1.6k
Other Decks in Technology
See All in Technology
NewSQLや分散データベースを支えるRaftの仕組み - 仕組みを理解して知る得意不得意
hacomono
PRO
2
130
無意味な開発生産性の議論から抜け出すための予兆検知とお金とAI
i35_267
4
13k
SaaS型なのに自由度の高い本格CMSでサイト構築と運用のコスパ&タイパUP! MovableType.net の便利機能とユーザー事例のご紹介
masakah
0
110
スタートアップに選択肢を 〜生成AIを活用したセカンダリー事業への挑戦〜
nstock
0
170
OPENLOGI Company Profile for engineer
hr01
1
34k
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
7
5.2k
Tokyo_reInforce_2025_recap_iam_access_analyzer
hiashisan
0
180
赤煉瓦倉庫勉強会「Databricksを選んだ理由と、絶賛真っ只中のデータ基盤移行体験記」
ivry_presentationmaterials
2
360
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
300
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
180
Glacierだからってコストあきらめてない? / JAWS Meet Glacier Cost
taishin
1
160
Flutter向けPDFビューア、pdfrxのpdfium WASM対応について
espresso3389
0
130
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
351
20k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Typedesign – Prime Four
hannesfritz
42
2.7k
How GitHub (no longer) Works
holman
314
140k
Faster Mobile Websites
deanohume
307
31k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Rails Girls Zürich Keynote
gr2m
95
14k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
Music & Morning Musume
bryan
46
6.6k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie