Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critica...
Search
Caitie McCaffrey
June 14, 2016
Technology
8
440
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
290
Argus Papers We Love
caitiem20
13
1.2k
The Verification of a Distributed System
caitiem20
22
2.2k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
790
The Verification of a Distributed System
caitiem20
12
750
The Verification of a Distributed System
caitiem20
6
750
A Brief History of Distributed Programming: RPC
caitiem20
31
6.5k
Building Scalable Stateful Services
caitiem20
12
1.5k
Other Decks in Technology
See All in Technology
[JSAC 2025 LT] Introduction to MITRE ATT&CK utilization tools by multiple LLM agents and RAG
4su_para
1
110
なぜfreeeはハブ・アンド・スポーク型の データメッシュアーキテクチャにチャレンジするのか?
shinichiro_joya
2
620
三菱電機で社内コミュニティを立ち上げた話
kurebayashi
1
360
あなたの人生も変わるかも?AWS認定2つで始まったウソみたいな話
iwamot
3
870
メンバーがオーナーシップを発揮しやすいチームづくり
ham0215
2
260
AWS re:Invent 2024 recap in 20min / JAWSUG 千葉 2025.1.14
shimy
1
110
Amazon Route 53, 待ちに待った TLSAレコードのサポート開始
kenichinakamura
0
180
商品レコメンドでのexplicit negative feedbackの活用
alpicola
2
410
AIアプリケーション開発でAzure AI Searchを使いこなすためには
isidaitc
1
130
タイミーのデータ活用を支えるdbt Cloud導入とこれから
ttccddtoki
1
310
GoogleのAIエージェント論 Authors: Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic
customercloud
PRO
0
190
dbtを中心にして組織のアジリティとガバナンスのトレードオンを考えてみた
gappy50
0
330
Featured
See All Featured
The Pragmatic Product Professional
lauravandoore
32
6.4k
KATA
mclloyd
29
14k
Statistics for Hackers
jakevdp
797
220k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.6k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
We Have a Design System, Now What?
morganepeng
51
7.3k
4 Signs Your Business is Dying
shpigford
182
22k
A designer walks into a library…
pauljervisheath
205
24k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
960
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
113
50k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie