Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critical Failures
Search
Caitie McCaffrey
June 14, 2016
Technology
8
410
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
325
21k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
230
Argus Papers We Love
caitiem20
13
1.1k
The Verification of a Distributed System
caitiem20
22
2.1k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
730
The Verification of a Distributed System
caitiem20
12
690
The Verification of a Distributed System
caitiem20
6
710
A Brief History of Distributed Programming: RPC
caitiem20
31
6.4k
Building Scalable Stateful Services
caitiem20
12
1.4k
Other Decks in Technology
See All in Technology
LLMアプリケーションの評価の実践と課題 ~PharmaXにおける今後の展望~
pharma_x_tech
2
160
Luupの開発組織におけるインシデントマネジメントの変遷 ver.RoadtoSRENEXT2024
grimoh
1
270
AWSでRAGを作る法方
sonoda_mj
1
140
ペパボのオブザーバビリティ研修2024 説明資料
kesompochy
0
1.1k
ギークの理想が7つ集まるエムスリーで夢を叶えよう - エムスリー株式会社
m3_engineering
1
260
楽しくGoを学び合う、LayerXの勉強会文化 / LayerX's study culture of having fun and learning Go together
ar_tama
2
350
Azure OpenAI Service Dev Day / LLMでできる!使える!生成AIエージェント
masahiro_nishimi
3
750
サーバーレスAPI(API Gateway+Lambda)とNext.jsで 個人ブログを作ろう!
shuntaka
PRO
0
560
大規模ドラレコデータ収集・機械学習基盤を支える AWS CDK 〜導入・運用事例紹介〜
pemugi
0
110
データ分析基盤を作ってみよう~設計編~
nrinetcom
PRO
1
110
OSSコミットしてZennの課題を解決した話
dyoshikawa1993
0
150
サービス開発を前に進めるために 新米リードエンジニアが 取り組んだこと / Steps Taken by a Novice Lead Engineer to Advance Service Development
nologyance
0
180
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
52k
Ruby is Unlike a Banana
tanoku
96
10k
Web Components: a chance to create the future
zenorocha
307
41k
Embracing the Ebb and Flow
colly
81
4.3k
The Power of CSS Pseudo Elements
geoffreycrofte
64
5.2k
A Modern Web Designer's Workflow
chriscoyier
689
190k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
26
1.8k
Writing Fast Ruby
sferik
623
60k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
121
18k
Fashionably flexible responsive web design (full day workshop)
malarkey
399
65k
A Philosophy of Restraint
colly
200
16k
Mobile First: as difficult as doing things right
swwweet
219
8.8k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie