Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
The Walking Dead - A Survival Guide to Resilien...
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Michael Nitschinger
April 23, 2015
Programming
390
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
The Walking Dead - A Survival Guide to Resilient Reactive Applications
This talk was given at JAX 2015 in Mainz.
Michael Nitschinger
April 23, 2015
Transcript
54 slides
Michael Nitschinger | Couchbase, Inc. The Walking Dead A Survival
Guide to Reactive Resilient Applications
the right Mindset 2
– U.S. Marine Corps “The more you sweat in peace,
the less you bleed in war.” 3
4
5
Not so fast, mister fancy tests! 6
What can go wrong? Always ask yourself 7
Fault Tolerance 101 8
Fault Error Failure A fault is a latent defect that
can cause an error when activated. 9
Fault Error Failure Errors are the manifestations of faults. 10
Fault Error Failure Failure occurs when the service no longer
complies with its specifications. 11
Fault Error Failure Errors are inevitable. We need to detect,
recover and mitigate them before they become failures. 12
Reliability is the probability that a system will perform failure
free for a given amount of time. MTTF Mean Time To Failure MTTR Mean Time To Repair 13
Availability is the percentage of time the system is able
to perform its function. availability = MTTF MTTF + MTTR 14
Expression Downtime/Year Three 9s 99.9% 525.6 min Four 9s 99.99%
52.56 min Four 9s and a 5 99.995% 26.28 min Five 9s 99.999% 5.256 min Six 9s 99.9999% 0.5256 min 100% 0 15
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability ??? ??? ??? 16
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability 99.99% 17 99.99% 99.99%
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability ~99.999% ~99.999% ~99.999% 18
Fault Tolerant Architecture 19
Units of Mitigation are the basic units of error containment
and recovery. 20
Escalation is used when recovery or mitigation is not possible
inside the unit. 21
Escalation 22 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 23 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 24 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 25 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Redundancy Cost Active/Active Active/Standby N+M Active/Passive Cost Time To Recover
26
The Fault Observer receives system and error events and can
guide and orchestrate detection and recovery Unit Unit Observer Listener Listener Unit Unit 27
28
29
Detecting Errors 30
A silent system is a dead system. 31
A System Monitor helps to study behaviour and to make
sure it is operating as specified. 32 http://cdn-www.airliners.net/aviation-photos/photos/9/2/1/0982129.jpg
https://github.com/Netflix/Turbine 33
Periodic Checking Heartbeats monitor tasks or remote services and initiate
recovery Routine Exercises prevent idle unit starvation and surface malfunctions 34
35 Encoder( Encoder( Ne*y( Writes( Ne*y( Reads( Decoder( Decoder( Event
on Idle No Traffic Endpoint
Riding over Transients is used to defer error recovery if
the error is temporary. “‘Patience is a virtue’ to allow the true signature of an error to show itself.” - Robert S. Hanmer 36
37 The Leaky Bucket
And more! • Complete Parameter Checking • Watchdogs • Voting
• Checksums • Routine Audits 38
Recovery and Mitigation of Errors 39
Timeout to not wait forever and keep holding up the
resource. 40 X
Failover to a redundant unit when the error has been
detected and isolated. Cost Active/Active Active/Standby N+M Cost Time To Recover Redundancy Reminder 41
Intelligent Retries Time between Retries Number of Attempts Fixed Linear
Exponential 42
Restart can be used as a last resort with the
trade-off to lose state and time. 43
Fail Fast to shed load and give a partial great
service than a complete bad one. Boundary 44
Backpressure & Batching! 45
Case Study: Hystrix https://raw.githubusercontent.com/wiki/Netflix/Hystrix/images/hystrix-flow-chart-original.png 46
And more! • Rollback • Roll-Forward • Checkpoints • Data
Reset Recovery Mitigation • Bounded Queuing • Expansive Controls • Marking Data • Error Correcting Codes 47
And more! • Rollback • Roll-Forward • Checkpoints • Data
Reset Recovery Mitigation • Bounded Queuing • Expansive Controls • Marking Data • Error Correcting Codes 48
Recommended Reading 49
Patterns for Fault-Tolerant Software by Robert S. Hanmer 50
Release It! by Michael T. Nygard 51
Announcement CB Server 4.0 dp! 52 http://blog.couchbase.com/introducing-developer-preview-for-couchbase-server-4.0
Any Questions? 53
twitter @daschl email
[email protected]
Thank you! 54
More Decks by Michael Nitschinger
See All by Michael Nitschinger
High Performance JVM Networking with Netty
daschl
5
1.3k
Reactive Data Access with RxJava... and N1QL!
daschl
0
230
Spark with Couchbase
daschl
0
180
Reactive Data Access with RxJava ... and N1QL!
daschl
0
200
The Walking Dead - A Survival Guide to Resilient Reactive Applications
daschl
0
250
State of the Art JVM Networking with Netty
daschl
2
470
The Walking Dead - A Survival Guide to Resilient Reactive Applications
daschl
1
480
The Walking Dead - A Survival Guide to Resilient Applications
daschl
0
1.4k
Building a Reactive Database Driver on the JVM
daschl
2
990
Other Decks in Programming
See All in Programming
AI時代のFlutter品質設計 — アプリ品質を守るQuality Layers
omatty198
0
380
AIがコードを書いてくれる時代、我々は何を確かめるのか
yosuke_furukawa
PRO
13
6k
AWS CDKを理解するために公式の定義を深掘って解説してみる / Understanding AWS CDK by Digging into Its Official Definition
akihisaikeda
3
310
Lambda durable functionsでHuman in the Loopを楽に…楽に……
har1101
2
330
Augmenting AI with the Power of Jakarta EE
ivargrimstad
0
360
Getting Started with Vibe Coding
miccheng
0
240
高橋メソッドでCVE解説シリーズ 〜CVE-2026-112816〜
hakkadaikon
0
270
Scalaの型でマルチテナントの認可を守る
nextbeatdev
0
150
小さなアプリを作って公開するまで
onozaty
0
190
[DroidKnights 2026] Workflow AX: From Auto-Documented Policies to E2E Testing
raeminkang
0
580
社内AIエージェント「Askワンバン」における コストチューニングの取り組み
t0yohei
0
170
baseline으로 마이그레이션, 리팩토링 안전하게 맡기기
fornewid
1
1k
Featured
See All Featured
AI in Enterprises - Java and Open Source to the Rescue
ivargrimstad
0
1.6k
Mozcon NYC 2025: Stop Losing SEO Traffic
samtorres
1
600
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
aleyda
1
2.3k
Prompt Engineering for Job Search
mfonobong
0
590
Producing Creativity
orderedlist
PRO
348
41k
Building a Modern Day E-commerce SEO Strategy
aleyda
45
9.3k
RailsConf 2023
tenderlove
30
1.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
371
20k
The Impact of AI in SEO - AI Overviews June 2024 Edition
aleyda
6
1.3k
Neural Spatial Audio Processing for Sound Field Analysis and Control
skoyamalab
0
620
Making the Leap to Tech Lead
cromwellryan
135
10k
Optimizing for Happiness
mojombo
378
71k