Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Servers are doomed to fail
Search
JBD
May 17, 2019
Technology
3
1.5k
Servers are doomed to fail
JBD
May 17, 2019
Tweet
Share
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2k
eBPF in Microservices Observability
rakyll
1
1.6k
OpenTelemetry at AWS
rakyll
1
1.8k
Debugging Code Generation in Go
rakyll
5
1.5k
Are you ready for production?
rakyll
8
2.6k
Serverless Containers
rakyll
1
240
Critical Path Analysis
rakyll
0
500
Monitoring and Debugging Containers
rakyll
2
1.1k
CPDD
rakyll
0
4.1k
Other Decks in Technology
See All in Technology
Mocking in Rust Applications
taiki45
2
410
タイミーのレコメンドにおける ABテストの運用
ozeshun
1
140
DroidKaigi 2024 たすけて!ViewModel
mhidaka
5
940
Road to Single Activity
yurihondo
2
240
どこよりも遅めなWinActor Ver.7.5.0 新機能紹介
tamai_63
0
210
PDF Viewer作成の今までとこれから
hunachi
0
470
持続可能なソフトウェア開発を支える『GitHub CI/CD実践ガイド』
tmknom
7
1.4k
AIで変わるテスト自動化:最新ツールの多様なアプローチ/ 20240910 Takahiro Kaneyama
shift_evolve
0
240
『GRANBLUE FANTASY: Relink』最高の「没入感」を実現するカットシーン制作手法とそれを支える技術
cygames
1
140
Google CloudのLLM活用の選択肢を広げるVertex AIのパートナーモデル
nayuts
0
130
サーバレスでモバイルアプリ開発! NTTコム「ビジネスdアプリ」のアーキテクチャ / The architecture of business d app
nttcom
12
240
テスト”ケース”駆動開発 で手戻りをなくそう
ryohma0510
0
320
Featured
See All Featured
Imperfection Machines: The Place of Print at Facebook
scottboms
263
13k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
Ruby is Unlike a Banana
tanoku
96
11k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
103
48k
4 Signs Your Business is Dying
shpigford
179
21k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Building Flexible Design Systems
yeseniaperezcruz
325
38k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Six Lessons from altMBA
skipperchong
26
3.4k
Happy Clients
brianwarren
96
6.6k
10 Git Anti Patterns You Should be Aware of
lemiorhan
653
58k
It's Worth the Effort
3n
182
27k
Transcript
Servers are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Serverless is also doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Systems are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Is failure OK? Is failure an unexpected case?
Failure is not an exception. Systems change all the time.
“I haven’t touched the code for a century, it should
just work.” Said no one ever.
Failure is expected. Yes, it is.
None
@rakyll monitoring debugging postmortem
Monitoring is about saying if something is broken.
“99.99% of the requests should return in 100ms.”
@rakyll
@rakyll
Debugging
Debugging is collaborative.
Debugging comes in flavors. Logs Traces Metrics ...
Postmortems
Postmortems
Postmortems
Blameless? Focus on identifying problems.
Collaboration Design for collaboration.
Design for failure Set SLOs, plan for instrumentation, plan for
debugging.
Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google
Correlation Jump from monitoring/debugging data to data.
On-call debugging Jump from distributed tracing data to on-call information.
who to page?
Dynamic collection Capability to enable more collection in production when
needed.
Continuous collection Continuously collect signals, generate fleet-wide analysis reports.
Introspection Introspection pages provided from the services.
@rakyll monitoring debugging postmortem
Thank you Jaana B. Dogan Google
[email protected]