Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Servers are doomed to fail
Search
JBD
May 17, 2019
Technology
1.6k
3
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Servers are doomed to fail
JBD
May 17, 2019
Transcript
30 slides
Servers are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Serverless is also doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Systems are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Is failure OK? Is failure an unexpected case?
Failure is not an exception. Systems change all the time.
“I haven’t touched the code for a century, it should
just work.” Said no one ever.
Failure is expected. Yes, it is.
None
@rakyll monitoring debugging postmortem
Monitoring is about saying if something is broken.
“99.99% of the requests should return in 100ms.”
@rakyll
@rakyll
Debugging
Debugging is collaborative.
Debugging comes in flavors. Logs Traces Metrics ...
Postmortems
Postmortems
Postmortems
Blameless? Focus on identifying problems.
Collaboration Design for collaboration.
Design for failure Set SLOs, plan for instrumentation, plan for
debugging.
Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google
Correlation Jump from monitoring/debugging data to data.
On-call debugging Jump from distributed tracing data to on-call information.
who to page?
Dynamic collection Capability to enable more collection in production when
needed.
Continuous collection Continuously collect signals, generate fleet-wide analysis reports.
Introspection Introspection pages provided from the services.
@rakyll monitoring debugging postmortem
Thank you Jaana B. Dogan Google
[email protected]
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2.3k
eBPF in Microservices Observability
rakyll
1
1.8k
OpenTelemetry at AWS
rakyll
1
1.9k
Debugging Code Generation in Go
rakyll
5
1.7k
Are you ready for production?
rakyll
8
3.1k
Serverless Containers
rakyll
1
310
Critical Path Analysis
rakyll
0
730
Monitoring and Debugging Containers
rakyll
2
1.2k
CPDD
rakyll
0
4.3k
Other Decks in Technology
See All in Technology
AI×アジャイルで、石橋を高速で叩きながら渡る~電力会社がゼロから築いた会員サイト改善プロセス~ / 20261112 Ayami Kanazawa & Shigeru Hirai
shift_evolve
PRO
1
110
そのイメージ、どのCIが作った? Artifact Attestationsで来歴を固定する / Which CI system created that image? Pinning provenance with Artifact Attestations
chmikata
0
180
Datadog の権限統制 〜Org 統合に向けて調べた注意点 / JDDUG-Meetup_24
visional_engineering_and_design
1
200
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
620
From vibe coding to reliable (agentic) development
jcabot
0
140
kubellの最新データ分析環境
kubell_hr
2
210
cdkdがどんだけ早いか、実際に動かして見てみよう
smt7174
1
140
変更差分を読むために環境を整備しよう
pipepasta
1
490
共通基盤の実装に Kotlin - Quarkus を選定した理由と採用したアーキテクチャ
nealle
1
150
信頼性を犠牲にせず、PR マージからデプロイまで人の介入を完全にゼロにするまでの道のり
knwoop
3
1.9k
AWS Client VPNの設定でハマったポイントとその過程での気づき
amarelo_n24
1
420
[関西Kaggler会 20261113] 続・コミュニティコンペのススメ
nejumi
0
180
Featured
See All Featured
Information Architects: The Missing Link in Design Systems
soysaucechin
1
1.3k
sira's awesome portfolio website redesign presentation
elsirapls
0
520
Navigating Team Friction
lara
192
16k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
chikuwait
0
960
Producing Creativity
orderedlist
PRO
348
41k
How GitHub (no longer) Works
holman
316
150k
Agile that works and the tools we love
rasmusluckow
331
22k
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
szymonslowik
1
1.4k
svc-hook: hooking system calls on ARM64 by binary rewriting
retrage
2
680
職位にかかわらず全員がリーダーシップを発揮するチーム作り / Building a team where everyone can demonstrate leadership regardless of position
madoxten
70
69k
Intergalactic Javascript Robots from Outer Space
tanoku
273
28k
WENDY [Excerpt]
tessaabrams
15
41k