Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Tracing for Granularity
Search
JBD
June 02, 2018
Programming
1.9k
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Tracing for Granularity
JBD
June 02, 2018
Transcript
36 slides
tracing for granularity JBD, Google (@rakyll)
@rakyll
@rakyll tracing? What is tracing and why do we trace?
@rakyll
@rakyll clogged?
@rakyll leaking?
@rakyll path and direction?
@rakyll 100% availability (is a lie)
“ @rakyll A service is available if users cannot tell
there was an outage.
@rakyll Without an SLO, your team has no principled way
of saying what level of downtime is acceptable. • Error rate • Latency or throughput expectations Service Level Objectives (SLOs)
@rakyll 28 ms 100 ms 172 ms 56 ms 356
ms what user sees what else we can see sec.Check auth.AccessToken cache.Lookup spanner.Query GET /messages
@rakyll 182 ms 56 ms 245 ms what user sees
what else we can see sec.Check auth.AccessToken GET /messages 7 ms cache.Lookup
@rakyll latency...
@rakyll Go is the language to write servers. Many runtime
activities occur during the program execution: • scheduling • memory allocation • garbage collection Hard to associate a request with its impact on the runtime.
@rakyll clogged?
“ @rakyll There is no easy way to tell why
latency is high for certain requests. Is it due to GC, scheduler or syscalls? Can you review the code and tell us why? -SRE
@rakyll Execution tracer $ go tool trace • Reports fine-grained
runtime events in the lifetime of a goroutine. • Reports utilization of CPU cores. But cannot easily tell how handling a request impacts the runtime.
@rakyll 28 ms 100 ms 172 ms 56 ms 356
ms GET /messages auth.AccessToken cache.Lookup spanner.Query GET /messages
@rakyll 5 68µs 8 123µs networking serialization + deserialization garbage
collection blocking syscall what actually happens 172 ms auth.AccessToken
@rakyll 5 68µs 8 123µs epoll executing sys gc netwrite
@rakyll How? • Mark sections in code using runtime/trace. •
Enable execution tracer temporarily and record data. • Examine the recorded data.
@rakyll Go 1.11 introduces... • User regions, tasks and annotations.
• Association between user code and runtime. • Association with distributed traces.
@rakyll Go 1.11 runtime/trace import “runtime/trace” ctx, task := trace.NewTask(ctx,
“myHandler”) defer task.End() // Handler code here....
@rakyll region #1 task #1 Go 1.11 runtime/trace region #2
region #3 region #4 region #5 goroutine #1 goroutine #4 goroutine #5
@rakyll import _ "net/http/pprof" go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()
@rakyll $ curl http://server:6060/debug/pprof/trace?seconds=5 -o trace.out $ go tool trace
trace.out 2018/05/04 10:39:59 Parsing trace... 2018/05/04 10:39:59 Splitting trace... 2018/05/04 10:39:59 Opening browser. Trace viewer is listening on http://127.0.0.1:51803
Execution tracer tasks for RPCs (/usertasks)
Execution tracer tasks for RPCs (/usertasks)
RPCs overlapping with garbage collection
Execution tracer regions (/userregions)
Region summary for conn.ready
@rakyll Record in production $ curl http://server/debug/pprof/trace?seconds=5 -o trace.out $
go tool trace trace.out
@rakyll Try It! Install the Go 1.11 beta1! golang.org/dl
@rakyll $ go get go.opencensus.io/trace import rt “runtime/trace” ctx, span
:= trace.StartSpan(ctx, “/messages”) defer span.End() rt.WithRegion(ctx, “foo”, func(ctx) { // Do something... })
@rakyll Limitations • Execution tracer cannot do accounting for cross-goroutine
operations automatically. • Exposition format is hard to parse if `go trace tool` is not used.
thank you! JBD, Google
[email protected]
@rakyll
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2.3k
eBPF in Microservices Observability
rakyll
1
1.8k
OpenTelemetry at AWS
rakyll
1
1.9k
Debugging Code Generation in Go
rakyll
5
1.7k
Are you ready for production?
rakyll
8
3.1k
Servers are doomed to fail
rakyll
3
1.6k
Serverless Containers
rakyll
1
310
Critical Path Analysis
rakyll
0
730
Monitoring and Debugging Containers
rakyll
2
1.2k
Other Decks in Programming
See All in Programming
GraphQL の onError と Nullability のこれから
quramy
0
220
flutter_test のその先へ — Flutter アプリの LLM 機能に「評価」という品質保証を
kosukesaigusa
0
210
ビジネスロジックを堅牢に~Scala 3のドメインプリミティブ~
j5ik2o
0
210
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
3
210
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
2
390
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか?〜 / Introduction to IaC in the AI Era
akihisaikeda
6
1.6k
AI駆動開発4年目の現在地(『あなたはFDEになるべきか?』イベント資料)
kumaiu
0
680
読みやすい!と評判(?)の発表の準備にかける私のこだわり
shoheimitani
0
110
社内AIエージェント「Askワンバン」における コストチューニングの取り組み
t0yohei
0
180
はじめてのPLATEAU
nokonoko1203
1
660
Ruby から Emacs を操作したい
phigasui
0
370
Crash Auto Pipelines
veronikapj
1
440
Featured
See All Featured
From Legacy to Launchpad: Building Startup-Ready Communities
dugsong
0
380
Un-Boring Meetings
codingconduct
0
500
Building AI with AI
inesmontani
PRO
1
1.4k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
130k
The Anti-SEO Checklist Checklist. Pubcon Cyber Week
ryanjones
0
310
Efficient Content Optimization with Google Search Console & Apps Script
katarinadahlin
PRO
1
1k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3.6k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
12
1.4k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
23k
Unlocking the hidden potential of vector embeddings in international SEO
frankvandijk
0
1.1k
Organizational Design Perspectives: An Ontology of Organizational Design Elements
kimpetersen
PRO
1
880
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
helenjbeal
2
400