Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenCensusでcustom context propagationとexporterを...
Search
takashabe
June 27, 2019
Technology
0
1.6k
OpenCensusでcustom context propagationとexporterを書いた話 / OpenCensus with custom context propagation and exporter
OpenCensus/OpenTelemetry meetup vol.2 の発表資料です。
https://opencensus.connpass.com/event/132588/
takashabe
June 27, 2019
Tweet
Share
More Decks by takashabe
See All by takashabe
より良いターミナルでの生活を求めて
takashabe
0
44
pubsub with concurrent
takashabe
1
880
社内ISUCONを開催した話
takashabe
0
1.6k
ISUCON大反省会
takashabe
0
1.8k
gitのブランチ戦略
takashabe
8
5.8k
サルでもわかるgit
takashabe
0
1.4k
playで複数DBする
takashabe
0
1.6k
MySQLで高トラフィックに立ち向かう
takashabe
0
1.8k
GitHubの良さ
takashabe
2
2.2k
Other Decks in Technology
See All in Technology
QA/SDETの現在と、これからの挑戦
imtnd
0
120
Classmethod AI Talks(CATs) #21 司会進行スライド(2025.04.17) / classmethod-ai-talks-aka-cats_moderator-slides_vol21_2025-04-17
shinyaa31
0
580
The Tale of Leo: Brave Lion and Curious Little Bug
canalun
1
120
生成AIによるCloud Native基盤構築の可能性と実践的ガードレールの敷設について
nwiizo
7
690
Amazon CloudWatch を使って NW 監視を行うには
o11yfes2023
0
160
React ABC Questions
hirotomoyamada
0
150
サーバレス、コンテナ、データベース特化型機能をご紹介。CloudWatch をもっと使いこなそう!
o11yfes2023
0
170
クォータ監視、AWS Organizations環境でも楽勝です✌️
iwamot
PRO
1
310
彩の国で始めよう。おっさんエンジニアから共有したい、当たり前のことを当たり前にする技術
otsuki
0
150
Amazon CloudWatch Application Signals ではじめるバーンレートアラーム / Burn rate alarm with Amazon CloudWatch Application Signals
ymotongpoo
5
510
4/17/25 - CIJUG - Java Meets AI: Build LLM-Powered Apps with LangChain4j (part 2)
edeandrea
PRO
0
110
アセスメントで紐解く、10Xのデータマネジメントの軌跡
10xinc
1
430
Featured
See All Featured
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
178
53k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
It's Worth the Effort
3n
184
28k
Documentation Writing (for coders)
carmenintech
69
4.7k
Building Adaptive Systems
keathley
41
2.5k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
104
19k
Code Reviewing Like a Champion
maltzj
522
40k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Navigating Team Friction
lara
184
15k
Transcript
OpenCensus/OpenTelemetry meetup vol.2 @takashabe OpenCensusͰcustom context propagationͱexporterΛॻ͍ ͨ
whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • GoͰࠂ৴ϓϩμΫτΛ࡞͍ͬͯ·͢
• KubernetesͰMicroservicesతͳͷ • ਪ͠ΩʔϘʔυirisͰ͢ • Corne Cherry࡞த…
͡Ίʹ • Go + OpenCensus + Stackdriver Λલఏͱ͍ͯ͠·͢
࣍ • ࢄτϨʔγϯάͷ֓ཁ • context propagation • exporter • ·ͱΊ
ࢄτϨʔγϯά
ࢄτϨʔγϯά https://github.com/census-instrumentation/opencensus-go • 1 traceʹରͯ͠ɺෳͷspan͕ඥͮ͘ • spanಉҰΞϓϦέʔγϣϯͰෳൃߦͯ͠ྑ͍͠ɺҟͳΔΞϓϦέʔγϣϯͰ τϨʔεID͕ಉҰͰ͋Εඥͮ͘
context propagation
context propagation is …
context propagation is … • τϨʔεIDΛൖ͢ΔͨΊͷϓϩτίϧతͳͷ • opencensus-goͰocgrpcochttpͰ࣮͞Ε͍ͯΔ • SpanContextΛհͯ͠σʔλΛൖͤ͞Δ
• TraceID, SpanID, TraceOptions(αϯϓϦϯάର͔Ͳ ͏͔) type SpanContext struct { TraceID TraceID // [16]byte SpanID SpanID // [8]byte TraceOptions TraceOptions // uint32 Tracestate *tracestate.Tracestate // Ұ෦ͷpropagation༻ }
ocgrpc, ochttp ocgrpc ochttp
HTTPFormatͷ࣮ • ocgrpc࣮͕1ͭ • ochttpͰHTTPFormatΛ࣮ͨ͠b3ͱtracecontextͱ͍͏2 ͭͷ࣮͕ଘࡏ͢Δ • b3: ZipkinͰఆٛ͞Εͨpropagation(σϑΥϧτ) •
https://github.com/openzipkin/b3-propagation • tracecontext: w3cͰఆٛ͞Εͨpropagation • https://github.com/w3c/trace-context type HTTPFormat interface { SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) SpanContextToRequest(sc trace.SpanContext, req *http.Request) }
B3 Propagation https://github.com/openzipkin/b3-propagation
B3 Propagation https://github.com/openzipkin/b3-propagation 4QBO$POUFYU'SPN3FRVFTU 4QBO$POUFYU5P3FRVFTU
OpenCensusͰͷ࣮(ToRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextToRequest(sc trace.SpanContext, req *http.Request) { req.Header.Set(TraceIDHeader, hex.EncodeToString(sc.TraceID[:])) req.Header.Set(SpanIDHeader, hex.EncodeToString(sc.SpanID[:])) var sampled string if sc.IsSampled() { sampled = "1" } else { sampled = "0" } req.Header.Set(SampledHeader, sampled) } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
OpenCensusͰͷ࣮(FromRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) { tid, ok := ParseTraceID(req.Header.Get(TraceIDHeader)) if !ok { return trace.SpanContext{}, false } sid, ok := ParseSpanID(req.Header.Get(SpanIDHeader)) if !ok { return trace.SpanContext{}, false } sampled, _ := ParseSampled(req.Header.Get(SampledHeader)) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, }, true } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
ochttpΛར༻͢Δ // import "go.opencensus.io/plugin/ochttp" mux := http.NewServeMux() mux.Handle("/users", ochttp.WithRouteTag(usersHandler, "/users"))
log.Fatal(http.ListenAndServe("localhost:8080", &ochttp.Handler{ Handler: mux, Propagation: &b3.HTTPFormat{}, })) https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/example_test.go • ochttp.Handler͕ϛυϧΣΞͱͯ͠τϨʔγϯάपΓ Λ໘ݟͯ͘ΕΔ • context propagationҙࣝ͠ͳͯ͘ྑ͍
custom context propagation
Cloud Pub/Sub ʹద༻ͨ͠ • GCPͷϚωʔδυPub/Sub • B3 PropagationΛϕʔεʹͯ͠ɺϝοηʔδͷ AttributesʹσʔλΛ٧Ίͯൖͤ͞Δ •
https://github.com/takashabe/oc-propagation-demo • ΄΅ಉ͡ͷΛϓϩμΫγϣϯʹಋೖࡁΈ͕ͩɺڥ ґଘ͕͋ΔͷͰαϯϓϧ࣮Λ༻ҙ • αʔό༻ҙ͢Δͷ໘ͰCloud Run༻ʹHTTPαʔό ͱ࣮͍ͯͯ͠͠ΔͷͰएׯݟͮΒ͍
Pub/Sub൛ͷ࣮(ToRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func WrapMessage(ctx context.Context, m *pubsub.Message) *pubsub.Message { if m.Attributes != nil { if m.Attributes[TraceIDField] != "" || m.Attributes[SpanIDField] != "" { return m } } else { m.Attributes = make(map[string]string, 3) } sc := trace.FromContext(ctx).SpanContext() m.Attributes[TraceIDField] = sc.TraceID.String() m.Attributes[SpanIDField] = sc.SpanID.String() m.Attributes[SampledField] = fmt.Sprintf("%t", sc.IsSampled()) return m }
Pub/Sub൛ͷ࣮(FromRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func SpanContextFromMessage(m *pubsub.Message) trace.SpanContext { if m.Attributes == nil { return trace.SpanContext{} } tid, ok := parseTraceID(m.Attributes[TraceIDField]) if !ok { return trace.SpanContext{} } sid, ok := parseSpanID(m.Attributes[SpanIDField]) if !ok { return trace.SpanContext{} } sampled := parseSampled(m.Attributes[SampledField]) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, } }
ར༻ଆ // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... ctx, span := trace.StartSpan(context.Background(), "publish")
defer span.End() msg := &pubsub.Message{ Data: []byte("foo"), } topic.Publish(ctx, propagation.WrapMessage(ctx, msg)).Get(ctx) // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { sc := propagation.SpanContextFromMessage(msg) _, span := trace.StartSpanWithRemoteParent(ctx, "receive", sc) defer span.End() publisher(client) subscriber(server)
Stackdriver Trace্ͷදࣔ publisher subscriber Pub/Sub
exporter
exporter is …
exporter is … • StackdriverDatadogͳͲͷόοΫΤϯυαʔϏεʹ τϨʔεɺ͋Δ͍ϝτϦΫεΛૹΔͨΊͷΈ • σʔλऩू෦ͱexporterʹΑΔόοΫΤϯυαʔϏε ͕͞Ε͍ͯΔͷ͕OpenCensusͷ1ͭͷಛ
exporterͷ࣮(Go) type Exporter interface { ExportSpan(s *SpanData) } • ExporterΠϯλϑΣʔεΛ࣮͢Δ͚ͩͰྑ͍
• ҙͷόοΫΤϯυαʔϏεAPIͳͲΛୟ͍ͯσʔλΛ ૹ͍ͬͯΔ • ࣮ࡍʹύϑΥʔϚϯεͰ͔ͳΓؾΛ͍ͬͯΔ(ͣ)
custom exporter
Tail LatencyΛิ͢ΔͨΊͷexporter • Tail Latency • 99ύʔηϯλΠϧͷΑ͏ͳͷ • ௨ৗɺτϥϑΟοΫͷଟ͍αʔϏεͩͱαϯϓϦϯά ͷඞཁ͕͋Δ͕ɺͦ͏͢ΔͱTail
Latency͕શ͘ه͞ Εͳ͍ • Tail LatencyͰΜͰ͍Δͱ͖मਖ਼ɺܭଌͷαΠΫ ϧΛճ͍͕ͨ͠ɺ؍ଌ͢ΔͨΊʹ࣌ؒҎ্͔͔Δ ͜ͱ͋Δ…
edge exporter • https://github.com/takashabe/edge-exporter • ϓϩμΫγϣϯೖग़དྷͯͳ͍ͷͰࢀߟ࣮ͱͯ͠… • ύϑΥʔϚϯεվળͷ༨͕େ͍ʹͬͯΔ • ߏΛ࿅͍ͬͯΔ࣌ʹϓϩδΣΫτҠಈʹͳͬͨ
• ଞexporterͷલஈͰproxyͱͯ͠ಈ࡞ͤ͞Δ • શ݅αϯϓϦϯάͭͭ͠ɺҰఆ࣌ؒ͝ͱʹ࠷ϨΠςϯ γͷߴ͍τϨʔε͚ͩΛ࠾༻͢Δ • Φʔόʔϔου͋Δ
spanϥΠϑαΠΫϧ • ࠓճͷΑ͏ʹTail LatencyΛั·͑ΔͨΊʹશ݅αϯ ϓϦϯά͢Δඞཁ͕ग़ͯ͘Δ • αϯϓϦϯάରͰͳ͍spanʹର͢Δૢ࡞ૣظϦ λʔϯ͢ΔΑ͏ʹͳ͍ͬͯΔ • αϯϓϦϯάରʹͳ͍ͬͯΔ͚ͩͰएׯͷΦʔόʔ
ϔου͕͋Δ _, span := trace.StartSpan(ctx, name) // αϯϓϦϯάܾఆ. SpanDatasੜ span.AddAttributes(...) // αϯϓϦϯάର֎ͳΒεΩοϓ ... span.End() // ExportSpan͕ݺΕΔ
edge exporter࣮ type EdgeExporter struct { // Stackdriver exporterͳͲͷଞexporterΛอ࣋͢Δ exporters
exportersList // Tail LatencyͷܭଌִؒͱRateLimit interval time.Duration limiter *rate.Limiter // interval͝ͱͷTail LatencyΛอ࣋͢Δ tail *trace.SpanData tailLatency time.Duration tailMu sync.Mutex } • EdgeExporterߏମ
edge exporter࣮ • ExportSpan() func (e *EdgeExporter) ExportSpan(sd *trace.SpanData) {
e.storeTailLatencySpan(sd) if !e.limiter.Allow() { return } e.tailMu.Lock() defer e.tailMu.Unlock() if e.tail == nil { return } for _, exp := range e.exporters.Load() { exp.ExportSpan(e.tail) } e.tail = nil e.tailLatency = 0 }
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go 10ඵ͝ͱʹTail Latency SpanΛ Stackdriver exporterʹྲྀ͢
Stackdriver Trace্ͷදࣔ $ echo "GET https://…” | vegeta attack -rate=50
-duration=1m
custom exporterͷεεϝ • ࠓճTail LatencyΛରͱ͕ͨ͠ɺྫ͑Τϥʔ͕ൃ ੜͨ͠ͱ͖͚ͩɺಛఆͷϦΫΤετ͚ͩͳͲԠ༻ޮ ͖ͦ͏ • ܭଌ։࢝࣌(StartSpan)Ͱஅग़དྷͳ͍ͷΛܭଌ͢ ΔͨΊʹࠓճͷΑ͏ʹύϑΥʔϚϯεͱͷτϨʔυ
ΦϑʹͳΓͦ͏
Conclusion
Conclusion • Cloud Pub/Sub༻ͷcontext propagationͱɺTail Latency༻ͷexporterΛ࡞ͬͯΈͨ • ಠ࣮ࣗͷϋʔυϧͦΕ΄Ͳߴ͘ͳ͍ • ؍ଌ͍ͨ͠ͷΛ؍ଌग़དྷΔΑ͏ʹ͠Α͏