Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenCensusでcustom context propagationとexporterを...
Search
takashabe
June 27, 2019
Technology
0
1.7k
OpenCensusでcustom context propagationとexporterを書いた話 / OpenCensus with custom context propagation and exporter
OpenCensus/OpenTelemetry meetup vol.2 の発表資料です。
https://opencensus.connpass.com/event/132588/
takashabe
June 27, 2019
Tweet
Share
More Decks by takashabe
See All by takashabe
より良いターミナルでの生活を求めて
takashabe
0
55
pubsub with concurrent
takashabe
1
920
社内ISUCONを開催した話
takashabe
0
1.7k
ISUCON大反省会
takashabe
0
1.9k
gitのブランチ戦略
takashabe
8
5.9k
サルでもわかるgit
takashabe
0
1.5k
playで複数DBする
takashabe
0
1.6k
MySQLで高トラフィックに立ち向かう
takashabe
0
1.8k
GitHubの良さ
takashabe
2
2.2k
Other Decks in Technology
See All in Technology
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
320
KMP の Swift export
kokihirokawa
0
310
コンテキストエンジニアリングとは? 考え方と応用方法
findy_eventslides
4
870
多野優介
tanoyusuke
1
170
Geospatialの世界最前線を探る [2025年版]
dayjournal
3
470
"複雑なデータ処理 × 静的サイト" を両立させる、楽をするRails運用 / A low-effort Rails workflow that combines “Complex Data Processing × Static Sites”
hogelog
3
1.7k
“2件同時配達”の開発舞台裏 〜出前館PMが挑んだダブルピック実現に向けた体験設計〜
demaecan
0
180
Azure Well-Architected Framework入門
tomokusaba
0
200
Go Conference 2025: GoのinterfaceとGenericsの内部構造と進化 / Go type system internals
ryokotmng
3
590
いまさら聞けない ABテスト入門
skmr2348
1
180
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
GC25 Recap+: Advancing Go Garbage Collection with Green Tea
logica0419
1
340
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Raft: Consensus for Rubyists
vanstee
139
7.1k
Making the Leap to Tech Lead
cromwellryan
135
9.5k
Optimizing for Happiness
mojombo
379
70k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Site-Speed That Sticks
csswizardry
11
870
Building an army of robots
kneath
306
46k
Agile that works and the tools we love
rasmusluckow
331
21k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
32
2.2k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Typedesign – Prime Four
hannesfritz
42
2.8k
Transcript
OpenCensus/OpenTelemetry meetup vol.2 @takashabe OpenCensusͰcustom context propagationͱexporterΛॻ͍ ͨ
whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • GoͰࠂ৴ϓϩμΫτΛ࡞͍ͬͯ·͢
• KubernetesͰMicroservicesతͳͷ • ਪ͠ΩʔϘʔυirisͰ͢ • Corne Cherry࡞த…
͡Ίʹ • Go + OpenCensus + Stackdriver Λલఏͱ͍ͯ͠·͢
࣍ • ࢄτϨʔγϯάͷ֓ཁ • context propagation • exporter • ·ͱΊ
ࢄτϨʔγϯά
ࢄτϨʔγϯά https://github.com/census-instrumentation/opencensus-go • 1 traceʹରͯ͠ɺෳͷspan͕ඥͮ͘ • spanಉҰΞϓϦέʔγϣϯͰෳൃߦͯ͠ྑ͍͠ɺҟͳΔΞϓϦέʔγϣϯͰ τϨʔεID͕ಉҰͰ͋Εඥͮ͘
context propagation
context propagation is …
context propagation is … • τϨʔεIDΛൖ͢ΔͨΊͷϓϩτίϧతͳͷ • opencensus-goͰocgrpcochttpͰ࣮͞Ε͍ͯΔ • SpanContextΛհͯ͠σʔλΛൖͤ͞Δ
• TraceID, SpanID, TraceOptions(αϯϓϦϯάର͔Ͳ ͏͔) type SpanContext struct { TraceID TraceID // [16]byte SpanID SpanID // [8]byte TraceOptions TraceOptions // uint32 Tracestate *tracestate.Tracestate // Ұ෦ͷpropagation༻ }
ocgrpc, ochttp ocgrpc ochttp
HTTPFormatͷ࣮ • ocgrpc࣮͕1ͭ • ochttpͰHTTPFormatΛ࣮ͨ͠b3ͱtracecontextͱ͍͏2 ͭͷ࣮͕ଘࡏ͢Δ • b3: ZipkinͰఆٛ͞Εͨpropagation(σϑΥϧτ) •
https://github.com/openzipkin/b3-propagation • tracecontext: w3cͰఆٛ͞Εͨpropagation • https://github.com/w3c/trace-context type HTTPFormat interface { SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) SpanContextToRequest(sc trace.SpanContext, req *http.Request) }
B3 Propagation https://github.com/openzipkin/b3-propagation
B3 Propagation https://github.com/openzipkin/b3-propagation 4QBO$POUFYU'SPN3FRVFTU 4QBO$POUFYU5P3FRVFTU
OpenCensusͰͷ࣮(ToRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextToRequest(sc trace.SpanContext, req *http.Request) { req.Header.Set(TraceIDHeader, hex.EncodeToString(sc.TraceID[:])) req.Header.Set(SpanIDHeader, hex.EncodeToString(sc.SpanID[:])) var sampled string if sc.IsSampled() { sampled = "1" } else { sampled = "0" } req.Header.Set(SampledHeader, sampled) } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
OpenCensusͰͷ࣮(FromRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) { tid, ok := ParseTraceID(req.Header.Get(TraceIDHeader)) if !ok { return trace.SpanContext{}, false } sid, ok := ParseSpanID(req.Header.Get(SpanIDHeader)) if !ok { return trace.SpanContext{}, false } sampled, _ := ParseSampled(req.Header.Get(SampledHeader)) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, }, true } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
ochttpΛར༻͢Δ // import "go.opencensus.io/plugin/ochttp" mux := http.NewServeMux() mux.Handle("/users", ochttp.WithRouteTag(usersHandler, "/users"))
log.Fatal(http.ListenAndServe("localhost:8080", &ochttp.Handler{ Handler: mux, Propagation: &b3.HTTPFormat{}, })) https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/example_test.go • ochttp.Handler͕ϛυϧΣΞͱͯ͠τϨʔγϯάपΓ Λ໘ݟͯ͘ΕΔ • context propagationҙࣝ͠ͳͯ͘ྑ͍
custom context propagation
Cloud Pub/Sub ʹద༻ͨ͠ • GCPͷϚωʔδυPub/Sub • B3 PropagationΛϕʔεʹͯ͠ɺϝοηʔδͷ AttributesʹσʔλΛ٧Ίͯൖͤ͞Δ •
https://github.com/takashabe/oc-propagation-demo • ΄΅ಉ͡ͷΛϓϩμΫγϣϯʹಋೖࡁΈ͕ͩɺڥ ґଘ͕͋ΔͷͰαϯϓϧ࣮Λ༻ҙ • αʔό༻ҙ͢Δͷ໘ͰCloud Run༻ʹHTTPαʔό ͱ࣮͍ͯͯ͠͠ΔͷͰएׯݟͮΒ͍
Pub/Sub൛ͷ࣮(ToRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func WrapMessage(ctx context.Context, m *pubsub.Message) *pubsub.Message { if m.Attributes != nil { if m.Attributes[TraceIDField] != "" || m.Attributes[SpanIDField] != "" { return m } } else { m.Attributes = make(map[string]string, 3) } sc := trace.FromContext(ctx).SpanContext() m.Attributes[TraceIDField] = sc.TraceID.String() m.Attributes[SpanIDField] = sc.SpanID.String() m.Attributes[SampledField] = fmt.Sprintf("%t", sc.IsSampled()) return m }
Pub/Sub൛ͷ࣮(FromRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func SpanContextFromMessage(m *pubsub.Message) trace.SpanContext { if m.Attributes == nil { return trace.SpanContext{} } tid, ok := parseTraceID(m.Attributes[TraceIDField]) if !ok { return trace.SpanContext{} } sid, ok := parseSpanID(m.Attributes[SpanIDField]) if !ok { return trace.SpanContext{} } sampled := parseSampled(m.Attributes[SampledField]) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, } }
ར༻ଆ // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... ctx, span := trace.StartSpan(context.Background(), "publish")
defer span.End() msg := &pubsub.Message{ Data: []byte("foo"), } topic.Publish(ctx, propagation.WrapMessage(ctx, msg)).Get(ctx) // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { sc := propagation.SpanContextFromMessage(msg) _, span := trace.StartSpanWithRemoteParent(ctx, "receive", sc) defer span.End() publisher(client) subscriber(server)
Stackdriver Trace্ͷදࣔ publisher subscriber Pub/Sub
exporter
exporter is …
exporter is … • StackdriverDatadogͳͲͷόοΫΤϯυαʔϏεʹ τϨʔεɺ͋Δ͍ϝτϦΫεΛૹΔͨΊͷΈ • σʔλऩू෦ͱexporterʹΑΔόοΫΤϯυαʔϏε ͕͞Ε͍ͯΔͷ͕OpenCensusͷ1ͭͷಛ
exporterͷ࣮(Go) type Exporter interface { ExportSpan(s *SpanData) } • ExporterΠϯλϑΣʔεΛ࣮͢Δ͚ͩͰྑ͍
• ҙͷόοΫΤϯυαʔϏεAPIͳͲΛୟ͍ͯσʔλΛ ૹ͍ͬͯΔ • ࣮ࡍʹύϑΥʔϚϯεͰ͔ͳΓؾΛ͍ͬͯΔ(ͣ)
custom exporter
Tail LatencyΛิ͢ΔͨΊͷexporter • Tail Latency • 99ύʔηϯλΠϧͷΑ͏ͳͷ • ௨ৗɺτϥϑΟοΫͷଟ͍αʔϏεͩͱαϯϓϦϯά ͷඞཁ͕͋Δ͕ɺͦ͏͢ΔͱTail
Latency͕શ͘ه͞ Εͳ͍ • Tail LatencyͰΜͰ͍Δͱ͖मਖ਼ɺܭଌͷαΠΫ ϧΛճ͍͕ͨ͠ɺ؍ଌ͢ΔͨΊʹ࣌ؒҎ্͔͔Δ ͜ͱ͋Δ…
edge exporter • https://github.com/takashabe/edge-exporter • ϓϩμΫγϣϯೖग़དྷͯͳ͍ͷͰࢀߟ࣮ͱͯ͠… • ύϑΥʔϚϯεվળͷ༨͕େ͍ʹͬͯΔ • ߏΛ࿅͍ͬͯΔ࣌ʹϓϩδΣΫτҠಈʹͳͬͨ
• ଞexporterͷલஈͰproxyͱͯ͠ಈ࡞ͤ͞Δ • શ݅αϯϓϦϯάͭͭ͠ɺҰఆ࣌ؒ͝ͱʹ࠷ϨΠςϯ γͷߴ͍τϨʔε͚ͩΛ࠾༻͢Δ • Φʔόʔϔου͋Δ
spanϥΠϑαΠΫϧ • ࠓճͷΑ͏ʹTail LatencyΛั·͑ΔͨΊʹશ݅αϯ ϓϦϯά͢Δඞཁ͕ग़ͯ͘Δ • αϯϓϦϯάରͰͳ͍spanʹର͢Δૢ࡞ૣظϦ λʔϯ͢ΔΑ͏ʹͳ͍ͬͯΔ • αϯϓϦϯάରʹͳ͍ͬͯΔ͚ͩͰएׯͷΦʔόʔ
ϔου͕͋Δ _, span := trace.StartSpan(ctx, name) // αϯϓϦϯάܾఆ. SpanDatasੜ span.AddAttributes(...) // αϯϓϦϯάର֎ͳΒεΩοϓ ... span.End() // ExportSpan͕ݺΕΔ
edge exporter࣮ type EdgeExporter struct { // Stackdriver exporterͳͲͷଞexporterΛอ࣋͢Δ exporters
exportersList // Tail LatencyͷܭଌִؒͱRateLimit interval time.Duration limiter *rate.Limiter // interval͝ͱͷTail LatencyΛอ࣋͢Δ tail *trace.SpanData tailLatency time.Duration tailMu sync.Mutex } • EdgeExporterߏମ
edge exporter࣮ • ExportSpan() func (e *EdgeExporter) ExportSpan(sd *trace.SpanData) {
e.storeTailLatencySpan(sd) if !e.limiter.Allow() { return } e.tailMu.Lock() defer e.tailMu.Unlock() if e.tail == nil { return } for _, exp := range e.exporters.Load() { exp.ExportSpan(e.tail) } e.tail = nil e.tailLatency = 0 }
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go 10ඵ͝ͱʹTail Latency SpanΛ Stackdriver exporterʹྲྀ͢
Stackdriver Trace্ͷදࣔ $ echo "GET https://…” | vegeta attack -rate=50
-duration=1m
custom exporterͷεεϝ • ࠓճTail LatencyΛରͱ͕ͨ͠ɺྫ͑Τϥʔ͕ൃ ੜͨ͠ͱ͖͚ͩɺಛఆͷϦΫΤετ͚ͩͳͲԠ༻ޮ ͖ͦ͏ • ܭଌ։࢝࣌(StartSpan)Ͱஅग़དྷͳ͍ͷΛܭଌ͢ ΔͨΊʹࠓճͷΑ͏ʹύϑΥʔϚϯεͱͷτϨʔυ
ΦϑʹͳΓͦ͏
Conclusion
Conclusion • Cloud Pub/Sub༻ͷcontext propagationͱɺTail Latency༻ͷexporterΛ࡞ͬͯΈͨ • ಠ࣮ࣗͷϋʔυϧͦΕ΄Ͳߴ͘ͳ͍ • ؍ଌ͍ͨ͠ͷΛ؍ଌग़དྷΔΑ͏ʹ͠Α͏