Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
OpenCensusでcustom context propagationとexporterを...
Search
takashabe
June 27, 2019
Technology
0
1.6k
OpenCensusでcustom context propagationとexporterを書いた話 / OpenCensus with custom context propagation and exporter
OpenCensus/OpenTelemetry meetup vol.2 の発表資料です。
https://opencensus.connpass.com/event/132588/
takashabe
June 27, 2019
Tweet
Share
More Decks by takashabe
See All by takashabe
より良いターミナルでの生活を求めて
takashabe
0
41
pubsub with concurrent
takashabe
1
870
社内ISUCONを開催した話
takashabe
0
1.6k
ISUCON大反省会
takashabe
0
1.8k
gitのブランチ戦略
takashabe
8
5.8k
サルでもわかるgit
takashabe
0
1.4k
playで複数DBする
takashabe
0
1.6k
MySQLで高トラフィックに立ち向かう
takashabe
0
1.8k
GitHubの良さ
takashabe
2
2.2k
Other Decks in Technology
See All in Technology
LINE API Deep Dive Q1 2025: Unlocking New Possibilities
linedevth
1
160
ウェブアクセシビリティとは
lycorptech_jp
PRO
0
260
ClineにNext.jsのプロジェクト改善をお願いしてみた / 20250321_reacttokyo_LT
optim
1
1.3k
チームビルディング「脅威モデリング」ワークショップ
koheiyoshikawa
0
140
SpannerとAurora DSQLの同時実行制御の違いに想いを馳せる
masakikato5
0
570
ルートユーザーの活用と管理を徹底的に深掘る
yuobayashi
6
720
Multitenant 23ai の全貌 - 機能・設計・実装・運用からマイクロサービスまで
oracle4engineer
PRO
2
120
大規模アジャイル開発のリアル!コミュニケーション×進捗管理×高品質
findy_eventslides
0
500
React Server Componentは 何を解決し何を解決しないのか / What do React Server Components solve, and what do they not solve?
kaminashi
6
1.2k
ペアプログラミングにQAが加わった!職能を超えたモブプログラミングの事例と学び
tonionagauzzi
1
140
コード品質向上で得られる効果と実践的取り組み
ham0215
2
200
3/26 クラウド食堂LT #2 GenU案件を通して学んだ教訓 登壇資料
ymae
1
200
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Thoughts on Productivity
jonyablonski
69
4.5k
A designer walks into a library…
pauljervisheath
205
24k
Product Roadmaps are Hard
iamctodd
PRO
52
11k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Code Review Best Practice
trishagee
67
18k
Scaling GitHub
holman
459
140k
Become a Pro
speakerdeck
PRO
27
5.2k
Typedesign – Prime Four
hannesfritz
41
2.6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
31
4.7k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.6k
Transcript
OpenCensus/OpenTelemetry meetup vol.2 @takashabe OpenCensusͰcustom context propagationͱexporterΛॻ͍ ͨ
whoami • Takashi Abe (@takashabe) • גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫελδΦ • GoͰࠂ৴ϓϩμΫτΛ࡞͍ͬͯ·͢
• KubernetesͰMicroservicesతͳͷ • ਪ͠ΩʔϘʔυirisͰ͢ • Corne Cherry࡞த…
͡Ίʹ • Go + OpenCensus + Stackdriver Λલఏͱ͍ͯ͠·͢
࣍ • ࢄτϨʔγϯάͷ֓ཁ • context propagation • exporter • ·ͱΊ
ࢄτϨʔγϯά
ࢄτϨʔγϯά https://github.com/census-instrumentation/opencensus-go • 1 traceʹରͯ͠ɺෳͷspan͕ඥͮ͘ • spanಉҰΞϓϦέʔγϣϯͰෳൃߦͯ͠ྑ͍͠ɺҟͳΔΞϓϦέʔγϣϯͰ τϨʔεID͕ಉҰͰ͋Εඥͮ͘
context propagation
context propagation is …
context propagation is … • τϨʔεIDΛൖ͢ΔͨΊͷϓϩτίϧతͳͷ • opencensus-goͰocgrpcochttpͰ࣮͞Ε͍ͯΔ • SpanContextΛհͯ͠σʔλΛൖͤ͞Δ
• TraceID, SpanID, TraceOptions(αϯϓϦϯάର͔Ͳ ͏͔) type SpanContext struct { TraceID TraceID // [16]byte SpanID SpanID // [8]byte TraceOptions TraceOptions // uint32 Tracestate *tracestate.Tracestate // Ұ෦ͷpropagation༻ }
ocgrpc, ochttp ocgrpc ochttp
HTTPFormatͷ࣮ • ocgrpc࣮͕1ͭ • ochttpͰHTTPFormatΛ࣮ͨ͠b3ͱtracecontextͱ͍͏2 ͭͷ࣮͕ଘࡏ͢Δ • b3: ZipkinͰఆٛ͞Εͨpropagation(σϑΥϧτ) •
https://github.com/openzipkin/b3-propagation • tracecontext: w3cͰఆٛ͞Εͨpropagation • https://github.com/w3c/trace-context type HTTPFormat interface { SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) SpanContextToRequest(sc trace.SpanContext, req *http.Request) }
B3 Propagation https://github.com/openzipkin/b3-propagation
B3 Propagation https://github.com/openzipkin/b3-propagation 4QBO$POUFYU'SPN3FRVFTU 4QBO$POUFYU5P3FRVFTU
OpenCensusͰͷ࣮(ToRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextToRequest(sc trace.SpanContext, req *http.Request) { req.Header.Set(TraceIDHeader, hex.EncodeToString(sc.TraceID[:])) req.Header.Set(SpanIDHeader, hex.EncodeToString(sc.SpanID[:])) var sampled string if sc.IsSampled() { sampled = "1" } else { sampled = "0" } req.Header.Set(SampledHeader, sampled) } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
OpenCensusͰͷ࣮(FromRequest) const ( TraceIDHeader = "X-B3-TraceId" SpanIDHeader = "X-B3-SpanId" SampledHeader
= "X-B3-Sampled" ) func (f *HTTPFormat) SpanContextFromRequest(req *http.Request) (sc trace.SpanContext, ok bool) { tid, ok := ParseTraceID(req.Header.Get(TraceIDHeader)) if !ok { return trace.SpanContext{}, false } sid, ok := ParseSpanID(req.Header.Get(SpanIDHeader)) if !ok { return trace.SpanContext{}, false } sampled, _ := ParseSampled(req.Header.Get(SampledHeader)) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, }, true } https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/propagation/b3/b3.go
ochttpΛར༻͢Δ // import "go.opencensus.io/plugin/ochttp" mux := http.NewServeMux() mux.Handle("/users", ochttp.WithRouteTag(usersHandler, "/users"))
log.Fatal(http.ListenAndServe("localhost:8080", &ochttp.Handler{ Handler: mux, Propagation: &b3.HTTPFormat{}, })) https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/example_test.go • ochttp.Handler͕ϛυϧΣΞͱͯ͠τϨʔγϯάपΓ Λ໘ݟͯ͘ΕΔ • context propagationҙࣝ͠ͳͯ͘ྑ͍
custom context propagation
Cloud Pub/Sub ʹద༻ͨ͠ • GCPͷϚωʔδυPub/Sub • B3 PropagationΛϕʔεʹͯ͠ɺϝοηʔδͷ AttributesʹσʔλΛ٧Ίͯൖͤ͞Δ •
https://github.com/takashabe/oc-propagation-demo • ΄΅ಉ͡ͷΛϓϩμΫγϣϯʹಋೖࡁΈ͕ͩɺڥ ґଘ͕͋ΔͷͰαϯϓϧ࣮Λ༻ҙ • αʔό༻ҙ͢Δͷ໘ͰCloud Run༻ʹHTTPαʔό ͱ࣮͍ͯͯ͠͠ΔͷͰएׯݟͮΒ͍
Pub/Sub൛ͷ࣮(ToRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func WrapMessage(ctx context.Context, m *pubsub.Message) *pubsub.Message { if m.Attributes != nil { if m.Attributes[TraceIDField] != "" || m.Attributes[SpanIDField] != "" { return m } } else { m.Attributes = make(map[string]string, 3) } sc := trace.FromContext(ctx).SpanContext() m.Attributes[TraceIDField] = sc.TraceID.String() m.Attributes[SpanIDField] = sc.SpanID.String() m.Attributes[SampledField] = fmt.Sprintf("%t", sc.IsSampled()) return m }
Pub/Sub൛ͷ࣮(FromRequest) const ( TraceIDField = "X-Pubsub-TraceId" SpanIDField = "X-Pubsub-SpanId" SampledField
= "X-Pubsub-Sampled" ) func SpanContextFromMessage(m *pubsub.Message) trace.SpanContext { if m.Attributes == nil { return trace.SpanContext{} } tid, ok := parseTraceID(m.Attributes[TraceIDField]) if !ok { return trace.SpanContext{} } sid, ok := parseSpanID(m.Attributes[SpanIDField]) if !ok { return trace.SpanContext{} } sampled := parseSampled(m.Attributes[SampledField]) return trace.SpanContext{ TraceID: tid, SpanID: sid, TraceOptions: sampled, } }
ར༻ଆ // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... ctx, span := trace.StartSpan(context.Background(), "publish")
defer span.End() msg := &pubsub.Message{ Data: []byte("foo"), } topic.Publish(ctx, propagation.WrapMessage(ctx, msg)).Get(ctx) // import “github.com/takashabe/oc-propagation-demo/internal/propagation" ... subscription.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) { sc := propagation.SpanContextFromMessage(msg) _, span := trace.StartSpanWithRemoteParent(ctx, "receive", sc) defer span.End() publisher(client) subscriber(server)
Stackdriver Trace্ͷදࣔ publisher subscriber Pub/Sub
exporter
exporter is …
exporter is … • StackdriverDatadogͳͲͷόοΫΤϯυαʔϏεʹ τϨʔεɺ͋Δ͍ϝτϦΫεΛૹΔͨΊͷΈ • σʔλऩू෦ͱexporterʹΑΔόοΫΤϯυαʔϏε ͕͞Ε͍ͯΔͷ͕OpenCensusͷ1ͭͷಛ
exporterͷ࣮(Go) type Exporter interface { ExportSpan(s *SpanData) } • ExporterΠϯλϑΣʔεΛ࣮͢Δ͚ͩͰྑ͍
• ҙͷόοΫΤϯυαʔϏεAPIͳͲΛୟ͍ͯσʔλΛ ૹ͍ͬͯΔ • ࣮ࡍʹύϑΥʔϚϯεͰ͔ͳΓؾΛ͍ͬͯΔ(ͣ)
custom exporter
Tail LatencyΛิ͢ΔͨΊͷexporter • Tail Latency • 99ύʔηϯλΠϧͷΑ͏ͳͷ • ௨ৗɺτϥϑΟοΫͷଟ͍αʔϏεͩͱαϯϓϦϯά ͷඞཁ͕͋Δ͕ɺͦ͏͢ΔͱTail
Latency͕શ͘ه͞ Εͳ͍ • Tail LatencyͰΜͰ͍Δͱ͖मਖ਼ɺܭଌͷαΠΫ ϧΛճ͍͕ͨ͠ɺ؍ଌ͢ΔͨΊʹ࣌ؒҎ্͔͔Δ ͜ͱ͋Δ…
edge exporter • https://github.com/takashabe/edge-exporter • ϓϩμΫγϣϯೖग़དྷͯͳ͍ͷͰࢀߟ࣮ͱͯ͠… • ύϑΥʔϚϯεվળͷ༨͕େ͍ʹͬͯΔ • ߏΛ࿅͍ͬͯΔ࣌ʹϓϩδΣΫτҠಈʹͳͬͨ
• ଞexporterͷલஈͰproxyͱͯ͠ಈ࡞ͤ͞Δ • શ݅αϯϓϦϯάͭͭ͠ɺҰఆ࣌ؒ͝ͱʹ࠷ϨΠςϯ γͷߴ͍τϨʔε͚ͩΛ࠾༻͢Δ • Φʔόʔϔου͋Δ
spanϥΠϑαΠΫϧ • ࠓճͷΑ͏ʹTail LatencyΛั·͑ΔͨΊʹશ݅αϯ ϓϦϯά͢Δඞཁ͕ग़ͯ͘Δ • αϯϓϦϯάରͰͳ͍spanʹର͢Δૢ࡞ૣظϦ λʔϯ͢ΔΑ͏ʹͳ͍ͬͯΔ • αϯϓϦϯάରʹͳ͍ͬͯΔ͚ͩͰएׯͷΦʔόʔ
ϔου͕͋Δ _, span := trace.StartSpan(ctx, name) // αϯϓϦϯάܾఆ. SpanDatasੜ span.AddAttributes(...) // αϯϓϦϯάର֎ͳΒεΩοϓ ... span.End() // ExportSpan͕ݺΕΔ
edge exporter࣮ type EdgeExporter struct { // Stackdriver exporterͳͲͷଞexporterΛอ࣋͢Δ exporters
exportersList // Tail LatencyͷܭଌִؒͱRateLimit interval time.Duration limiter *rate.Limiter // interval͝ͱͷTail LatencyΛอ࣋͢Δ tail *trace.SpanData tailLatency time.Duration tailMu sync.Mutex } • EdgeExporterߏମ
edge exporter࣮ • ExportSpan() func (e *EdgeExporter) ExportSpan(sd *trace.SpanData) {
e.storeTailLatencySpan(sd) if !e.limiter.Allow() { return } e.tailMu.Lock() defer e.tailMu.Unlock() if e.tail == nil { return } for _, exp := range e.exporters.Load() { exp.ExportSpan(e.tail) } e.tail = nil e.tailLatency = 0 }
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go
example func main() { sd, _ := stackdriver.NewExporter(stackdriver.Options{ ProjectID: os.Getenv("PROJECT_ID"),
}) edge := edgeexporter.New(edgeexporter.WithExportInterval(10*time.Second)) edge.RegisterExporter(sd) trace.RegisterExporter(edge) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), }) http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) } var cnt int64 func handler(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(context.Background(), "handler") c := atomic.LoadInt64(&cnt) if c%5 == 0 { time.Sleep(50 * time.Millisecond) } span.End() atomic.AddInt64(&cnt, 1) } https://github.com/takashabe/edge-exporter/blob/master/example/main.go 10ඵ͝ͱʹTail Latency SpanΛ Stackdriver exporterʹྲྀ͢
Stackdriver Trace্ͷදࣔ $ echo "GET https://…” | vegeta attack -rate=50
-duration=1m
custom exporterͷεεϝ • ࠓճTail LatencyΛରͱ͕ͨ͠ɺྫ͑Τϥʔ͕ൃ ੜͨ͠ͱ͖͚ͩɺಛఆͷϦΫΤετ͚ͩͳͲԠ༻ޮ ͖ͦ͏ • ܭଌ։࢝࣌(StartSpan)Ͱஅग़དྷͳ͍ͷΛܭଌ͢ ΔͨΊʹࠓճͷΑ͏ʹύϑΥʔϚϯεͱͷτϨʔυ
ΦϑʹͳΓͦ͏
Conclusion
Conclusion • Cloud Pub/Sub༻ͷcontext propagationͱɺTail Latency༻ͷexporterΛ࡞ͬͯΈͨ • ಠ࣮ࣗͷϋʔυϧͦΕ΄Ͳߴ͘ͳ͍ • ؍ଌ͍ͨ͠ͷΛ؍ଌग़དྷΔΑ͏ʹ͠Α͏