An introduction to monitoring Go Application with OpenCensus

Go(Un)Conference 3kg
https://gounconference.connpass.com/event/92794/

Transcript

1. An introduc+on to monitoring Go Applica+on with OpenCensus @munisystem
2. Agenda • APM ͱ͸ͳʹ͔ɺݱঢ়ͷ՝୊ • OpenCensus ͷ֓ཁ • OpenCensus ʹΑΔ Go ͷύϑΥʔϚϯεϞχλϦϯά
3. APM ͱ͸ͳʹ͔ɺݱঢ়ͷ՝୊
4. αʔϏε։ൃʹ͓͚ΔϝτϦΫε αʔϏε͕͖ͪΜͱఏڙͰ͖͍ͯΔͷ͔ΛܭΔͨΊͷࢦඪ = ϝτϦΫε ϝτϦΫε͸େ͖͘3ͭʹ෼ྨͰ͖Δ • Business metrics • Applica1on metrics • Infrastructure metrics
5. Business metrics Ϗδωεͷ໨ඪΛୡ੒͢ΔͨΊͷݸʑͷϓϩηεʹର͢ΔϝτϦ Ϋε • Conversion Rate • DAU / WAU / MAU • PV • Ac7vity Log
6. Applica'on metrics ΞϓϦέʔγϣϯͷύϑΥʔϚϯεʹର͢ΔϝτϦΫε • Throughput / API latency • RPS / RPM • Error rate • Language specific metrics
7. Infrastructure metrics ΞϓϦέʔγϣϯΛσϦόϦʔ͢Δ؀ڥʹର͢ΔϝτϦΫε • Memory / CPU / Disk usage • Load Average • Network latency
8. Applica'on metrics ͷॏཁੑ Կނ Applica(on metrics Λऔಘ͢Δ͜ͱ͕ॏཁͳͷ͔ʁ • ύϑΥʔϚϯε͕αʔϏεʹ༩͑ΔΠϯύΫτ • αʔϏεͷ੒௕ͱػೳ௥Ճʹର͢ΔજࡏతͳϦεΫ • ϚΠΫϩαʔϏεʹΑΔύϑΥʔϚϯε෼ੳͷࠔ೉Խ
9. ύϑΥʔϚϯε͕αʔϏεʹ༩͑ΔΠϯύΫτ ΞϓϦέʔγϣϯͷύϑΥʔϚϯε͕ CVR ΍ϦςϯγϣϯʹӨڹ Λ༩͑Δ͜ͱ͕Ұൠతʹ஌ΒΕ͍ͯΔ 1 Our experiments demonstrate that slowing down the search results page by 100 to 400 milliseconds has a measurable impact on the number of searches per user of -0.2% to -0.6% (averaged over four or six weeks depending on the experiment). That's 0.2% to 0.6% fewer searches for changes under half a second! 1 Speed Ma)ers h)ps:/ /ai.googleblog.com/2009/06/speed-ma)ers.html
10. αʔϏεͷ੒௕ͱػೳ௥Ճʹର͢ΔજࡏతͳϦεΫ αʔϏε͕੒௕͢ΔʹͭΕΞϓϦέʔγϣϯͷن໛ͱ૊৫΋େ͖ ͘ͳΓɺ·ͨજࡏతͳϦεΫ΋ߴ͘ͳΔ • σʔλྔͷ૿Ճʹ଱͑ΒΕͳ͍ SQL ͷൃߦ • ผαʔόʹର͢Δ API call ͷ૿Ճ αʔϏεͷ৴པੑΛҡ࣋͢ΔͨΊʹ໰୊Λ͍ͪૣ͘ݕ஌͠ରԠ͢ Δ͜ͱ͕ٻΊΒΕΔ
11. ɹϚΠΫϩαʔϏεʹΑΔύϑΥʔϚϯε෼ੳͷࠔ೉Խ ϚΠΫϩαʔϏεͰ͸αʔϏεؒͷ௨৴͕ෳࡶԽɾෆಁ໌Խ͢Δ ͦͷ݁Ռͱͯ͠αʔϏεશମͷৼΔ෣͍΍αʔϏεͷґଘؔ܎ͷ ೺ѲɺγεςϜͷύϑΥʔϚϯεͷ෼ੳɺ·ͨΤϥʔ࣌ʹͲͷ αʔϏε͕໰୊ͩͬͨͷ͔ͷௐ͕ࠪϞϊϦγοΫΞʔΩςΫνϟ ͱൺֱͯ͠ࠔ೉ʹͳΔ શମ૾Λ೺Ѳ͠ϘτϧωοΫΛௐࠪ͢ΔͨΊͷج൫ͱͯ͠෼ࢄτ Ϩʔγϯά͕ඞཁ
12. APM (Applica+on Performance Monitoring) ఏڙ͍ͯ͠ΔαʔϏεͷ৴པੑͱՁ஋Λམͱ͞ͳ͍ͨΊʹɺΞϓ ϦέʔγϣϯͷύϑΥʔϚϯεΛ؂ࢹɺՄࢹԽ͢Δ͜ͱʹΑͬͯ ໰୊Λૣظൃݟ͠ະવʹ๷͙ͨΊͷ࢓૊Έ
13. APM ͷݱঢ় APM ͸େ͖͘2ͭͷ༻్͕͋ΓɺͦΕʹԠͨ͡αʔϏεͷΫϥΠ ΞϯτΛར༻͍ͯ͠Δ • ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷՄࢹԽɾ؂ࢹ • Datadog / CloudWatch / Mackerel / Prometheus • ϚΠΫϩαʔϏεͷαʔϏεؒ௨৴ͷՄࢹԽ (෼ࢄτϨʔγϯ ά) • Datadog APM / Stackdriver Trace / Amazon X-Ray / Zipkin
14. APM ͷ՝୊ ϝτϦΫεΛऩू͠όοΫΤϯυʹૹΔػߏ͸େ൒ͷΫϥΠΞϯ τͰ͸ڞ௨Խ͞Ε͍ͯͳ͍ͨΊɺΞϓϦέʔγϣϯͷ࣮૷͕αʔ Ϗεʹґଘͯ͠͠·͏ ͜Ε͸૊৫΍νʔϜɺΞϓϦέʔγϣϯΛߏங͢ΔΠϯϑϥʹ Αͬͯద੾ͳαʔϏεΛબͿΫϥ΢υωΠςΟϒͳ։ൃελΠϧ ʹ͸ෆ޲͖ => ಛఆͷϕϯμʔʹґଘ͠ͳ͍ڞ௨ͷϑϨʔϜϫʔΫ͕ඞཁ
15. OpenCensus ͷ֓ཁ
16. OpenCensus ͱ͸ APM ʹඞཁͳϝτϦΫεΛऩू͠ɺ֎෦αʔϏεʹఏڙ͢ΔͨΊ ͷϑϨʔϜϫʔΫ • ϝτϦΫεΛऩू͢ΔͨΊͷϓϩτίϧٴͼ֎෦αʔϏεʹఏ ڙ͢ΔͨΊͷΠϯλʔϑΣʔεͷఆٛ • ্هΛ࣮૷֤ͨ͠ݴޠͷϥΠϒϥϦ • Go / Ruby / Python / Java / C++ / Erlang / PHP
17. Ecosystem Design 2 2 h$ps:/ /raw.githubusercontent.com/census-instrumenta8on/opencensus-specs/master/drawings/ EcosystemLayers.png
18. Ecosystem Design ొ৔ਓ෺ • ϝτϦΫεΛѻ͏ͨΊͷ OpenCensus Library • ্هΛར༻ͯ͠ Web Framework ͳͲͰࣗಈతʹϝτϦΫεΛ ऩू͢ΔͨΊͷ Library • औಘͨ͠ϝτϦΫεΛ֎෦αʔϏεʹఏڙ͢ΔͨΊͷ Client Library (Exporter)
19. Exporter OpenCensus Library ଆͰϝτϦΫεΛ֎෦αʔϏεʹఏڙ͢Δͨ ΊͷΠϯλʔϑΣʔε͕ఆٛ͞Ε͍ͯΔ αʔϏεϓϩόΠμ͸ΠϯλʔϑΣʔεΛຬͨ͢Α͏ʹϝτϦΫ εΛม׵͠όοΫΤϯυʹૹΔΑ͏ͳ Client Library Λ࣮૷͢Δ͜ ͱͰ OpenCensus ͷ࿮૊Έʹ৐Δ͜ͱ͕Ͱ͖Δ type Exporter interface { ExportSpan(s *SpanData) }
20. Components 3 3 h$ps:/ /raw.githubusercontent.com/census-instrumenta8on/opencensus-specs/master/drawings/ LibraryComponents.png
21. Components Request local ʹϝτϦΫεΛऔΓѻ͏ͨΊͷ Context ͷ࣮૷ͱɺ ͦΕΛར༻ͨ͠ϝτϦΫεͷͨΊͷίϯϙʔωϯτ • Trace API • ෼ࢄτϨʔγϯάΛ࣮ݱ͢ΔͨΊͷίϯϙʔωϯτ • Tags API (Stats API) • ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεΛऔΓѻ͏ͨΊͷίϯ ϙʔωϯτ
22. OpenCensus Λ࢖͏͜ͱͷϝϦοτ • ։ൃऀ • ڞ௨ͷϑϨʔϜϫʔΫΛ࢖͏͜ͱͰɺϝτϦΫεΛऩू͢Δ ࣮૷Λมߋ͢Δ͜ͱͳ͘αʔϏεͷ੾Γସ͕͑ՄೳʹͳΔ • αʔϏεϓϩόΠμ • Exporter ʹରԠ͢Δ͜ͱͰɺࣗ෼ͷαʔϏεͷ࣮૷Λม͑Δ ͜ͱͳ͘ OpenCensus ͷ࿮૊Έʹ৐Δ͜ͱ͕Ͱ͖Δ
23. OpenCensus ʹΑΔ Go ͷ ύϑΥʔϚϯεϞχλϦϯά
24. Go ͷ OpenCensus Library ͷ࣮૷ঢ়گ Go ͸ http ͱ google.golang.org/grpc ͰϝτϦΫεΛऔΓ ѻ͏ͨΊ࣮૷͕ OpenCensus Library ʹؚ·Ε͍ͯΔ • Request local ͳίϯςΩετٴͼ࣍ͷαʔόʹ఻ൖͤ͞ΔͨΊ ͷ࣮૷ • ͲͷΑ͏ͳσʔλߏ଄Ͱ఻ൖͤ͞Δ͔ͷ࢓༷͸ W3C ͷ Working Group ͰܾΊΒΕ͍ͯΔ • RPC ʹඥͮ͘ϝτϦΫεΛཪͰऔಘ͢ΔͨΊͷ࣮૷
25. Go ͷ OpenCensus Library ͷ࣮૷ঢ়گ Exporter ʹҎԼͷ࣮૷͕͋Δ • Prometheus • Stackdriver • Amazon X-Ray • DataDog • and more...
26. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ Stackdriver ༻ͷ Exporter ͷ ini1alize exporter, err := stackdriver.NewExporter(stackdriver.Options{ ProjectID: projectID, }) if err != nil { log.Fatal(err) }
27. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ http.Handler ʹର͢ΔϝτϦΫε (Stats) Λऔಘɾ؅ཧ͢Δ View ͷઃఆͱɺExporter ͷొ࿥ view.RegisterExporter(exporter) if err := view.Register( ochttp.DefaultServerViews..., ); err != nil { log.Fatal(err) }
28. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ ෼ࢄτϨʔγϯάͰऔಘ͢ΔϝτϦΫε (Traces) ͷઃఆͱɺ Exporter ͷొ࿥ trace.RegisterExporter(exporter) trace.ApplyConfig(trace.Config{ DefaultSampler: trace.AlwaysSample(), })
29. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷ Handler ͷઃఆ http.HandleFunc("/a", func(w http.ResponseWriter, req *http.Request) { client := &http.Client{Transport: &ochttp.Transport{ Propagation: &tracecontext.HTTPFormat{}, }} ctx, span := trace.StartSpan(req.Context(), "in a") defer span.End() r, _ := http.NewRequest("GET", "http://localhost:50030/b", nil) resp, err := client.Do(r.WithContext(ctx)) if err != nil { log.Println(err) } _ = resp fmt.Fprintf(w, "I am /a") }) http.HandleFunc("/b", func(w http.ResponseWriter, req *http.Request) { _, span := trace.StartSpan(req.Context(), "in b") defer span.End() fmt.Fprintf(w, "I am /b") }) log.Fatal(http.ListenAndServe(":50030", &ochttp.Handler{ Propagation: &tracecontext.HTTPFormat{}, }))
30. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷
31. OpenCensus + Stackdriver ʹΑΔࢀߟ࣮૷
32. Conclusion αʔϏεͷ৴པੑͱύϑΥʔϚϯεΛҡ࣋͢ΔͨΊʹ Applica'on Performance Monitoring (APM) ͸ඞཁෆՄܽͰ͋Δ OpenCensus Λ࢖͏͜ͱͰόοΫΤϯυͷαʔϏεΛҙࣝͤͣʹ APM ʹඞཁͳϝτϦΫεͷऩू͕ՄೳʹͳΔ