ペパボOpenTelemetry革命

社内のイベントでお話しました

Transcript

1. ϖύϘOpenTelemetryֵ໋ ʙ͜ͷલMOROHAΈͯେ߸ٽ͠·ͨ͠ʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   
   
   
   
   
   1SPYZαʔό։ൃɺ0QFO5FMFNFUSZ
3. 45/4
   -JOVY
   /44
   4FSWFS
   TUOTKQ
5. ࠓ೔࿩͢͜ͱ OpenTelemetry֓ཁ OpenTelemetryΛಋೖ͖͔͚ͨͬ͠ OpenTelemetryಋೖঢ়گ ࠓޙͷల๬
6. 🍉🌴🏄 GoConference’19 🏄🌴🍉 6 0QFO5FMFNFUSZ
   ֓ཁ
7. OpenTelemetry چདྷɺ4BB4΍044͝ͱʹ࣮૷͞Ε͍ͯͨςϨϝτϦσʔλͷඪ४Խ IUUQTDMPVEOBUJWFEBZTKQPZ https://opentelemetry.connpass.com/
8. ςϨϝτϦσʔλ “ςϨϝτϦσʔλ͸ɺΦϒβʔόϏϦςΟΛ࣮ݱ͢ΔͨΊʹγεςϜ ιʔε͔Βऩू͞ΕΔग़ྗΛࢦ͠·͢ɻ͜ΕΒͷग़ྗΛ౷߹తʹ෼ੳ͢ Δ͜ͱʹΑΓɺ෼ࢄ؀ڥͰͷγεςϜؒͷؔ܎΍ґଘؔ܎͕Θ͔Γ· ͢ɻओʹ࢖༻͞ΕΔσʔλΫϥε͸ ϩάɺϝτϦΫεɺτϨʔεͷ3ͭͰɺ͜Ε͸ ʮΦϒβʔόϏϦςΟͷ3ͭͷபʯͱ΋ݺ͹Ε·͢ɻ” https://www.splunk.com/ja_jp/data-insider/what-is-opentelemetry.html ΑΓҾ༻
9. ΦϒβʔόϏϦςΟ ΦϒβʔόϏϦςΟɾΤϯδχΞϦϯά 6PΑΓҾ༻ “؆୯ʹݴ͏ͱɺࢲ͕ͨͪߟ͑Διϑτ΢ΣΞγεςϜͷ ʮΦϒβʔόϏϦςΟʯͱ͸ɺγεςϜ͕ͲͷΑ͏ͳঢ়ଶʹͳͬͨͱ͠ ͯ΋ɺͦΕ͕ͲΜͳʹ࢐৽Ͱحົͳ΋ͷͰ͋ͬͯ΋ɺͲΕ͚ͩཧղ͠આ໌ Ͱ͖Δ͔Λࣔ͢ई౓Ͱ͢ɻ·ͨɺͦͷΑ͏ͳ࢐৽Ͱحົͳঢ়ଶʹରͯ͠ ΋ɺࣄલʹσόοάͷඞཁੑΛఆٛͨ͠Γ༧ଌͨ͠Γ͢Δ͜ͱͳ͘ɺ γεςϜͷঢ়ଶσʔλͷ͋ΒΏΔσΟϝϯγϣϯ΍ͦΕΒͷ૊Έ߹Θͤʹ ͍ͭͯΞυϗοΫʹௐࠪ͠ɺΑΓσόοά͕ՄೳͰ͋ΔΑ͏ʹ͢Δඞཁ͕ ͋Γ·͢ɻ΋͠ɺ৽͍͠ίʔυΛσϓϩΠ͢Δඞཁ͕ͳ͘ɺͲΜͳ࢐৽Ͱ حົͳঢ়ଶͰ΋ཧղͰ͖ΔͳΒɺΦϒβʔόϏϦςΟ͕͋Δͱݴ͑·͢ɻ
10. https://opentelemetry.io/blog/2024/pro fi ling/
11. OpenTelemetry https://opentelemetry.io/docs/ ֤ݴޠͷSDK΍ ࣗಈܭ૷ɺखಈܭ૷ ΍ϛυϧ΢ΣΞ͔Β ςϨϝτϦσʔλΛ ग़ྗ
12. OpenTelemetry https://opentelemetry.io/docs/ ඪ४Խ͞Εͨϓϩτίϧʹ ରԠͨ͠ΤʔδΣϯτ
13. OpenTelemetry https://opentelemetry.io/docs/collector/ GoͰ࣮૷͞ΕͨόΠφϦ͕ɺReceivers,Processors,ExportersΛ࣮ߦ
14. OpenTelemetry https://opentelemetry.io/docs/ 4BB4΍
    ϗετ͞Εͨ044
    ࠩ͠ସ͑Մೳ
15. OpenTelemetryΛ ಋೖ͖͔͚ͨͬ͠
17. ͓Θ͔ΓͩΖ͏͔ʁ https:// fi nance.yahoo.co.jp/quote/USDJPY=FX?styl=cndl&frm=mnthly&trm=mnthly&ovrIndctr=sma%2Cmma%2Clma
18. ϖύϘͷࣄ৘ ෼ࢄτϨʔγϯά͕ະಋೖ ؂ࢹαʔϏε͕MackerelͱPrometheusͰڞଘ
19. MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾ಺αʔϏεͷVM Kubernetes Node Monitor Servers OpenTelemetryਪਐલ ֎ܗ؂ࢹ ϓϥΠϕʔτΫϥ΢υͷHW
20. MackerelͱPrometheus Mackerel Prometheus ຊ൪αʔόͷVM Managed Service Kubernetes Control Plane ࣾ಺αʔϏεͷVM Kubernetes Node Monitor Servers Otel CollecotrΛ͍ΕΔͳΒɺ͜͏ͨ͠΄͏͕ແବ͕ͳ͍ ֎ܗ؂ࢹ ϓϥΠϕʔτΫϥ΢υͷHW ຊ൪αʔόͷVM
21. OpenTelemetryಋೖঢ়گ
22. ίϯϙʔωϯτ VM k8s Metrics ̋ ̋ Trace ̋ ̋ Log ˚ ˚ OpenTelemetryಋೖঢ়گ ̋
    ϓϩμΫγϣϯͰಋೖࡁΈ
    ˚લճ.FFUVQͰ࿩ͨ͠ͱ͖ΑΓਐḿ
23. Metrics
24. Metrics ϗετͷϦιʔε͸hostmetricsͰऔಘ ɹVM͸ͦΕͧΕotel collectorΛΠϯετʔϧ ɹk8s͸DaemonSetͰ/Λmount receivers: hostmetrics: collection_interval: 1m scrapers: cpu: metrics: system.cpu.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: ...
25. Metrics %BTICPBSE͸(JU)VCͰϝτϦΫε໊Ͱൃ۷ͨ͠ https://ten-snapon.com/archives/2983
26. Metrics 4%,Λར༻ͯ͠ϝτϦΫεΛܭ૷͍ͯ͠Δ https://ten-snapon.com/archives/2983 if req.Method == http.MethodPost && strings.Contains(req.RequestURI, “cartin")){ if err == nil && res.StatusCode >= 200 && res.StatusCode < 300 { r.successCounter.Add(context.Background(), metric.WithAttributes(semconv.HostName(requestURL.Host))) } else { r.failCounter.Add(context.Background(), 1, metric.WithAttributes(semconv.HostName(requestURL.Host))) } }
27. Trace
28. Trace k8sͷμογϡϘʔυͱͯ͠GrafanaΛಋೖࡁΈ k8sͷϩάݕࡧʹGrafana Loki + S3ΛಋೖࡁΈ (SBGBOB
    5FNQPΛ࠾༻
29. Trace otelcol k8s DataCenter Tempo pod VM otelcol app VMͷTrace΋Otel CollectorΛ ܦ༝ͯ͠ɺk8sͷTempo΁
30. ΄ͱ͹͠ΔϚΠΫϩαʔϏεײ
31. ΄ͱ͹͠ΔϚΠΫϩαʔϏεײ
32. Log
33. طଘͷϩάύΠϓϥΠϯ ϩάͷϥΠϑλΠϜʹ߹ΘͤͨετΞʹ֨ೲ
34. Apache Kafka LBGLB
    DMVTUFS /var/log/audit.log LBGLBCSPLFS BVEJUMPH
    UPQJD LBGLBCSPLFS TZTMPH
    UPQJD ϩά͝ͱʹτϐοΫΛ෼͚͍ͯΔ /var/log/syslog
35. Kafka Exporter 5PQJD͕ݱঢ়͸ܾΊଧͪͬΆ͍ https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/kafkaexporter
36. Kafka Exporter ͜͏͍͏ײ͡Ͱॻ͚͍ͨ
37. ͳΜͱॻ͚·͢!!1
38. ࠓޙͷల๬ Self Hosted APM ܭଌ͔ΒվળͷαΠΫϧΛճ͢ OpenTelemetry΁ͷߩݙ
39. Self Hosted APM APMͷՄ༻ੑ͸ߴ͘ͳ͍͍ͯ͘ͷͰɺ ӡ༻Ͱ͖ΔͳΒSelf HostedͰ͍͍ ݱࡏ͸SentryΛධՁத
40. Elastic APM͸΍Ίͨ Ҏલ͸DockerͰىಈͯͦ͜͠ʹϝτϦΫεΛૹ Δ͚͕ͩͩͬͨɺAgentͷΠϯετʔϧ͕ඞཁʹ ͳͬͨΓɺҠߦɺESɺkibanaͷϝϯςίετ͕ ௿͍ͱ͸ݴ͑ͳ͍
41. ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ܭଌͰ͖ΔΑ͏ʹͳ͕ͬͨɺܭଌͰ͖͍ͯΔ͔͸ผ໰୊ ܭଌͰ͖ͨͱͯ͠ɺվળͰ͖Δ͔͸ผ໰୊ πʔϧͷڭҭɺϋϯζΦϯ ϘτϧωοΫͷಛఆɺղফ
42. ܭଌ͔ΒվળͷαΠΫϧΛճ͢
43. 永遠のN+1 net.ipv4.tcp_max_syn_backlog Kubernetes Scheduler 突然のYahooトピックス 特定のバージョンだけで取られる謎のロック Slow Query
44. ܭଌ͔ΒվળͷαΠΫϧΛճ͢ վળʹ͸͍Ζ͍ΖͳεΩϧɺ஌͕ࣝඞཁ σʔλϕʔε: ΫΤϦύϑΥʔϚϯεɺύϥϝʔλʔνϡʔχϯά OS/Kernel: ωοτϫʔΫɺύϥϝʔλʔɺCPUɺϝϞϦʔ ιϑτ΢ΣΞ։ൃ: ܭࢉྔɺϦόʔεΤϯδχΞϦϯά ৘ใऩूೳྗ: GitHub IssueɺThe Linux Kernel ArchivesɺX
45. ܭଌ͔ΒվળͷαΠΫϧΛճ͢ ͠ΐ͏΋ͳ͍ɺ࢖ΘΕͳ͍ɺҙຯ͕ͳ͍ ͦΜͳ΋ͷͰ΋͍͍ɺͱʹ͔͘ॻ͘͜ͱͰ ॻ͚ΔΑ͏ʹͳΓɺؾ͕͍ͭͨΒເͷதͰ ઃܭ͢ΔΑ͏ʹͳΔ
46. OpenTelemetry΁ͷߩݙ OpenTelemetry(collector)ͷ࣮૷ൣғ͸ͱͯ΋޿͍ receiver,exporter,processor͝ͱʹϝϯςφ͕͍ͯɺ·ΔͰ૊৫ͷΑ͏ ࣮૷΋ࢀߟʹͳΔ͜ͱ͕ଟ͘ɺֶͼͷػձʹͳΔ
47. ͳΜͱɺ ೖࣾͨ͠Βશ෦ ΍Ε·͢ʂʂ1
48. ܅΋PUFMʹͳΒͳ͍͔ʁ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU