How to send distibuted traces to Datadog using build own OpenTelemetry-Lambda distribution

https://opentelemetry.connpass.com/event/317170/

https://scrapbox.io/aereal-tech/2024%2F6%2F25_OpenTelemetry_Meetup_2024-06

Transcript

1. 2024-06-24 @ OpenTelemetry Meetup 2024-06 OpenTelemetryͱAWS Lambda ͱDatadog id:aereal
2. ࿩͢͜ͱ • AWS Lambda͔ΒOpenTelemetryΛ஻ͬͯDatadogʹτϨʔεΛૹΔํ๏ • AWS Lambda Extensionͷ֓ཁ • opentelemetry-lambdaͷ֓ཁ • AWS Distribution for OpenTelemetry Lambdaͱopentelemetry-lambdaͷؔ܎ • DatadogʹτϨʔεΛૹΔͨΊʹඞཁͳΧελϜϏϧυͷ஌ࣝ
3. • id:aereal • GitHub: aereal • @aereal@mstdn.jp • גࣜձࣾClassi ֶशPMF෦ ΞʔΩςΫτ ࣗݾ঺հ
4. ঎༻؀ڥͰOpenTelemetry Collector Λಈ͔͍ͯ͠Δਓʑʙ✋
5. ΧελϜϏϧυͨ͠ 
 OpenTelemetry CollectorΛ࢖͍ͬͯΔਓʑʙ✋
6. AWS Lambda࢖͍ͬͯΔਓʑʙ✋
7. FaaS࢖͍ͬͯΔਓʑʙ✋
8. ΍Γ͍ͨ͜ͱ • AWS Lambda͔ΒOpenTelemetryΛ஻ͬͯDatadogʹτϨʔεΛૹΓ͍ͨ • Lambdaͷݺͼग़͠΋౰વ؍ଌ͍ͨ͠ • OpenTelemetryͷΤίγεςϜʹऒ͔ΕͯҠߦͨ͠ͷͰ
 OTelҎ֎ͷϥΠϒϥϦ͸࢖͍ͨ͘ͳ͍ • refs. https://tech.classi.jp/entry/opentelemetry-migration • DatadogΛ׆༻͍ͯ͠ΔͷͰAWS X-RayͦͷଞʹσʔλΛ෼ࢄͤͨ͘͞ͳ͍
9. DatadogͱClassi
10. OTelͱLambda: opentelemetry-lambda • OTel collectorΛLambda Extensionͱͯ͠ಈ͔࣮͢૷ • Lambda Extension APIΛ஻ͬͯcollectorͷϓϩηεΛಈ͔͢ • Layerͱͯ͠഑෍͞Ε͍ͯΔ • refs. https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html
11. AWS Lambda Extension • 2020೥ʹϦϦʔε͞Εͨؔ਺ຊମʹखΛೖΕͣؔ਺Λ֦ு͢ΔΤίγεςϜ • ॴఆͷҐஔʹόΠφϦΛஔ͍͓ͯ͘ͱInit, Restore, Invoke, ShutdownͦΕͧΕ ͷλΠϛϯάͰݺͼग़͞ΕΔ • InitΠϕϯτͰผϓϩηεΛ্ཱͪ͛ͨΓͳͲͰ͖Δ • ֦ுͷొ࿥ʹ͸࣮ߦ࣌ؒͷ੍໿͕͋Γ͜ΕΛ௒͑Δͱొ࿥ʹࣦഊ͠ɺͦͷ֦ு ͸࣮ߦ͞Εͳ͍ • ؔ਺ຊମͷ࣮ߦʹ͸Өڹ͠ͳ͍ɺ͋͘·Ͱແࢹ͞ΕΔ͚ͩ
12. https://docs.aws.amazon.com/images/lambda/latest/dg/images/Overview-Invoke-with-Error.png
13. https://docs.aws.amazon.com/images/lambda/latest/dg/images/telemetry-api-concept-diagram.png
14. OTelͱLambda (2) • AWS Distro for OpenTelemetry Lambda (ADOT Lambda) • AWS͕ϝϯς͍ͯ͠Δopentelemetry-lambdaͷσΟετϦϏϡʔγϣϯ • AWS͕બΜͩίϯϙʔωϯτ͔ΒͳΔcollectorΛಈ͔͢ • ͜Εʹdatadogexporter͸ؚ·Ε͍ͯͳ͍ˠࣗલͰϏϧυ͢Δ΄͔ͳ͍
15. https://github.com/aws-observability/aws-otel-lambda
16. OTel LambdaͷΧελϜϏϧυ • OTel collectorͷΧελϜϏϧυͱߟ͑ํ͸ಉ͡ • ࢖͍͍ͨίϯϙʔωϯτΛimportͯ͠Factoryʹ౉ͯ͋͛͠Δ • ocb͸͋͘·ͰcollectorΛϏϧυ͢ΔπʔϧͳͷͰ࢖͑ͳ͍͜ͱʹ஫ҙ • https://opentelemetry.io/docs/collector/custom-collector/
17. OTel LambdaͷΧελϜϏϧυ 1. opentelemetry-lambdaΛforkͯ͠ 2. collector/lambdacomponents/default.goʹ࢖͍͍ͨίϯϙʔωϯτΛ஫ೖ͠ 3. make͢Δ https://github.com/open-telemetry/opentelemetry-lambda/tree/main/collector
18. https://github.com/open-telemetry/opentelemetry-lambda/blob/c55ad8742d9d4714a950fe4eaf11ed3246673577/collector/ lambdacomponents/default.go
19. ࣮ࡍͷΧελϚΠζ ͓඼ॻ͖ • datadogexporterͷ௥Ճ • AWS Parameters Store confmap providerͷ௥Ճ • otel-lambda΁҆શʹAPIΩʔΛ౉͢ํ๏Λఏڙ͢Δ
20. https://github.com/open-telemetry/opentelemetry-lambda/compare/main...aereal:opentelemetry-lambda:support- datadog-exporter#di ff -f619591f6445be6380476b6844bbbe99a603301d4ddd5a9c9bb0ec2e59662894
21. Lambdaͱൿಗ৘ใ • Lambdaؔ਺ʹσϓϩΠෆཁͰ஋Λ౉͢ํ๏͸؀ڥม਺ͱAmazon EFSͷΈ • ؀ڥม਺͸ؔ਺ͷઃఆʹฏจͰهड़͢Δ • Amazon EFS͸ϚωʔδυNFSαʔϏε • → ͲͪΒ΋ൿಗ৘ใͷड͚౉͠ํ๏ͱͯ͠ෆద౰ • Parameters StoreͱSecrets Managerͱ͍͏αʔϏεʹอଘ͠ɺ
 otel-lambda͔ΒAPIΛ࢖ͬͯ͜ΕΒ͔Β஋Λऔಘ͢Ε͹Αͦ͞͏
22. confmapͱ͸ • OTel collectorͷઃఆʹ֎෦σʔλιʔε͔Βऔಘͨ͠஋ΛຒΊࠐΉ࢓૊Έ • ${env:DD_API_KEY} ͱॻ͘ͱ DD_API_KEY ͱ͍͏؀ڥม਺ͷ஋Ͱஔ͖׵͑Β ΕΔ • envͷ෦෼͕confmap providerͷ໊લΛࢦ͢ • ଞʹs3ͳͲ͕͋Δ
23. https://github.com/open-telemetry/opentelemetry-lambda/compare/main...aereal:opentelemetry-lambda:support- datadog-exporter#di ff -f301b7c709892efae0bdfdecfca8fea1aa2114d8c621b5e9993cd61390e07c43
24. https://this.aereal.org/entry/the-art-of-otel-col-confmap-provider
25. Ͱ͖͕͋ͬͨϏϧυͷ഑෍ͱར༻ • GitHub Packages Container RegistryΛ࢖ͬͯ഑෍ • Lambdaؔ਺͸ίϯςφΠϝʔδΛσϓϩΠ͢Δ͜ͱ͕Մೳ • ϕʔεΠϝʔδʹࢦఆͯ͋͛͠Δ • ద੾ʹݖݶΛઃఆ͢Ε͹GitHub Actions͔ΒγʔϜϨεʹpullͰ͖ͯศར
26. ࡶײ • ແࣄɺ໨త͸Ռͨ͞Εͨ͠OTelΤίγεςϜͷཧղ͕ਂ·ͬͨ • ΧελϜϏϧυͦͷ΋ͷ͸ඞཁͳෳࡶͩ͞ͱײ͡Δ • αϓϥΠνΣʔϯ߈ܸͷϦεΫ͕ೝ஌͞Ε͍ͯΔࡢࠓɺґଘπϦʔΛ୯७ʹอͪ೺ Ѳ͠ଓ͚Δҙٛ͸େ͖͍ • ܧଓతͳ্ྲྀͷมߋऔΓࠐΈ͕͍ͨ΁Μͳͷ͸ͳΜͱ͔͍ͨ͠ • ͳͥͳΒਓʑ͸େมͳ͜ͱ͸ແࢹͨ͘͠ͳΔ͔Β…… • ͱʹ্͔͘ྲྀΛrebaseͯ͠ςετ͕௨Ε͹Α͍ͷͰɺ͋ͱ͸΍Δ͚ͩ……
27. Fin.