Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレー...
Search
Shin Tanimoto
February 28, 2023
Programming
5
2.4k
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング
Java Developers Summit (2023-02-28)
Shin Tanimoto
February 28, 2023
Tweet
Share
More Decks by Shin Tanimoto
See All by Shin Tanimoto
Spring Bootで実装とインフラをこれでもかと分離するための試み
shintanimoto
7
1.4k
26 Java Years
shintanimoto
0
92
Let’s Have Fun with Reactive Programming, Using Reactor and WebFlux
shintanimoto
0
300
Monitoring and Visualizing Your (Micro)services
shintanimoto
1
390
現代に求められるJavaコミュニティとは / What should be the Java Community of Today?
shintanimoto
0
960
人生がときめく「学び」の魔法 / The Life-Changing Magic of Studying
shintanimoto
6
1.7k
業務で使いたいWebFluxによるReactiveプログラミング / Introduction to Reactive Programming using Spring WebFlux
shintanimoto
9
9.6k
from old Java to modern Java (2017) #jjug
shintanimoto
4
1.7k
Spring Cloud Sleuth + Zipkin with Elasticsearch #zipkin_jp
shintanimoto
1
1.3k
Other Decks in Programming
See All in Programming
旅行プランAIエージェント開発の裏側
ippo012
2
890
Design Foundational Data Engineering Observability
sucitw
3
190
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
110
TDD 実践ミニトーク
contour_gara
1
290
Rancher と Terraform
fufuhu
2
240
Go言語での実装を通して学ぶLLMファインチューニングの仕組み / fukuokago22-llm-peft
monochromegane
0
120
ファインディ株式会社におけるMCP活用とサービス開発
starfish719
0
290
Android端末で実現するオンデバイスLLM 2025
masayukisuda
1
120
AIコーディングAgentとの向き合い方
eycjur
0
270
Improving my own Ruby thereafter
sisshiki1969
1
160
OSS開発者という働き方
andpad
5
1.7k
CloudflareのChat Agent Starter Kitで簡単!AIチャットボット構築
syumai
2
470
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.5k
Code Reviewing Like a Champion
maltzj
525
40k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.7k
Producing Creativity
orderedlist
PRO
347
40k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.1k
A designer walks into a library…
pauljervisheath
207
24k
Transcript
ΫϥυωΠςΟϒ࣌ͷ ίϯςφڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά Everforth / Acroquest Technology / JJUG Shin
Tanimoto (@cero_t)
ࣗݾհ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ
• Acroquest Technologyגࣜձࣾ • ຊJavaϢʔβʔάϧʔϓʢදʣ • Twitter : @cero_t
ࠓͷϝΠϯςʔϚ
ྩʹ͓͚ΔΦϒβόϦςΟ ͷΞʔΩςΫνϟʹ͍ͭͯ
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε • ϩά • τϨʔε
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε : ࣌ܥྻͷใ • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ •
τϨʔε : ݺͼग़͠ͷه
ϝτϦΫε • ࣌ܥྻͷใ • ྫ • OS : CPU༻ɺϝϞϦ༻ྔʢʣɺσΟεΫ༻ྔʢʣ •
JavaVM : ώʔϓ༻ྔʢʣɺεϨουɺίωΫγϣϯϓʔϧ • App : ϦΫΤετɺΤϥʔʢʣ
ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ͔ɺूܭͨ͠Λऔಘ͢Δ • CPU༻ : ʮඵؒʯͷ༻ͷฏۉԿ%͔ •
ϝϞϦ༻ྔ : ʮ͍·ʯԿMB͍ͬͯΔ͔ • GCճ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
ϝτϦΫε • ฏͷϞχλϦϯά • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕେͨ͠ͷ • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ
૬ͳΓ͜Έ • ྩͰͯ͢औΔ͠
ϩά • ࣌ࠁΛ͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ ϑϨʔϜϫʔΫϛυϧΣΞͷϩά •
ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετใʢมఆͳͲʣ
ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه͢Δ • αϯϓϦϯάϕʔεͱҟͳΓɺͯ͢ه͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺͯαΠζ͕େ͖͍ •
1ͭͷϦΫΤετʹΑͬͯۀଆͷσʔλετΞʹॻ͖ࠐ·ΕΔใྔΑΓ ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓϩάͷσʔλετΞͷํ͕ ߴੑೳɾେ༰ྔͳͷΛٻΊΒΕΔ͜ͱ͕͋Δ
ϩά • ฏͷϞχλϦϯά • ͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ • ඞཁͳ࣌ʹμϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Εύτϥϯϓ͕ճΓ͕ͪ •
ྩͷϞχλϦϯάͰৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ͠ • ͕ͦͯ͠ݟ͔ͭͬͨΒSlackͳͲʹ௨͢Δ͠
τϨʔε • ݺͼग़͠ͷه • ෳͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳͷ • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺޭ͔ࣦͨ͠ഊ͔ͨ͠ • ΠϕϯτϕʔεͰऔಘ •
ͯ͢औಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔ઃఆՄೳ
τϨʔε • ελοΫτϨʔε • Javaϓϩηε͚ͩͰ؆ܿ͢Δݺͼग़͠֊ • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ • ͦΕͳΒෳΞϓϦέʔγϣϯΛލ͍ͩτϨʔεऔΓ͍ͨΑͶ •
τϨʔεɺൃੜ࣌ͷ֬ೝɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ
τϨʔε • ฏͷϞχλϦϯά • τϨʔεऔΒͳ͍ or औΕͳ͍ • ྩͰऔΓ·͠ΐ͏
Spring Boot 3.0࣌ͷ ϞχλϦϯά
σϞ
σϞͰͬͨιʔείʔυ • https://github.com/cero-t/spring-store-2022 • ࣮ߦखॱઃఆͷৄࡉͳ༰ϒϩάʹهࡌ • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢
ΞϓϦέʔγϣϯͷߏ BFF Payment Service Order Service Stock
Service Delivery Service Item Service Cart Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0
(without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus :
ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
ϞχλϦϯάͷߏ Java Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε
ϩά τϨʔε Java Process Java Process Spring Boot Application
Spring Boot + PrometheusͰ ϝτϦΫεऩू
Spring BootͰϝτϦΫεऩू • Spring BootΞϓϦέʔγϣϯͷϝτϦΫεใΛWeb APIͱͯ͠ ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application ϝτϦΫε औಘAPI Prometheus ఆظతʹऔಘ Grafana औಘͯ͠ՄࢹԽ
Spring BootͰϝτϦΫεऩू • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏ • Spring BootΞϓϦέʔγϣϯͷdependencyʹ ActuatorͱMicrometerΛՃ͢Δ
• application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ • Actuator : spring-boot-starter-actuator • Spring
BootͷઃఆϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OSJavaVMɺSpring BootͷϝτϦΫεใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <scope>runtime</scope> </dependency> <dependency>
<groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency>
ActuatorͷΤϯυϙΠϯτΛެ։ • application.properties ʹͯɺMicrometerʹؔ͢Δ ActuatorͷΤϯυϙΠϯτΛެ։͢Δ management.endpoints.web.exposure.include=* management.metrics.distribution.percentiles-histogram.http.server.requests=true management.metrics.tags.application=${spring.application.name} #
Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ߹ʹ࣍ͷઃఆͰྑ͍ # management.endpoints.web.exposure.include=prometheus
ActuatorͱMicrometerΛՃ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi
les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
σϞ
PrometheusͰϝτϦΫεऩू • ʢGrafanaPrometheusࣗମͷߏஙʹׂ͍ͭͯѪʣ • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ ͖ͪΜͱޙͰղઆ͠·͢
σϞ
GrafanaʹμογϡϘʔυΛՃ • GrafanaίϛϡχςΟ͕࡞ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ • https://grafana.com/grafana/dashboards/ • JVM (Micrometer) - 4701
͕Φεεϝ
σϞ
Prometheuspullܕ • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔͷΛʮPushܕʯͱݺͿ • ΞϓϦέʔγϣϯʹऔΓʹߦ͘ͷΛʮPullܕʯͱݺͿ • PrometheusʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧΣΞͱ ͯ͠ΘΓͱ͍͠෦ྨʢ*ݸਓͷײͰ͢ʣ •
PullܕͰࢄΞϓϦέʔγϣϯͷϝτϦΫεΛͳ͘ ऩूͰ͖ΔʢޙͰղઆ͠·͢ʣ
Prometheuspullܕ • Pullܕͷར • ࢹରͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏Λݕग़͍͢͠ • PushܕͰʮૹΒΕͯ͜ͳ͔ͬͨʯ =
ʮه͠ͳ͍ʯ͚ͩͱ ͳΓɺͦΕΛͱͯ͠ݕग़͢Δ͜ͱ͕͍͠ • ࢹର͕ଟ͘ͳͬͨࡍʹɺPushܕͰϝτϦΫεऩूαʔό͕ύϯΫͯ͠ ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूසΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔ʯ
Spring Boot + LokiͰ ϩάऩू
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠ Lokiʹૹ৴͢ΔʢPullܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring
Boot Application ϩά ϑΝΠϧ Loki ఆظతʹऩू Grafana औಘͯ͠ՄࢹԽ Promtail ૹ৴
Ͱ͏ɺϩάϑΝΠϧͱ͔ ͍ͨ͘ͳ͍Ͱ͢ΑͶ
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Loki4J Appender Loki ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Loki4JͷLogback AppenderΛՃ • pom.xml ͷdependencyʹloki-logback-appenderΛՃ • ϑΝΠϧίϯιʔϧͰͳ͘LokiʹϩάΛૹΔΑ͏ͳ LogbackͷAppender <dependency>
<groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> <version>1.3.2</version> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ࡞ • LogbackͷઃఆϑΝΠϧΛ࡞͠ɺAppenderʹLoki4jAppenderΛ ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url>
</http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺ࡞ͨ͠LogbackͷઃఆϑΝΠϧΛ ༗ޮʹ͢Δ • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ͏τϨʔεID ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠
logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.con fi g=classpath:logback-loki.xml
LokiͰϩάऩू • ʢLokiࣗମͷߏஙʹׂ͍ͭͯѪʣ • LokiΛىಈ͢ΕΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ LokiଆʹಛʹඞཁͳઃఆͳͲͳ͠
σϞ
ϩάpush͔pull͔ • ྺ࢙తʹϩάPullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • FluentdLogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷෆ҆ •
ϩάʮγεςϜͷෆਖ਼ऴྃલͷ࠷ظͷʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰϩά͚͍ͩͨ͠ • PushܕͰड͚औΔଆͷϛυϧΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
ͦΜͳʹॏཁͳใΛ ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ ͦ͏ͱ͢Δͷ͔ʁ
ϩάͷ֓೦Λݟ͖͢ • ఏݴ : ϩάͱ͍͏֓೦ͦͷͷΛݟ͖͢ • ϩάॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ͚ܾͩΊͯޙ։ൃऀͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚ΕౖΒΕΔʣ •
ॏཁͳใͱɺͦ͏Ͱͳ͍ใ͕ࡶଟʹࠞࡏ͢Δ • ຊʹॏཁͳͷɺϩάͰͳ͘ۀσʔλͱͯ͠ଊ͑Δ͖ • ຊʹॏཁͳใԿ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ ৴པੑͷߴ͍σʔλετΞʹه͢Δ • ϩάͱ͍͏ΈʹཔΔͷʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ͖
ϩάpush͔pull͔ • ͦͷ্ͰɺϩάPushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafkaRabbitMQΛڬΜͰෛՙΛࢄͰ͖Δ •
Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢ΔࡍʹɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ͕ඞཁͳ͍ʢ͘͠JSONܗࣜͰग़ྗ͢Δʣ߹ ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅Ͱͳ͍߹ͳͲͰ༗ޮ
Spring Boot + TempoͰ τϨʔεऩू
Spring BootͰτϨʔεऩू • τϨʔεऩूͷͨΊʹ࣍ͷ2͕ͭඞཁ • ᶃτϨʔεIDͷ • ʮͲͷϦΫΤετʹඥ͘ॲཧ͔ʯʮ୭͔ΒݺΕ͔ͨʯͱ͍͏ ใΛΞϓϦέʔγϣϯؒͰͤ͞Δ •
ᶄτϨʔεใͷੵ • τϨʔεใΛσʔλετΞʹૹͬͯੵ͢Δ
Spring BootͰτϨʔεIDͷ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTPAMQPʣͷ ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot
Application1 Spring Boot Application2 Spring Boot Application4 Spring Boot Application3 traceparent = req001-01 traceparent = req001-02 traceparent = req001-01
Spring BootͰτϨʔεIDͷ • τϨʔεIDͷૹड৴ʹSpring Boot 2.x·ͰSpring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot
3.0͔ΒMicrometer Tracing͕ ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot Application Micrometer Tracing RestTemplate Spring Boot Application Micrometer Tracing Rest Controller HTTPϔομʹ τϨʔεIDͷ༩ ʢࣗಈʣ HTTPϔομ͔Β τϨʔεIDͷऔಘ ʢࣗಈʣ
Micrometer TracingͷՃ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ -brave ΛՃ͢ΔʢͲͪΒͰৼΔ͍ʹҧ͍ͳ͍ʣ
• Spring BootͷRestTemplateSpring Cloud StreamΛ༻͍ͨ௨৴Ͱ τϨʔεIDͷ͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
Spring BootͰτϨʔεऩू • Spring BootΞϓϦέʔγϣϯͷτϨʔεใͷهΛ Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ • Grafana͕Tempo͔ΒτϨʔεใΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Open Telemetry Tempo ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Open TelemetryͷՃ • Spring Boot͔ΒτϨʔεใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔
zipkin-reporter-brave ΛՃ͢ΔʢৼΔ͍ʹҧ͍ͳ͍ʣ • ZipkinʹτϨʔεใΛૹΔϥΠϒϥϦ ʢTempo͕ZipkinޓAPIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺτϨʔεใΛͲΕ͘Β͍ͷׂ߹Ͱ TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ10%ʣ • ։ൃதσϞͰ100%ʹͨ͠ํ͕ྑ͍ • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ
100%ʹઃఆ͓ͯ͘͠ͱྑ͍ management.tracing.sampling.probability=1.0
TempoͰτϨʔεऩू • ʢTempoࣗମͷߏஙʹׂ͍ͭͯѪʣ • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓAPIΛ༗ޮʹ͢Δ server: http_listen_port: 3200 distributor: receivers:
zipkin:
σϞ
τϨʔεpushͷΈ • ZipkinʢޓAPIؚΉʣτϨʔεใΛड͚औΔͨΊͷAPIΛ ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔεใΛૹΔ ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔Εͳ͍
Spring Boot 3.0 + k8sͷ ϞχλϦϯά
k8s্ͰPullܕΛӡ༻͢Δ՝ • PrometheusͰϞχλϦϯάରͷΞυϨεΛྻڍͨ͠ • k8sͷΑ͏ʹಈతʹϞχλϦϯάର͕૿ݮɾมԽ͢Δ߹ʹ ͲͷΑ͏ʹऩू͢Δͷ͔ʁ
αʔϏεσΟεΧόϦ • ΞϓϦέʔγϣϯͷϧʔςΟϯάϩʔυόϥϯγϯάͷͨΊʹ ༻͍ΒΕΔαʔϏεσΟεΧόϦͷΈΛϞχλϦϯάͰར༻ ͢Δ • k8sͰCoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏
αʔϏεσΟεΧόϦ • 20ඵͰ͔ΔαʔϏεσΟεΧόϦ Spring Boot Application1 Service Registry
Spring Boot Application1 Spring Boot Application1 Spring Boot Application1 ᶃΞυϨεͷొ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot Application2
αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot Application1 Prometheus Promtail Service
Registry Spring Boot Application1 Spring Boot Application1 Spring Boot Application ᶃΞυϨεͷొ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ৭ʑͱઃఆ͕ඞཁ͔ͩΒ PrometheusPromtail helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
PrometheusͷαʔϏεσΟεΧόϦઃఆ • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ annotationsͰϙʔτ൪߸ActuatorͷύεΛࢦఆ͢Δ spec: template: metadata: labels: app:
bff annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: /actuator/prometheus
σϞ
ίϯςφ࣌ͰPullͰେৎ • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷΈͱซ༻͢Ε PullܕͰಛʹͳ͘ϞχλϦϯάڥΛߏஙͰ͖Δ
·ͱΊ
·ͱΊ • ϝτϦΫεɺϩάɺτϨʔεऔΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot
3.0ͰϞχλϦϯάपΓͷϥΠϒϥϦͳͲཧ͞Ε GrafanaελοΫͳͲͰϞχλϦϯά͘͢͠ͳ͍ͬͯΔ • ϩά͔ΒτϨʔεͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
·ͱΊ • PushܕɺPullܕͷ͍ͣΕͰͳ͘σʔλΛऩूͰ͖Δ • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ • DataDogAWS CloudWatchͳͲجຊతʹPushܕ • ΞϓϦέʔγϣϯଆͷιʔείʔυઃఆϑΝΠϧ͕
Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͍͢͠Α
Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot
3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442