Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレー...
Search
Shin Tanimoto
February 28, 2023
Programming
5
2.2k
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング
Java Developers Summit (2023-02-28)
Shin Tanimoto
February 28, 2023
Tweet
Share
More Decks by Shin Tanimoto
See All by Shin Tanimoto
26 Java Years
shintanimoto
0
49
Let’s Have Fun with Reactive Programming, Using Reactor and WebFlux
shintanimoto
0
230
Monitoring and Visualizing Your (Micro)services
shintanimoto
0
340
現代に求められるJavaコミュニティとは / What should be the Java Community of Today?
shintanimoto
0
880
人生がときめく「学び」の魔法 / The Life-Changing Magic of Studying
shintanimoto
6
1.6k
業務で使いたいWebFluxによるReactiveプログラミング / Introduction to Reactive Programming using Spring WebFlux
shintanimoto
9
9.1k
from old Java to modern Java (2017) #jjug
shintanimoto
4
1.6k
Spring Cloud Sleuth + Zipkin with Elasticsearch #zipkin_jp
shintanimoto
1
1.2k
Stream API 入門 #jjug #javajo
shintanimoto
4
3.9k
Other Decks in Programming
See All in Programming
선언형 UI에서의 상태관리
l2hyunwoo
0
160
今年のアップデートで振り返るCDKセキュリティのシフトレフト/2024-cdk-security-shift-left
tomoki10
0
200
テスト自動化失敗から再挑戦しチームにオーナーシップを委譲した話/STAC2024 macho
ma_cho29
1
1.3k
競技プログラミングへのお誘い@阪大BOOSTセミナー
kotamanegi
0
360
バグを見つけた?それAppleに直してもらおう!
uetyo
0
180
nekko cloudにおけるProxmox VE利用事例
irumaru
3
430
useSyncExternalStoreを使いまくる
ssssota
6
1k
Go の GC の不得意な部分を克服したい
taiyow
3
780
Effective Signals in Angular 19+: Rules and Helpers
manfredsteyer
PRO
0
100
採用事例の少ないSvelteを選んだ理由と それを正解にするためにやっていること
oekazuma
2
1k
PHPで学ぶプログラミングの教訓 / Lessons in Programming Learned through PHP
nrslib
2
230
Fibonacci Function Gallery - Part 1
philipschwarz
PRO
0
220
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Six Lessons from altMBA
skipperchong
27
3.5k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
Designing for Performance
lara
604
68k
BBQ
matthewcrist
85
9.4k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
Side Projects
sachag
452
42k
Practical Orchestrator
shlominoach
186
10k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Writing Fast Ruby
sferik
628
61k
Building an army of robots
kneath
302
44k
How STYLIGHT went responsive
nonsquared
95
5.2k
Transcript
ΫϥυωΠςΟϒ࣌ͷ ίϯςφڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά Everforth / Acroquest Technology / JJUG Shin
Tanimoto (@cero_t)
ࣗݾհ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ
• Acroquest Technologyגࣜձࣾ • ຊJavaϢʔβʔάϧʔϓʢදʣ • Twitter : @cero_t
ࠓͷϝΠϯςʔϚ
ྩʹ͓͚ΔΦϒβόϦςΟ ͷΞʔΩςΫνϟʹ͍ͭͯ
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε • ϩά • τϨʔε
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε : ࣌ܥྻͷใ • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ •
τϨʔε : ݺͼग़͠ͷه
ϝτϦΫε • ࣌ܥྻͷใ • ྫ • OS : CPU༻ɺϝϞϦ༻ྔʢʣɺσΟεΫ༻ྔʢʣ •
JavaVM : ώʔϓ༻ྔʢʣɺεϨουɺίωΫγϣϯϓʔϧ • App : ϦΫΤετɺΤϥʔʢʣ
ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ͔ɺूܭͨ͠Λऔಘ͢Δ • CPU༻ : ʮඵؒʯͷ༻ͷฏۉԿ%͔ •
ϝϞϦ༻ྔ : ʮ͍·ʯԿMB͍ͬͯΔ͔ • GCճ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
ϝτϦΫε • ฏͷϞχλϦϯά • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕେͨ͠ͷ • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ
૬ͳΓ͜Έ • ྩͰͯ͢औΔ͠
ϩά • ࣌ࠁΛ͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ ϑϨʔϜϫʔΫϛυϧΣΞͷϩά •
ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετใʢมఆͳͲʣ
ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه͢Δ • αϯϓϦϯάϕʔεͱҟͳΓɺͯ͢ه͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺͯαΠζ͕େ͖͍ •
1ͭͷϦΫΤετʹΑͬͯۀଆͷσʔλετΞʹॻ͖ࠐ·ΕΔใྔΑΓ ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓϩάͷσʔλετΞͷํ͕ ߴੑೳɾେ༰ྔͳͷΛٻΊΒΕΔ͜ͱ͕͋Δ
ϩά • ฏͷϞχλϦϯά • ͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ • ඞཁͳ࣌ʹμϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Εύτϥϯϓ͕ճΓ͕ͪ •
ྩͷϞχλϦϯάͰৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ͠ • ͕ͦͯ͠ݟ͔ͭͬͨΒSlackͳͲʹ௨͢Δ͠
τϨʔε • ݺͼग़͠ͷه • ෳͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳͷ • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺޭ͔ࣦͨ͠ഊ͔ͨ͠ • ΠϕϯτϕʔεͰऔಘ •
ͯ͢औಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔ઃఆՄೳ
τϨʔε • ελοΫτϨʔε • Javaϓϩηε͚ͩͰ؆ܿ͢Δݺͼग़͠֊ • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ • ͦΕͳΒෳΞϓϦέʔγϣϯΛލ͍ͩτϨʔεऔΓ͍ͨΑͶ •
τϨʔεɺൃੜ࣌ͷ֬ೝɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ
τϨʔε • ฏͷϞχλϦϯά • τϨʔεऔΒͳ͍ or औΕͳ͍ • ྩͰऔΓ·͠ΐ͏
Spring Boot 3.0࣌ͷ ϞχλϦϯά
σϞ
σϞͰͬͨιʔείʔυ • https://github.com/cero-t/spring-store-2022 • ࣮ߦखॱઃఆͷৄࡉͳ༰ϒϩάʹهࡌ • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢
ΞϓϦέʔγϣϯͷߏ BFF Payment Service Order Service Stock
Service Delivery Service Item Service Cart Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0
(without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus :
ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
ϞχλϦϯάͷߏ Java Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε
ϩά τϨʔε Java Process Java Process Spring Boot Application
Spring Boot + PrometheusͰ ϝτϦΫεऩू
Spring BootͰϝτϦΫεऩू • Spring BootΞϓϦέʔγϣϯͷϝτϦΫεใΛWeb APIͱͯ͠ ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application ϝτϦΫε औಘAPI Prometheus ఆظతʹऔಘ Grafana औಘͯ͠ՄࢹԽ
Spring BootͰϝτϦΫεऩू • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏ • Spring BootΞϓϦέʔγϣϯͷdependencyʹ ActuatorͱMicrometerΛՃ͢Δ
• application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ • Actuator : spring-boot-starter-actuator • Spring
BootͷઃఆϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OSJavaVMɺSpring BootͷϝτϦΫεใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <scope>runtime</scope> </dependency> <dependency>
<groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency>
ActuatorͷΤϯυϙΠϯτΛެ։ • application.properties ʹͯɺMicrometerʹؔ͢Δ ActuatorͷΤϯυϙΠϯτΛެ։͢Δ management.endpoints.web.exposure.include=* management.metrics.distribution.percentiles-histogram.http.server.requests=true management.metrics.tags.application=${spring.application.name} #
Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ߹ʹ࣍ͷઃఆͰྑ͍ # management.endpoints.web.exposure.include=prometheus
ActuatorͱMicrometerΛՃ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi
les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
σϞ
PrometheusͰϝτϦΫεऩू • ʢGrafanaPrometheusࣗମͷߏஙʹׂ͍ͭͯѪʣ • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ ͖ͪΜͱޙͰղઆ͠·͢
σϞ
GrafanaʹμογϡϘʔυΛՃ • GrafanaίϛϡχςΟ͕࡞ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ • https://grafana.com/grafana/dashboards/ • JVM (Micrometer) - 4701
͕Φεεϝ
σϞ
Prometheuspullܕ • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔͷΛʮPushܕʯͱݺͿ • ΞϓϦέʔγϣϯʹऔΓʹߦ͘ͷΛʮPullܕʯͱݺͿ • PrometheusʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧΣΞͱ ͯ͠ΘΓͱ͍͠෦ྨʢ*ݸਓͷײͰ͢ʣ •
PullܕͰࢄΞϓϦέʔγϣϯͷϝτϦΫεΛͳ͘ ऩूͰ͖ΔʢޙͰղઆ͠·͢ʣ
Prometheuspullܕ • Pullܕͷར • ࢹରͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏Λݕग़͍͢͠ • PushܕͰʮૹΒΕͯ͜ͳ͔ͬͨʯ =
ʮه͠ͳ͍ʯ͚ͩͱ ͳΓɺͦΕΛͱͯ͠ݕग़͢Δ͜ͱ͕͍͠ • ࢹର͕ଟ͘ͳͬͨࡍʹɺPushܕͰϝτϦΫεऩूαʔό͕ύϯΫͯ͠ ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूසΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔ʯ
Spring Boot + LokiͰ ϩάऩू
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠ Lokiʹૹ৴͢ΔʢPullܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring
Boot Application ϩά ϑΝΠϧ Loki ఆظతʹऩू Grafana औಘͯ͠ՄࢹԽ Promtail ૹ৴
Ͱ͏ɺϩάϑΝΠϧͱ͔ ͍ͨ͘ͳ͍Ͱ͢ΑͶ
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Loki4J Appender Loki ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Loki4JͷLogback AppenderΛՃ • pom.xml ͷdependencyʹloki-logback-appenderΛՃ • ϑΝΠϧίϯιʔϧͰͳ͘LokiʹϩάΛૹΔΑ͏ͳ LogbackͷAppender <dependency>
<groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> <version>1.3.2</version> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ࡞ • LogbackͷઃఆϑΝΠϧΛ࡞͠ɺAppenderʹLoki4jAppenderΛ ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url>
</http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺ࡞ͨ͠LogbackͷઃఆϑΝΠϧΛ ༗ޮʹ͢Δ • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ͏τϨʔεID ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠
logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.con fi g=classpath:logback-loki.xml
LokiͰϩάऩू • ʢLokiࣗମͷߏஙʹׂ͍ͭͯѪʣ • LokiΛىಈ͢ΕΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ LokiଆʹಛʹඞཁͳઃఆͳͲͳ͠
σϞ
ϩάpush͔pull͔ • ྺ࢙తʹϩάPullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • FluentdLogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷෆ҆ •
ϩάʮγεςϜͷෆਖ਼ऴྃલͷ࠷ظͷʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰϩά͚͍ͩͨ͠ • PushܕͰड͚औΔଆͷϛυϧΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
ͦΜͳʹॏཁͳใΛ ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ ͦ͏ͱ͢Δͷ͔ʁ
ϩάͷ֓೦Λݟ͖͢ • ఏݴ : ϩάͱ͍͏֓೦ͦͷͷΛݟ͖͢ • ϩάॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ͚ܾͩΊͯޙ։ൃऀͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚ΕౖΒΕΔʣ •
ॏཁͳใͱɺͦ͏Ͱͳ͍ใ͕ࡶଟʹࠞࡏ͢Δ • ຊʹॏཁͳͷɺϩάͰͳ͘ۀσʔλͱͯ͠ଊ͑Δ͖ • ຊʹॏཁͳใԿ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ ৴པੑͷߴ͍σʔλετΞʹه͢Δ • ϩάͱ͍͏ΈʹཔΔͷʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ͖
ϩάpush͔pull͔ • ͦͷ্ͰɺϩάPushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafkaRabbitMQΛڬΜͰෛՙΛࢄͰ͖Δ •
Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢ΔࡍʹɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ͕ඞཁͳ͍ʢ͘͠JSONܗࣜͰग़ྗ͢Δʣ߹ ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅Ͱͳ͍߹ͳͲͰ༗ޮ
Spring Boot + TempoͰ τϨʔεऩू
Spring BootͰτϨʔεऩू • τϨʔεऩूͷͨΊʹ࣍ͷ2͕ͭඞཁ • ᶃτϨʔεIDͷ • ʮͲͷϦΫΤετʹඥ͘ॲཧ͔ʯʮ୭͔ΒݺΕ͔ͨʯͱ͍͏ ใΛΞϓϦέʔγϣϯؒͰͤ͞Δ •
ᶄτϨʔεใͷੵ • τϨʔεใΛσʔλετΞʹૹͬͯੵ͢Δ
Spring BootͰτϨʔεIDͷ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTPAMQPʣͷ ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot
Application1 Spring Boot Application2 Spring Boot Application4 Spring Boot Application3 traceparent = req001-01 traceparent = req001-02 traceparent = req001-01
Spring BootͰτϨʔεIDͷ • τϨʔεIDͷૹड৴ʹSpring Boot 2.x·ͰSpring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot
3.0͔ΒMicrometer Tracing͕ ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot Application Micrometer Tracing RestTemplate Spring Boot Application Micrometer Tracing Rest Controller HTTPϔομʹ τϨʔεIDͷ༩ ʢࣗಈʣ HTTPϔομ͔Β τϨʔεIDͷऔಘ ʢࣗಈʣ
Micrometer TracingͷՃ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ -brave ΛՃ͢ΔʢͲͪΒͰৼΔ͍ʹҧ͍ͳ͍ʣ
• Spring BootͷRestTemplateSpring Cloud StreamΛ༻͍ͨ௨৴Ͱ τϨʔεIDͷ͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
Spring BootͰτϨʔεऩू • Spring BootΞϓϦέʔγϣϯͷτϨʔεใͷهΛ Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ • Grafana͕Tempo͔ΒτϨʔεใΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Open Telemetry Tempo ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Open TelemetryͷՃ • Spring Boot͔ΒτϨʔεใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔
zipkin-reporter-brave ΛՃ͢ΔʢৼΔ͍ʹҧ͍ͳ͍ʣ • ZipkinʹτϨʔεใΛૹΔϥΠϒϥϦ ʢTempo͕ZipkinޓAPIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺτϨʔεใΛͲΕ͘Β͍ͷׂ߹Ͱ TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ10%ʣ • ։ൃதσϞͰ100%ʹͨ͠ํ͕ྑ͍ • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ
100%ʹઃఆ͓ͯ͘͠ͱྑ͍ management.tracing.sampling.probability=1.0
TempoͰτϨʔεऩू • ʢTempoࣗମͷߏஙʹׂ͍ͭͯѪʣ • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓAPIΛ༗ޮʹ͢Δ server: http_listen_port: 3200 distributor: receivers:
zipkin:
σϞ
τϨʔεpushͷΈ • ZipkinʢޓAPIؚΉʣτϨʔεใΛड͚औΔͨΊͷAPIΛ ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔεใΛૹΔ ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔Εͳ͍
Spring Boot 3.0 + k8sͷ ϞχλϦϯά
k8s্ͰPullܕΛӡ༻͢Δ՝ • PrometheusͰϞχλϦϯάରͷΞυϨεΛྻڍͨ͠ • k8sͷΑ͏ʹಈతʹϞχλϦϯάର͕૿ݮɾมԽ͢Δ߹ʹ ͲͷΑ͏ʹऩू͢Δͷ͔ʁ
αʔϏεσΟεΧόϦ • ΞϓϦέʔγϣϯͷϧʔςΟϯάϩʔυόϥϯγϯάͷͨΊʹ ༻͍ΒΕΔαʔϏεσΟεΧόϦͷΈΛϞχλϦϯάͰར༻ ͢Δ • k8sͰCoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏
αʔϏεσΟεΧόϦ • 20ඵͰ͔ΔαʔϏεσΟεΧόϦ Spring Boot Application1 Service Registry
Spring Boot Application1 Spring Boot Application1 Spring Boot Application1 ᶃΞυϨεͷొ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot Application2
αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot Application1 Prometheus Promtail Service
Registry Spring Boot Application1 Spring Boot Application1 Spring Boot Application ᶃΞυϨεͷొ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ৭ʑͱઃఆ͕ඞཁ͔ͩΒ PrometheusPromtail helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
PrometheusͷαʔϏεσΟεΧόϦઃఆ • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ annotationsͰϙʔτ൪߸ActuatorͷύεΛࢦఆ͢Δ spec: template: metadata: labels: app:
bff annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: /actuator/prometheus
σϞ
ίϯςφ࣌ͰPullͰେৎ • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷΈͱซ༻͢Ε PullܕͰಛʹͳ͘ϞχλϦϯάڥΛߏஙͰ͖Δ
·ͱΊ
·ͱΊ • ϝτϦΫεɺϩάɺτϨʔεऔΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot
3.0ͰϞχλϦϯάपΓͷϥΠϒϥϦͳͲཧ͞Ε GrafanaελοΫͳͲͰϞχλϦϯά͘͢͠ͳ͍ͬͯΔ • ϩά͔ΒτϨʔεͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
·ͱΊ • PushܕɺPullܕͷ͍ͣΕͰͳ͘σʔλΛऩूͰ͖Δ • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ • DataDogAWS CloudWatchͳͲجຊతʹPushܕ • ΞϓϦέʔγϣϯଆͷιʔείʔυઃఆϑΝΠϧ͕
Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͍͢͠Α
Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot
3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442