Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレー...
Search
Shin Tanimoto
February 28, 2023
Programming
5
2.2k
クラウドネイティブ時代のコンテナ環境におけるJavaアプリケーションのメトリクス・ログ・トレースモニタリング
Java Developers Summit (2023-02-28)
Shin Tanimoto
February 28, 2023
Tweet
Share
More Decks by Shin Tanimoto
See All by Shin Tanimoto
26 Java Years
shintanimoto
0
44
Let’s Have Fun with Reactive Programming, Using Reactor and WebFlux
shintanimoto
0
220
Monitoring and Visualizing Your (Micro)services
shintanimoto
0
330
現代に求められるJavaコミュニティとは / What should be the Java Community of Today?
shintanimoto
0
850
人生がときめく「学び」の魔法 / The Life-Changing Magic of Studying
shintanimoto
6
1.6k
業務で使いたいWebFluxによるReactiveプログラミング / Introduction to Reactive Programming using Spring WebFlux
shintanimoto
9
8.8k
from old Java to modern Java (2017) #jjug
shintanimoto
4
1.5k
Spring Cloud Sleuth + Zipkin with Elasticsearch #zipkin_jp
shintanimoto
1
1.2k
Stream API 入門 #jjug #javajo
shintanimoto
4
3.8k
Other Decks in Programming
See All in Programming
pytest プラグインを開発して DRY に自動テストを書こう
inuatsu
2
260
XP2024 っていう国際会議に行ってきたよの記
bonotake
4
240
ビット演算の話 / Let's play with bit operations
kaityo256
PRO
4
180
[KR] Server Driven Compose With Firebase
skydoves
2
200
自分だけの世界を創るクリエイティブコーディング / Creative Coding: Creating Your Own World
chobishiba
2
1.7k
実践Dash - 手を抜きながら本気で作るデータApplicationの基本と応用 / Dash for Python and Baseball
shinyorke
2
580
Hi, have you met Kotlin Multiplatform? | DevFest Vienna 2024
prof18
0
140
The Efficiency Paradox and How to Save Yourself and the World
hollycummins
0
180
Vue :: Better Testing 2024
up1
1
400
Jakarta EE as Seen Trough the Lens of the ASF
ivargrimstad
0
250
What is TDD?
urakawa_jinsei
1
220
Introduce dRuby
ledsun
0
120
Featured
See All Featured
Designing Experiences People Love
moore
138
23k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
How GitHub (no longer) Works
holman
311
140k
Fontdeck: Realign not Redesign
paulrobertlloyd
81
5.2k
The Mythical Team-Month
searls
218
43k
No one is an island. Learnings from fostering a developers community.
thoeni
19
2.9k
4 Signs Your Business is Dying
shpigford
180
21k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Fireside Chat
paigeccino
32
3k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
364
22k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Designing the Hi-DPI Web
ddemaree
280
34k
Transcript
ΫϥυωΠςΟϒ࣌ͷ ίϯςφڥʹ͓͚ΔJavaΞϓϦέʔγϣϯͷ ϝτϦΫεɾϩάɾτϨʔεϞχλϦϯά Everforth / Acroquest Technology / JJUG Shin
Tanimoto (@cero_t)
ࣗݾհ • ୩ຊ ৺ (Shin Tanimoto) • 2ࣾͰಇ͘ΤϯδχΞ • גࣜձࣾEverforthʢCTOʣ
• Acroquest Technologyגࣜձࣾ • ຊJavaϢʔβʔάϧʔϓʢදʣ • Twitter : @cero_t
ࠓͷϝΠϯςʔϚ
ྩʹ͓͚ΔΦϒβόϦςΟ ͷΞʔΩςΫνϟʹ͍ͭͯ
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε • ϩά • τϨʔε
ϞχλϦϯάͷ3େཁૉ • ϝτϦΫε : ࣌ܥྻͷใ • ϩά : λΠϜελϯϓ͖ͭͷจࣈྻ •
τϨʔε : ݺͼग़͠ͷه
ϝτϦΫε • ࣌ܥྻͷใ • ྫ • OS : CPU༻ɺϝϞϦ༻ྔʢʣɺσΟεΫ༻ྔʢʣ •
JavaVM : ώʔϓ༻ྔʢʣɺεϨουɺίωΫγϣϯϓʔϧ • App : ϦΫΤετɺΤϥʔʢʣ
ϝτϦΫε • αϯϓϦϯάϕʔε • ͦͷॠؒͷ͔ɺूܭͨ͠Λऔಘ͢Δ • CPU༻ : ʮඵؒʯͷ༻ͷฏۉԿ%͔ •
ϝϞϦ༻ྔ : ʮ͍·ʯԿMB͍ͬͯΔ͔ • GCճ : ʮ͍··ͰʯʹԿճGCΛߦ͔ͬͨ • αϯϓϦϯάִ͕ؒখ͍͞΄Ͳਫ਼্͕͕Δ͕ɺσʔλྔ͕૿͑Δ
ϝτϦΫε • ฏͷϞχλϦϯά • OSϨϕϧͷϝτϦΫε͚ͩऔΓ͕ͪ • JavaϨϕϧͷϝτϦΫεΛऔ͍ͬͯΕେͨ͠ͷ • ΞϓϦέʔγϣϯϨϕϧͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δͷ
૬ͳΓ͜Έ • ྩͰͯ͢औΔ͠
ϩά • ࣌ࠁΛ͏จࣈྻ • ྫ • ΞϓϦέʔγϣϯϩάɺOSͷϩάɺGCϩάɺ ϑϨʔϜϫʔΫϛυϧΣΞͷϩά •
ॲཧΛߦͬͨɺΤϥʔ͕ى͖ͨɺԿΒ͔ͷෆ߹ͳ͜ͱ͕ى͖ͨ • ͦͷ࣌ͷίϯςΩετใʢมఆͳͲʣ
ϩά • Πϕϯτϕʔε • ࣄ͕ى͖ͨ࣌ʹه͢Δ • αϯϓϦϯάϕʔεͱҟͳΓɺͯ͢ه͞ΕΔ • จࣈྻͳͷͰϝτϦΫεʹൺͯαΠζ͕େ͖͍ •
1ͭͷϦΫΤετʹΑͬͯۀଆͷσʔλετΞʹॻ͖ࠐ·ΕΔใྔΑΓ ϩάͷํ͕େ͖͘ͳΔ͜ͱ͕͋Δ • ΞϓϦέʔγϣϯͷσʔλετΞΑΓϩάͷσʔλετΞͷํ͕ ߴੑೳɾେ༰ྔͳͷΛٻΊΒΕΔ͜ͱ͕͋Δ
ϩά • ฏͷϞχλϦϯά • ͝ͱʹzipͯ͠ωοτϫʔΫετϨʔδʹஔ͖͕ͪ • ඞཁͳ࣌ʹμϯϩʔυͯ͠։͔͘ɺviͰݟʹߦ͖͕ͪ • ʮERRORʯͱ͍͏ϝοηʔδ͕͋Εύτϥϯϓ͕ճΓ͕ͪ •
ྩͷϞχλϦϯάͰৗʹݕࡧͰ͖Δঢ়ଶʹ͢Δ͠ • ͕ͦͯ͠ݟ͔ͭͬͨΒSlackͳͲʹ௨͢Δ͠
τϨʔε • ݺͼग़͠ͷه • ෳͷΞϓϦέʔγϣϯϓϩηεΛލ͍ͩελοΫτϨʔεͷΑ͏ͳͷ • ͍ͭɺ୭͕୭ΛݺΜͰ͔ɺͲΕ͘Βֻ͍͔͔ͬͨɺޭ͔ࣦͨ͠ഊ͔ͨ͠ • ΠϕϯτϕʔεͰऔಘ •
ͯ͢औಘ͢Δ͔ɺҰఆͷׂ߹Ͱऔಘ͢Δ͔ઃఆՄೳ
τϨʔε • ελοΫτϨʔε • Javaϓϩηε͚ͩͰ؆ܿ͢Δݺͼग़͠֊ • ओʹྫ֎ൃੜ࣌ʹऔಘ͢Δ • ͦΕͳΒෳΞϓϦέʔγϣϯΛލ͍ͩτϨʔεऔΓ͍ͨΑͶ •
τϨʔεɺൃੜ࣌ͷ֬ೝɺϘτϧωοΫݕग़ʹར༻Ͱ͖Δ
τϨʔε • ฏͷϞχλϦϯά • τϨʔεऔΒͳ͍ or औΕͳ͍ • ྩͰऔΓ·͠ΐ͏
Spring Boot 3.0࣌ͷ ϞχλϦϯά
σϞ
σϞͰͬͨιʔείʔυ • https://github.com/cero-t/spring-store-2022 • ࣮ߦखॱઃఆͷৄࡉͳ༰ϒϩάʹهࡌ • ຊεϥΠυͷ࠷ऴϖʔδʹ·ͱΊ·͢
ΞϓϦέʔγϣϯͷߏ BFF Payment Service Order Service Stock
Service Delivery Service Item Service Cart Service Queue ECαΠτͷόοΫΤϯυΛΠϝʔδ
ར༻ٕͨ͠ज़ελοΫ • ϚΠΫϩαʔϏεΞϓϦέʔγϣϯ • Java 17 • Spring Boot 3.0
(without Spring Cloud) • Spring Boot Web • Micrometer • OpenTelemetry
ར༻ٕͨ͠ज़ελοΫ • GrafanaελοΫ • Grafana : ϞχλϦϯάͷUI • Prometheus :
ϝτϦΫεͷऩूͱσʔλετΞ • Loki : ϩάͷσʔλετΞ • Tempo : τϨʔεͷσʔλετΞ
ϞχλϦϯάͷߏ Java Process Grafana Tempo Prometheus Loki औಘͯ͠ՄࢹԽ ϝτϦΫε
ϩά τϨʔε Java Process Java Process Spring Boot Application
Spring Boot + PrometheusͰ ϝτϦΫεऩू
Spring BootͰϝτϦΫεऩू • Spring BootΞϓϦέʔγϣϯͷϝτϦΫεใΛWeb APIͱͯ͠ ެ։ͯ͠ɺPrometheus͔Βఆظతʹऔಘͯ͠ऩूɾอଘ͢Δ • Grafana͕Prometheus͔ΒϝτϦΫεΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application ϝτϦΫε औಘAPI Prometheus ఆظతʹऔಘ Grafana औಘͯ͠ՄࢹԽ
Spring BootͰϝτϦΫεऩू • ϝτϦΫεऔಘͷWeb APIΛެ։͢Δํ๏ • Spring BootΞϓϦέʔγϣϯͷdependencyʹ ActuatorͱMicrometerΛՃ͢Δ
• application.propertiesʹͯActuatorͷΤϯυϙΠϯτΛެ։͢Δ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ • Actuator : spring-boot-starter-actuator • Spring
BootͷઃఆϝτϦΫεΛWeb APIͱͯ͠ެ։͢ΔͨΊͷϥΠϒϥϦ • Micrometer : micrometer-registry-prometheus • OSJavaVMɺSpring BootͷϝτϦΫεใΛऔಘ͢ΔͨΊͷϥΠϒϥϦ • “-prometheus” Prometheus͔ΒΞΫηε͢ΔͨΊͷΤϯυϙΠϯτΛ ActuatorΛ༻͍ͯެ։͢Δɻ • “-datadog” ͱ͍͏ϥΠϒϥϦఆظతʹDataDogʹϝτϦΫεΛૹ৴͢ΔͳͲ ϥΠϒϥϦʹΑͬͯڍಈ͕ҟͳΔ
ActuatorͱMicrometerΛՃ • pom.xml ͷdependencyʹ࣍ͷ2ͭΛՃ <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <scope>runtime</scope> </dependency> <dependency>
<groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency>
ActuatorͷΤϯυϙΠϯτΛެ։ • application.properties ʹͯɺMicrometerʹؔ͢Δ ActuatorͷΤϯυϙΠϯτΛެ։͢Δ management.endpoints.web.exposure.include=* management.metrics.distribution.percentiles-histogram.http.server.requests=true management.metrics.tags.application=${spring.application.name} #
Micrometerʹؔ͢ΔϝτϦΫε͚ͩެ։͢Δ߹ʹ࣍ͷઃఆͰྑ͍ # management.endpoints.web.exposure.include=prometheus
ActuatorͱMicrometerΛՃ • http://(ΞϓϦέʔγϣϯͷΞυϨε)/actuator/prometheus ʹ ΞΫηε͢ΔͱɺϝτϦΫεҰཡΛऔಘͰ͖Δ process_ fi les_max_ fi
les{application="bff",} 10240.0 process_uptime_seconds{application="bff",} 15057.411 jvm_threads_peak_threads{application="bff",} 43.0 jvm_threads_states_threads{application="bff",state="new",} 0.0 jvm_threads_states_threads{application="bff",state="runnable",} 11.0 jvm_threads_states_threads{application="bff",state="terminated",} 0.0 jvm_threads_states_threads{application="bff",state="waiting",} 12.0 jvm_threads_states_threads{application="bff",state="timed-waiting",} 9.0 jvm_threads_states_threads{application="bff",state="blocked",} 0.0
σϞ
PrometheusͰϝτϦΫεऩू • ʢGrafanaPrometheusࣗମͷߏஙʹׂ͍ͭͯѪʣ • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ
PrometheusͰϝτϦΫεऩू • PrometheusͷઃఆϑΝΠϧʹSpring BootΞϓϦέʔγϣϯͷ ΞυϨεΛྻڍ͠ɺϝτϦΫεΛऩूͰ͖ΔΑ͏ʹ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" static_con fi gs: - targets: [ "host.docker.internal:9000", "host.docker.internal:9001", "host.docker.internal:9002", "host.docker.internal:9010" ]
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ
ΫϥυωΠςΟϒ࣌ʹ ΞϓϦέʔγϣϯͷΞυϨεΛ ྻڍ͢Δͱ͔ਖ਼ؾͰ͔͢ʁ ͖ͪΜͱޙͰղઆ͠·͢
σϞ
GrafanaʹμογϡϘʔυΛՃ • GrafanaίϛϡχςΟ͕࡞ͨ͠μογϡϘʔυΛެ։͍ͯ͠Δ • https://grafana.com/grafana/dashboards/ • JVM (Micrometer) - 4701
͕Φεεϝ
σϞ
Prometheuspullܕ • ΞϓϦέʔγϣϯଆ͔ΒσʔλΛૹΔͷΛʮPushܕʯͱݺͿ • ΞϓϦέʔγϣϯʹऔΓʹߦ͘ͷΛʮPullܕʯͱݺͿ • PrometheusʮPullܕʯͰ͋ΓϝτϦΫεऩूͷϛυϧΣΞͱ ͯ͠ΘΓͱ͍͠෦ྨʢ*ݸਓͷײͰ͢ʣ •
PullܕͰࢄΞϓϦέʔγϣϯͷϝτϦΫεΛͳ͘ ऩूͰ͖ΔʢޙͰղઆ͠·͢ʣ
Prometheuspullܕ • Pullܕͷར • ࢹରͱͳΔΞϓϦέʔγϣϯ͕໌֬Ͱ͋ΔͨΊ ʮϝτϦΫεΛऔಘͰ͖ͳ͔ͬͨʯͱ͍͏Λݕग़͍͢͠ • PushܕͰʮૹΒΕͯ͜ͳ͔ͬͨʯ =
ʮه͠ͳ͍ʯ͚ͩͱ ͳΓɺͦΕΛͱͯ͠ݕग़͢Δ͜ͱ͕͍͠ • ࢹର͕ଟ͘ͳͬͨࡍʹɺPushܕͰϝτϦΫεऩूαʔό͕ύϯΫͯ͠ ͠·͏Մೳੑ͕͋Δ͕ɺPullܕͳΒऩूසΛ੍ޚ͢Δ͜ͱ͕Ͱ͖Δʮ͔ʯ
Spring Boot + LokiͰ ϩάऩू
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛPromtailͰఆظతʹऩूͯ͠ Lokiʹૹ৴͢ΔʢPullܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ Spring
Boot Application ϩά ϑΝΠϧ Loki ఆظతʹऩू Grafana औಘͯ͠ՄࢹԽ Promtail ૹ৴
Ͱ͏ɺϩάϑΝΠϧͱ͔ ͍ͨ͘ͳ͍Ͱ͢ΑͶ
Spring BootͰϩάऩू • Spring BootΞϓϦέʔγϣϯͷϩάΛLoki4JͷLogback AppenderΛ ༻͍ͯఆظతʹLokiʹૹ৴͢ΔʢPushܕʣ • Grafana͕Loki͔ΒϩάΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Loki4J Appender Loki ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Loki4JͷLogback AppenderΛՃ • pom.xml ͷdependencyʹloki-logback-appenderΛՃ • ϑΝΠϧίϯιʔϧͰͳ͘LokiʹϩάΛૹΔΑ͏ͳ LogbackͷAppender <dependency>
<groupId>com.github.loki4j</groupId> <artifactId>loki-logback-appender</artifactId> <version>1.3.2</version> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ࡞ • LogbackͷઃఆϑΝΠϧΛ࡞͠ɺAppenderʹLoki4jAppenderΛ ࢦఆ͢Δʢlogback-loki.xmlʣ <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender"> <http> <url>http://localhost:3100/loki/api/v1/push</url>
</http> <format> <label> <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern> </label> <message> <pattern>${FILE_LOG_PATTERN}</pattern> </message> <sortByTime>true</sortByTime> </format> </appender>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺ࡞ͨ͠LogbackͷઃఆϑΝΠϧΛ ༗ޮʹ͢Δ • ͍ͭͰʹɺޙͰτϨʔεऩूͷ࣌ʹ͏τϨʔεID ϩάʹग़ΔΑ͏ʹ͓ͯ͘͠
logging.pattern.level=%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] logging.con fi g=classpath:logback-loki.xml
LokiͰϩάऩू • ʢLokiࣗମͷߏஙʹׂ͍ͭͯѪʣ • LokiΛىಈ͢ΕΞϓϦέʔγϣϯଆ͔ΒϩάΛPush͢ΔͷͰ LokiଆʹಛʹඞཁͳઃఆͳͲͳ͠
σϞ
ϩάpush͔pull͔ • ྺ࢙తʹϩάPullܕͰऩू • ϑΝΠϧʹग़ྗ͠ɺόονͰωοτϫʔΫετϨʔδʹҠಈͤ͞Δ • FluentdLogstashɺFilebeatͳͲͰऩूͯ͠ElasticsearchʹૹΔ • ϩάΛPushܕʹ͢Δͷෆ҆ •
ϩάʮγεςϜͷෆਖ਼ऴྃલͷ࠷ظͷʯ • ΞϓϦέʔγϣϯϓϩηε͕ࢮΜͰϩά͚͍ͩͨ͠ • PushܕͰड͚औΔଆͷϛυϧΣΞ͕ࢭ·Δͱϩά͕ࣦΘΕΔ
ͦΜͳʹॏཁͳใΛ ͳͥϩάͳͲͱ͍͏ᐆດͳํ๏Ͱ ͦ͏ͱ͢Δͷ͔ʁ
ϩάͷ֓೦Λݟ͖͢ • ఏݴ : ϩάͱ͍͏֓೦ͦͷͷΛݟ͖͢ • ϩάॏཁͱ͞ΕΔΘΓʹݫີʹઃܭ͞Εͳ͍ࣄ͕ଟ͍ • ํ͚ܾͩΊͯޙ։ൃऀͤʢͦͷΘΓʹɺ༗༻Ͱͳ͚ΕౖΒΕΔʣ •
ॏཁͳใͱɺͦ͏Ͱͳ͍ใ͕ࡶଟʹࠞࡏ͢Δ • ຊʹॏཁͳͷɺϩάͰͳ͘ۀσʔλͱͯ͠ଊ͑Δ͖ • ຊʹॏཁͳใԿ͔Λਫ਼ࠪ͠ɺݫີʹઃܭͨ͠͏͑Ͱ ৴པੑͷߴ͍σʔλετΞʹه͢Δ • ϩάͱ͍͏ΈʹཔΔͷʮ͋ͬͨΒخ͍͠ʯʹཹΊΔ͖
ϩάpush͔pull͔ • ͦͷ্ͰɺϩάPushܕ͕ྑ͍ͷ͔ɺPullܕ͕ྑ͍ͷ͔ • Pullܕͷํ͕ॊೈੑ͕ߴ͍ • ऩूͯ͠Ճ͢Δͱ͍͏ϓϩηεΛܦΔ͜ͱ͕Ͱ͖Δ • ͦͷաఔʹKafkaRabbitMQΛڬΜͰෛՙΛࢄͰ͖Δ •
Pushܕͷํ͕༰қ • ϩʔΧϧPCͰ࣮ߦ͢ΔࡍʹɺϩάϑΝΠϧΛग़ྗ͠ͳͯ͘ࡁΉ • Ճ͕ඞཁͳ͍ʢ͘͠JSONܗࣜͰग़ྗ͢Δʣ߹ ϩάऩूαʔόʹߴෛՙֻ͕͔ΔΑ͏ͳ݅Ͱͳ͍߹ͳͲͰ༗ޮ
Spring Boot + TempoͰ τϨʔεऩू
Spring BootͰτϨʔεऩू • τϨʔεऩूͷͨΊʹ࣍ͷ2͕ͭඞཁ • ᶃτϨʔεIDͷ • ʮͲͷϦΫΤετʹඥ͘ॲཧ͔ʯʮ୭͔ΒݺΕ͔ͨʯͱ͍͏ ใΛΞϓϦέʔγϣϯؒͰͤ͞Δ •
ᶄτϨʔεใͷੵ • τϨʔεใΛσʔλετΞʹૹͬͯੵ͢Δ
Spring BootͰτϨʔεIDͷ • Spring BootΞϓϦέʔγϣϯͷؒͷ௨৴ʢHTTPAMQPʣͷ ϔομʹτϨʔεIDΛೖΕͯૹ৴ɺड৴͢Δ Spring Boot
Application1 Spring Boot Application2 Spring Boot Application4 Spring Boot Application3 traceparent = req001-01 traceparent = req001-02 traceparent = req001-01
Spring BootͰτϨʔεIDͷ • τϨʔεIDͷૹड৴ʹSpring Boot 2.x·ͰSpring Cloud Sleuth Λ༻͍͍͕ͯͨɺSpring Boot
3.0͔ΒMicrometer Tracing͕ ར༻Ͱ͖ΔΑ͏ʹͳͬͨ Spring Boot Application Micrometer Tracing RestTemplate Spring Boot Application Micrometer Tracing Rest Controller HTTPϔομʹ τϨʔεIDͷ༩ ʢࣗಈʣ HTTPϔομ͔Β τϨʔεIDͷऔಘ ʢࣗಈʣ
Micrometer TracingͷՃ • pom.xml ͷdependencyʹ micrometer-tracing-bridge-otel ͔ -brave ΛՃ͢ΔʢͲͪΒͰৼΔ͍ʹҧ͍ͳ͍ʣ
• Spring BootͷRestTemplateSpring Cloud StreamΛ༻͍ͨ௨৴Ͱ τϨʔεIDͷ͕ߦΘΕΔʢࣗಈʣ <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <scope>runtime</scope> </dependency>
Spring BootͰτϨʔεऩू • Spring BootΞϓϦέʔγϣϯͷτϨʔεใͷهΛ Open Telemetry͔BraveͷϥΠϒϥϦΛ༻͍ͯTempoʹૹ৴͢Δ • Grafana͕Tempo͔ΒτϨʔεใΛऔಘͯ͠ՄࢹԽ͢Δ
Spring Boot Application Open Telemetry Tempo ఆظతʹૹ৴ Grafana औಘͯ͠ՄࢹԽ
Open TelemetryͷՃ • Spring Boot͔ΒτϨʔεใΛૹ৴͢Δํ๏ • pom.xml ͷdependencyʹ opentelemetry-exporter-zipkin ͔
zipkin-reporter-brave ΛՃ͢ΔʢৼΔ͍ʹҧ͍ͳ͍ʣ • ZipkinʹτϨʔεใΛૹΔϥΠϒϥϦ ʢTempo͕ZipkinޓAPIΛඋ͍͑ͯΔͨΊར༻Ͱ͖Δʣ <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-zipkin</artifactId> <scope>runtime</scope> </dependency>
LogbackͷઃఆϑΝΠϧΛ༗ޮʹ͢Δ • application.properties ʹͯɺτϨʔεใΛͲΕ͘Β͍ͷׂ߹Ͱ TempoʹૹΔ͔Λઃఆ͢ΔʢσϑΥϧτ10%ʣ • ։ൃதσϞͰ100%ʹͨ͠ํ͕ྑ͍ • ӡ༻தͷΤϥʔൃੜ࣌ʹ༨ͣ͞τϨʔεΛऔಘ͢ΔͨΊʹ
100%ʹઃఆ͓ͯ͘͠ͱྑ͍ management.tracing.sampling.probability=1.0
TempoͰτϨʔεऩू • ʢTempoࣗମͷߏஙʹׂ͍ͭͯѪʣ • TempoͷઃఆϑΝΠϧʹͯɺZipkinޓAPIΛ༗ޮʹ͢Δ server: http_listen_port: 3200 distributor: receivers:
zipkin:
σϞ
τϨʔεpushͷΈ • ZipkinʢޓAPIؚΉʣτϨʔεใΛड͚औΔͨΊͷAPIΛ ެ։͓ͯ͠Γ֤ΞϓϦέʔγϣϯ͔Βͦ͜ʹτϨʔεใΛૹΔ ʮPushܕʯͷΞʔΩςΫνϟ͕࠾༻͞Ε͍ͯΔ • ͍ͣΕPullܕͷΤʔδΣϯτ͕ग़ͯ͘Δ͔Εͳ͍
Spring Boot 3.0 + k8sͷ ϞχλϦϯά
k8s্ͰPullܕΛӡ༻͢Δ՝ • PrometheusͰϞχλϦϯάରͷΞυϨεΛྻڍͨ͠ • k8sͷΑ͏ʹಈతʹϞχλϦϯάର͕૿ݮɾมԽ͢Δ߹ʹ ͲͷΑ͏ʹऩू͢Δͷ͔ʁ
αʔϏεσΟεΧόϦ • ΞϓϦέʔγϣϯͷϧʔςΟϯάϩʔυόϥϯγϯάͷͨΊʹ ༻͍ΒΕΔαʔϏεσΟεΧόϦͷΈΛϞχλϦϯάͰར༻ ͢Δ • k8sͰCoreDNSΛ༻͍ͨαʔϏεσΟεΧόϦΛߦ͏
αʔϏεσΟεΧόϦ • 20ඵͰ͔ΔαʔϏεσΟεΧόϦ Spring Boot Application1 Service Registry
Spring Boot Application1 Spring Boot Application1 Spring Boot Application1 ᶃΞυϨεͷొ ᶄݺͼग़͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠ Spring Boot Application2
αʔϏεσΟεΧόϦ • αʔϏεσΟεΧόϦΛར༻ͨ͠ϞχλϦϯά Spring Boot Application1 Prometheus Promtail Service
Registry Spring Boot Application1 Spring Boot Application1 Spring Boot Application ᶃΞυϨεͷొ ᶄϞχλϦϯά͍ͨ͠αʔϏεͷΞυϨεΛऔಘ ᶅݺͼग़͠
PrometheusͷαʔϏεσΟεΧόϦઃఆ • PrometheusͷઃఆϑΝΠϧʹͯɺkubernetes_sd_con fi g ͰαʔϏεσΟεΧόϦΛઃఆ͢Δ scrape_con fi gs: -
job_name: "apps" metrics_path: "/actuator/prometheus" kubernetes_sd_con fi g: - role: pod relabel_con fi gs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+?)(?::\d+)?;(\d+) replacement: $1:$2 ଞʹ৭ʑͱઃఆ͕ඞཁ͔ͩΒ PrometheusPromtail helmͰΠϯετʔϧ͢Δͷ͕Φεεϝ
PrometheusͷαʔϏεσΟεΧόϦઃఆ • ΞϓϦέʔγϣϯΛσϓϩΠ͢ΔϚχϑΣετϑΝΠϧʹͯ annotationsͰϙʔτ൪߸ActuatorͷύεΛࢦఆ͢Δ spec: template: metadata: labels: app:
bff annotations: prometheus.io/scrape: "true" prometheus.io/port: "9000" prometheus.io/path: /actuator/prometheus
σϞ
ίϯςφ࣌ͰPullͰେৎ • ϚΠΫϩαʔϏεͷΑ͏ʹಈతʹ૿ݮ͢ΔΞϓϦέʔγϣϯ܈Λ ϞχλϦϯά͢ΔࡍɺαʔϏεσΟεΧόϦͷΈͱซ༻͢Ε PullܕͰಛʹͳ͘ϞχλϦϯάڥΛߏஙͰ͖Δ
·ͱΊ
·ͱΊ • ϝτϦΫεɺϩάɺτϨʔεऔΕΔ͚ͩऔΓ·͠ΐ͏ • GrafanaελοΫͰ͋ΔɺGrafanaɺPrometheusɺLokiɺTempoͰ ·ͱΊͯϞχλϦϯάͰ͖Δ • Spring Boot
3.0ͰϞχλϦϯάपΓͷϥΠϒϥϦͳͲཧ͞Ε GrafanaελοΫͳͲͰϞχλϦϯά͘͢͠ͳ͍ͬͯΔ • ϩά͔ΒτϨʔεͷભҠͳͲɺ͜ΕΒͷ3ͭΛͦΕͧΕ ભҠͰ͖ΔΑ͏ʹ͢ΔτϨϯυ͕͋Δ
·ͱΊ • PushܕɺPullܕͷ͍ͣΕͰͳ͘σʔλΛऩूͰ͖Δ • ૯ͯ͡ɺPushܕͷํ͕༰қɺPullܕͷํ͕ॊೈͱͳ͍ͬͯΔ • DataDogAWS CloudWatchͳͲجຊతʹPushܕ • ΞϓϦέʔγϣϯଆͷιʔείʔυઃఆϑΝΠϧ͕
Πϯϑϥʹґଘ͠ͳ͍Α͏ཹҙ͢Δͱ։ൃ͍͢͠Α
Resources • ιʔείʔυ • https://github.com/cero-t/spring-store-2022 • υΩϡϝϯτ • Spring Boot
3.0ΞϓϦέʔγϣϯΛGrafanaελοΫͰՄࢹԽͯ͠Έͨɻ • https://cero-t.hatenadiary.jp/entry/2023/01/10/163547 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ1 Grafana + Prometheusฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/11/181715 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ2 Grafana + Lokiฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/12/130022 • GrafanaελοΫʹΑΔSpring BootΞϓϦέʔγϣϯࢹͷৄࡉʢͦͷ3 Grafana + Tempoฤʣ • https://cero-t.hatenadiary.jp/entry/2023/01/15/203442