計装を見直してアプリケーションパフォーマンスを改善させた話

3-shake SRE Tech Talk #12 オンサイト LT
https://3-shake.connpass.com/event/351861/

1. © 2025 Wantedly, Inc. ܭ૷Λݟ௚ͯ͠ΞϓϦέʔγϣϯ ύϑΥʔϚϯεΛվળͤͨ͞࿩ 3-shake SRE Tech Talk #12 LT May.16 2025 - Kazuki Obata 1
2. © 2025 Wantedly, Inc. Takeaways • ܭ૷ͷݟ௚͠ͰύϑΥʔϚϯεվળͷޮՌ͕͋Δ ◦ ෆཁͳܭ૷ΛऔΓআ͍ͨ݁ՌɺฏۉϨΠςϯγ͕ 1% վળ • ܭଌ͸େࣄ • ٕज़తෛ࠴ͷղফ͸ύϑΥʔϚϯεվળͷ༗ޮͳखஈʹͳΓ͏Δ 2
3. © 2025 Wantedly, Inc. ໨࣍ 01 ͳͥܭ૷ΛऔΓআ͘ͷ͔ 02 ໨తͱૂ͍ 03 ޮՌଌఆͷํ๏ 04 ݁Ռ 05 ·ͱΊͱֶͼ 3
4. © 2025 Wantedly, Inc. ͳͥܭ૷ΛऔΓআ͘ͷ͔ 01 4
5. © 2025 Wantedly, Inc. ͳͥܭ૷ΛऔΓআ͘ͷ͔ • ࡢ೥ͷ12݄͝Ζʹ New Relic ͔Β Datadog ΁Ҡߦͨ͠ • New Relic ͷܭ૷ࣗମ͸ෆཁʹͳͬͨ ◦ ΞϥʔτҠߦʹ͍ͭͯ͸೥຤ʹॻ͍ͨϒϩάͰগ͠৮Ε͍ͯΔ 5 - Datadog ΁ͷҠߦ
6. © 2025 Wantedly, Inc. ͳͥܭ૷ΛऔΓআ͘ͷ͔ • ܭ૷ϥΠϒϥϦ͕িಥͯ͠ΞοϓσʔτΛ્֐͍ͯͨ͠ ◦ ৽͍͠ػೳΛ࢖͑ͳ͍ ◦ ։ൃऀͷੜ࢈ੑ͕ଛͳΘΕ͍ͯΔ • ΞϓϦέʔγϣϯ΁ͷΦʔόʔϔου͕৐ͬͨ··ʹͳΔ ◦ ෆཁͳͷͰऔΓআ͖͍ͨ ◦ ύϑΥʔϚϯε͕վળ͞ΕΔ...͔΋ʁ 6 - ࢒ͬͨ՝୊
7. © 2025 Wantedly, Inc. ܭ૷আ٫ͷ໨తͱૂ͍ 02 7
8. © 2025 Wantedly, Inc. ໨త • ґଘϥΠϒϥϦ͕όʔδϣϯΞοϓͰ͖ΔΑ͏ʹ͢Δ ◦ ܭ૷ΛऔΓআ͍ͯϥΠϒϥϦΛഇࢭ • ͋ΘΑ͘͹ΞϓϦέʔγϣϯύϑΥʔϚϯεͷվળ 8
9. © 2025 Wantedly, Inc. ໨త • ґଘϥΠϒϥϦ͕όʔδϣϯΞοϓͰ͖ΔΑ͏ʹ͢Δ ◦ ܭ૷ΛऔΓআ͍ͯϥΠϒϥϦΛഇࢭ • ͋ΘΑ͘͹ΞϓϦέʔγϣϯύϑΥʔϚϯεͷվળ 9 Ͳ͏΍ͬͯҠߦ͔ͨ͠͸ ผͷػձʹ͓࿩͠·͢
10. © 2025 Wantedly, Inc. ޮՌଌఆͷํ๏ 03 10
11. © 2025 Wantedly, Inc. ܭଌ͢Δͱ͜Ζ • ͲΕΛଌΔͷ͔ ◦ ࢖ΘΕ͍ͯΔαʔϏε͔ΒϐοΫΞοϓͯ͠ܭଌ ▪ ݴޠ͝ͱʹ͕ࠩͰΔͱݴΘΕ͍ͯΔ ◦ New Relic APM agent ΛೖΕ͍ͯͨฐࣾαʔϏε • ࢦඪ ◦ ֤αʔϏεͷฏۉϨεϙϯελΠϜ 11
12. © 2025 Wantedly, Inc. Ͳ͏΍ͬͯଌΔͷ͔ • Istio ͷτϨʔεϝτϦΫεΛར༻ ◦ 15ϲ݄࢒ΔͷͰൺֱ͕΍Γ΍͍͢ ▪ Indexed span ͩͱอ࣋ظؒ͸15೔ 12 a = sum:istio.mesh.request.duration.milliseconds.sum.total {namespace: wantedly, env:production, reporter:source}.as_count() b = sum:istio.mesh.request.count.total {namespace: wantedly, env:production , reporter:source}.as_count() ฏۉϨεϙϯελΠϜ: a / b
13. © 2025 Wantedly, Inc. ݁Ռ 04 13
14. © 2025 Wantedly, Inc. 14 ݁Ռ • ໨ඪͱͨ͠ฏۉϨεϙϯελΠϜ 1% վળΛୡ੒ 🎉 ◦ શαʔϏεฏۉͰ࠷େ6%վળ͍ͯͨ͠ ◦ αʔϏε͝ͱͰ͸࠷େͰ 98.7% վળ͍ͯͨ͠
15. © 2025 Wantedly, Inc. ⚠ ͢΂ͯͷΞϓϦέʔγϣϯͰվળͨ͠༁Ͱ͸ແ͍ 15 • ࠓݟͤͨͷ͸มԽͷେ͖͔ͬͨҰ෦ͷαʔϏε • ΞΫηεͷଟ͍αʔϏεͰ͸͋·Γվળ͕ݟ͑ͳ͔ͬͨ • ਫ਼ࠪग़དྷ͍ͯͳ͍͕ݴޠ͝ͱͷࠩ΋͋Δʁ ◦ େ͖͘վળ͞Ε͍ͯͨɿRuby, ΞΫηεස౓ͷগͳ͍ ▪ Ruby ʹ͍ͭͯ͸ଞͷਓ΋ݴٴ͍ͯ͠ΔͷͰͦ͏͍͏܏޲ʹ͋Δʁ ◦ ͋·Γվળ͞Ε͍ͯͳ͔ͬͨɿPython, Go, Node.js
16. © 2025 Wantedly, Inc. ·ͱΊͱֶͼ 05 16
17. © 2025 Wantedly, Inc. ·ͱΊͱֶͼ 17 • ܭ૷ͷݟ௚͠ͰύϑΥʔϚϯεվળͷޮՌ͕͋ͬͨ ◦ ͋·Γظ଴͍ͯ͠ͳ͔͕ͬͨͦΕͳΓʹޮՌ͸͋ͬͨ • ܭଌ͸େࣄ ◦ ఆྔతʹධՁͰ͖ΔΑ͏ʹ͓ͯ͘͠ ◦ Istio ͱ Datadog ͷ͓ӄͰύοͱݟΕΔঢ়ଶ • ٕज़తෛ࠴ͷղফ͸ύϑΥʔϚϯεվળͷखஈͷͻͱͭʹͳΔ ◦ ܶతͳมԽͰͳ͘ͱ΋஍ಓͳෛ࠴ฦࡁʹՁ஋͸͋Δ