アクセス数5倍の急拡大サービスと共に成長するチーム / Accelerate product and team growth with Datadog

2021/4/14 Datadog Observability Dayでの登壇資料です。
https://hopin.com/events/datadog-observability-day-spring-edition

1. ΞΫηε਺5ഒͷٸ֦େαʔϏε ͱڞʹ੒௕͢ΔνʔϜ 2021/04/14 Datadog Observability Day Takashi Kaga @ CAMPFIRE
2. ࣗݾ঺հ ɾՃզ وࢤ (͔͕ ͔ͨ͠)ɹ@TAKA_041 1 ɾגࣜձࣾCAMPFIRE ϓϩμΫτ։ൃ෦ ɹSREνʔϜॴଐ ɾ2020೥10݄ʹCAMPFIREೖࣾ ɹझຯ͸DatadogΛ͍͡Δ͜ͱ
3. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
4. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
5. CAMPFIREͷ঺հ
6. CAMPFIREͷ঺հ
7. CAMPFIREͷ঺հ
8. CAMPFIREͷ঺հ
9. CAMPFIREͷ։ൃνʔϜ
10. CAMPFIREͷΠϯϑϥߏ੒
11. CAMPFIREͷΠϯϑϥߏ੒
12. CAMPFIREͷΠϯϑϥߏ੒
13. CAMPFIREͷΠϯϑϥߏ੒
14. CAMPFIREͷΠϯϑϥߏ੒ ECS on EC2͔Β ECS FargateʹҠߦத
15. CAMPFIREͷαʔϏε੒௕
16. CAMPFIREͷαʔϏε੒௕ ϦΫΤετ਺ 4ഒ ϦΫΤετ਺ 5ഒ
17. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
18. CAMPFIREͷαʔϏε੒௕
19. CAMPFIREͷαʔϏε੒௕
20. ౰࣌ͷରԠ ɾΠϯελϯεͷ୆਺ͱεϖοΫͷݟ௚͠ ɾεϩʔΫΤϦͷղফ ɾϞχλϦϯάͷݟ௚͠ ɾϏδωεαΠυͱີͳίϛϡχέʔγϣϯ
21. ౰࣌ͷରԠ Ұϲ݄ͰΞΫηε͕4ഒʹͳͬͨ࿩ https://note.com/campfire_dev/n/n99fba1c86328
22. ౰࣌ͷରԠΛৼΓฦͬͯ Πϯϑϥ͚ͩͰ͸ͳ͘ ΞϓϦέʔγϣϯ͔Β΋ վળΛߦ͍͖͍ͬͯͨ
23. ΞϓϦέʔγϣϯ͔Βͷվળ վળʹ͋ͨͬͯͷݱঢ়੔ཧ ɹɾΠϯϑϥͷϝτϦΫε͸औಘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷϩά͸S3ʹอଘͰ͖͍ͯΔ
24. ΞϓϦέʔγϣϯ͔Βͷվળ վળʹ͋ͨͬͯͷݱঢ়੔ཧ ɹɾΠϯϑϥͷϝτϦΫε͸औಘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷϩά͸S3ʹอଘͰ͖͍ͯΔ ɹɾΞϓϦέʔγϣϯͷৄࡉͳಈ͖Λ೺ѲͰ͖͍ͯͳ͍
25. ΞϓϦέʔγϣϯ͔Βͷվળ ΞϓϦέʔγϣϯͷ ΦϒβʔόϏϦςΟ
26. ΦϒβʔόϏϦςΟʁ ɾObservability = Մ؍ଌੑ ɹɾγεςϜͷ಺෦ঢ়ଶΛ؍ଌ͢ΔͨΊͷೳྗ
27. ΦϒβʔόϏϦςΟʁ ɾObservability = Մ؍ଌੑ ɹɾγεςϜͷ಺෦ঢ়ଶΛ؍ଌ͢ΔͨΊͷೳྗ ɹɹɾͦͷγεςϜ͸Կ͕Ͳ͏ಈ͍͍ͯΔ͔
28. ϞχλϦϯάͱΦϒβʔόϏϦςΟ “ϞχλϦϯά͸νʔϜ͕γεςϜͷঢ়ଶΛ؂ࢹͯ͠೺ѲͰ͖ΔΑ͏ʹ͢Δπʔ ϧ΍ٕज़తͳιϦϡʔγϣϯͰ͢ɻϞχλϦϯάͰ͸ɺࣄલʹఆٛͨ͠ࢦඪ΍ϩ άͷऩूΛߦ͍·͢ɻ ΦϒβʔόϏϦςΟ͸νʔϜ͕γεςϜͷσόοάΛੵۃతʹߦ͑ΔΑ͏ʹ͢Δ πʔϧ΍ٕज़తͳιϦϡʔγϣϯͰ͢ɻΦϒβʔόϏϦςΟͰ͸ɺࣄલʹఆٛ͞ Ε͍ͯͳ͍ϓϩύςΟͱύλʔϯͷௐࠪΛߦ͍·͢ɻ” DevOps ଌఆ: ϞχλϦϯάͱΦϒβʔόϏϦςΟ https://cloud.google.com/solutions/devops/devops-measurement-monitoring-and-observability
29. ϗϫΠτϘοΫε ϞχλϦϯά “࠷΋Ұൠతͳཁૉͱͯ͠͸ɺࢦඪɺϩάɺτϨʔεͷ 3 ͕ͭ͋Γ·͢ɻҰ෦ͷ ϞχλϦϯά γεςϜͰ͸ΠϕϯτΛ௥੻ͯ͠ใࠂ͠·͢ɻΠϕϯτ͸ɺγες ϜͰͷϢʔβʔͷૢ࡞΍γεςϜࣗମͷঢ়ଶͷมԽΛද͠·͢ɻ ʢதུʣ ࢦඪɺϩάɺτϨʔεΛଌఆର৅ͷαʔόʔ·ͨ͸ྡ઀͢ΔΤʔδΣϯτ͔ΒϞ χλϦϯά γεςϜʹૹ৴͠ɺγεςϜͷঢ়ଶΛ֬ೝ·ͨ͸ਪଌͰ͖·͢ɻ ” DevOps ଌఆ: ϞχλϦϯάͱΦϒβʔόϏϦςΟ https://cloud.google.com/solutions/devops/devops-measurement-monitoring-and-observability
30. CAMPFIREͱΦϒβʔόϏϦςΟ CloudWatchʹΑΓ ϝτϦΫεΛऔಘ ֤छϩά͸ S3ʹอଘ ͜͜ͰτϨʔεΛ ୲౰ͤ͞Δ
31. ΞϓϦέʔγϣϯͷτϨʔε ɾAPM(Application Performance Monitoring)Λ༻͍Δ ɹɾDatadog΍NewRelicɺSplunkͳͲ͕ఏڙ ɹɹɾಋೖ͸αʔϏεܦݧऀ͕ϦʔυͰ͖ͨํ͕ྑ͍ ɹɹɾSREνʔϜʹDatadogར༻ܦݧऀ͕2໊(ࣗ෼ؚΉ)
32. ΞϓϦέʔγϣϯͷτϨʔε ɾAPM(Application Performance Monitoring)Λ༻͍Δ ɹɾDatadog΍NewRelicɺSplunkͳͲ͕ఏڙ ɹɹɾಋೖ͸αʔϏεܦݧऀ͕ϦʔυͰ͖ͨํ͕ྑ͍ ɹɹɾSREνʔϜʹDatadogར༻ܦݧऀ͕2໊(ࣗ෼ؚΉ)
33. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
34. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɾDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ
35. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά
36. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά ɾ΍ͬͨ͜ͱ ɹɾDatadogಋೖͷॾʑ(APMɺμογϡϘʔυɺLogs ) ɹɾυΩϡϝϯτͷ੔උ ɹɾϨΫνϟʔձͷ։࠵
37. Datadogಋೖલ
38. Datadogಋೖޙ αΠυΧʔͰDatadog AgentίϯςφΛ഑ஔ fluentdίϯςφ͔Β Datadog Logsʹసૹ
39. DatadogಋೖͷྲྀΕ ɾ΍Γ͍ͨ͜ͱ ɹɾAPMಋೖʹΑΔΞϓϦέʔγϣϯͷτϨʔγϯά ɾ΍ͬͨ͜ͱ ɹɾDatadogಋೖͷॾʑ(APMɺμογϡϘʔυɺLogs ) ɹɾυΩϡϝϯτͷ੔උ ɹɾϨΫνϟʔձͷ։࠵
40. υΩϡϝϯτ&ϨΫνϟʔ ɾಋೖͯ͠ऴΘΓʹͨ͘͠ͳ͍ ɹɾͦ΋ͦ΋DatadogͰԿ͕Ͱ͖Δͷ͔Λ෍ڭ͢Δ ɹɾಋೖ͢ΔϝϦοτ͕͋ΔࣄΛ෍ڭ͢Δ ɹɾAPMͷ͢͝͞ͱศར͞Λ෍ڭ͢Δ ɹɾՃզ͕࣭໰૭ޱΛ୲౰͢Δ
41. υΩϡϝϯτ&ϨΫνϟʔ ɾಋೖͯ͠ऴΘΓʹͨ͘͠ͳ͍ ɹɾͦ΋ͦ΋DatadogͰԿ͕Ͱ͖Δͷ͔Λ෍ڭ͢Δ ɹɾಋೖ͢ΔϝϦοτ͕͋ΔࣄΛ෍ڭ͢Δ ɹɾAPMͷ͢͝͞ͱศར͞Λ෍ڭ͢Δ ɹɾՃզ͕࣭໰૭ޱΛ୲౰͢Δ ࣾ಺ͰͷΤόϯδΣϦετ తͳಈ͖Ͱαϙʔτ
42. DatadogಋೖͷྲྀΕ Datadog APMͷ׆༻Ͱര଎վળͷר https://note.com/campfire_dev/n/ne027713b8a61
43. Datadog APM “Datadog APM ͱ෼ࢄܕτϨʔγϯά͸ɺWeb αʔϏεɺΩϡʔɺσʔλϕʔε͕ ϦΫΤετɺΤϥʔɺϨΠςϯγʔΛ؂ࢹ͢ΔͨΊͷඪ४ͷύϑΥʔϚϯεμο γϡϘʔυΛ࢖༻ͯ͠ɺΞϓϦέʔγϣϯΛৄࡉʹՄࢹԽ͠·͢ɻ෼ࢄܕτϨʔε ͸ɺϗετɺίϯςφɺϓϩΩγɺαʔόʔϨεػೳશମͰɺϒϥ΢βηογϣϯɺ ϩάɺsynthetic νΣοΫɺωοτϫʔΫɺϓϩηεɺΠϯϑϥετϥΫνϟʔͷϝ τϦΫεʹγʔϜϨεʹؔ࿈෇͚ΒΕ·͢ɻ” APMͱ෼ࢄܕτϨʔγϯά https://docs.datadoghq.com/ja/tracing/
44. Datadog APMͰग़དྷΔ͜ͱ ɾϦΫΤετʙϨεϙϯεͷྲྀΕΛՄࢹԽ ɹɾspanͱ͍͏࿦ཧతͳ࡞ۀ୯ҐͰ·ͱΊΒΕΔ ɹɹɾnginx, rails, redis, mysqlͳͲͷҰ௨Γͷॲཧ ɹɹɾ࣮ߦ͞ΕͨSQLͷΫΤϦ·ͰՄࢹԽ
45. Datadog APMͰग़དྷΔ͜ͱ
46. Datadog APMͰग़དྷΔ͜ͱ
47. Datadog APMͰग़དྷΔ͜ͱ
48. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɹDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ
49. DatadogಋೖʹΑΔվળͷτϥΠ վળͷΰʔϧ ɹɹDatadog APMʹΑΓγεςϜͷ಺෦ঢ়ଶΛ؍ଌͰ͖Δ ɹɾ؍ଌ݁Ռ͔ΒΑΓྑ͍ϞχλϦϯά͕ߦ͍͑ͯΔ
 ɹɹ(in progress)
50. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ
51. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ ɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ ɹɾϏδωεཁ݅ʹӨڹ͕͋Δ΋ͷ
52. ϞχλϦϯάͷઃܭ ɾΑΓޮՌతͳϞχλϦϯάΛߟ͑Δ ɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ ɹɾϏδωεཁ݅ʹӨڹ͕͋Δ΋ͷ ΠϯϑϥدΓͷϞχλϦϯά͔Β ΞϓϦέʔγϣϯدΓͷϞχλϦϯά΁
53. Datadog͕΋ͨΒͨ͠΋ͷ νʔϜͷ։ൃྗ޲্
54. νʔϜͷ։ൃྗ޲্ ɾಉ͡ਓ਺Ͱ΋͞Βʹଟ͘ͷࣄ͕Ͱ͖ΔΑ͏ʹ ɹɾૣظͷΤϥʔݕ஌ ɹɾஅଓతʹൃੜ͍ͯ͠ΔΤϥʔͷղফ ɹɾఆྔతͳύϑΥʔϚϯεվળ ɹɾΑΓ౿ΈࠐΜͩΞϥʔτରԠͷٞ࿦
55. νʔϜͷ։ൃྗ޲্ ɾಉ͡ਓ਺Ͱ΋͞Βʹଟ͘ͷࣄ͕Ͱ͖ΔΑ͏ʹ ɹɾૣظͷΤϥʔݕ஌ ɹɾஅଓతʹൃੜ͍ͯ͠ΔΤϥʔͷղফ ɹɾఆྔతͳύϑΥʔϚϯεվળ ɹɾΑΓ౿ΈࠐΜͩΞϥʔτରԠͷٞ࿦
56. அଓతΤϥʔͷղফ ɾAPMಋೖޙɺஅଓతʹൃੜ͍ͯ͠Δ5XXΤϥʔ͕ՄࢹԽ ɹɾ։ൃνʔϜͱSREνʔϜͱͰ͜ΕΛௐࠪ ɹɾ౰೔தʹݪҼ൑໌ & σϓϩΠʹͯղফ ɹɾDev🤝Ops
57. அଓతΤϥʔͷղফ
58. அଓతΤϥʔͷղফ
59. Datadog͕΋ͨΒͨ͠΋ͷ Datadog͕ νʔϜͷ੒௕ΛՃ଎ͤ͞Δ
60. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
61. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠
62. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾ֎ܗ؂ࢹʹΑΔԠ౴଎౓ͷௐࠪ ɹɹɾίʔυϦʔσΟϯάʹΑΔௐࠪ
63. ܦӦ՝୊ʹ׆༻͢ΔDatadog Ͳ͏ͤͳΒDatadogΛ ׆༻ͯ͠Έ·ͤΜ͔ʁ
64. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾ֎ܗ؂ࢹʹΑΔԠ౴଎౓ͷௐࠪ ɹɹɾίʔυϦʔσΟϯάʹΑΔௐࠪ ɹɹɾDatadog APMʹΑΔΤϯυϙΠϯτ୯ҐͰͷௐࠪ ɹɹɾμογϡϘʔυͰͷܧଓతͳϝτϦΫενΣοΫ
65. ܦӦ՝୊ʹ׆༻͢ΔDatadog ಛఆͷׂ߹ͷϢʔβʔʹର͠ ଎౓վળͷϦϦʔεΛ࣮ࢪ ૝ఆͨ͠ޮՌ͕֬ೝͰ͖ͨͷͰ ඍௐ੔ͨ͠ޙʹ࠶ϦϦʔε
66. ܦӦ՝୊ʹ׆༻͢ΔDatadog ɾτοϓϖʔδͷ଎౓վળ͕ܦӦ՝୊ʹ ɹɾ͜Ε·Ͱ΋ܧଓͯ͠ௐࠪΛ࣮ࢪ͖ͯͨ͠ ɹɹɾDatadogʹΑΓৄࡉ͔ͭఆྔతͳݕূ͕Մೳʹ
67. ΞδΣϯμ 1. CAMPFIREͷ঺հ 2. αʔϏεͷ੒௕ͱݟ͖͑ͯͨ՝୊ 3. DatadogಋೖʹΑΔվળͷτϥΠ 4. ܦӦ՝୊ʹ׆༻͢ΔDatado g 5. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ
68. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ
69. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ
70. ϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾDatadog RUM(Real User Monitoring)ͷಋೖ
71. ϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾDatadog RUM(Real User Monitoring)ͷಋೖ ɹɾΑΓϢʔβʔʹ͍ۙҐஔ͔ΒͷϞχλϦϯά ɹɹɾϢʔβʔମݧʹӨڹ͕͋Δ΋ͷ
72. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ
73. ίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadog Continuous Profilerͷಋೖ
74. ίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadog Continuous Profilerͷಋೖ ɹɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɹɹɾ։ൃऀ͕ΑΓߴ඼࣭ͳίʔυΛॻ͚ΔΑ͏ʹ ɹɹɾطଘͷίʔυͷϦϑΝΫλϦϯάΛࢧԉ
75. ͞ΒͳΔαʔϏεվળʹ޲͚ͯ ɾϑϩϯτΤϯυ͔ΒͷαʔϏεվળ ɾίʔυϨϕϧͰͷϘτϧωοΫௐࠪ ɾDatadogͰͷSLI/SLO੔උ
76. DatadogͰͷSLI/SLO੔උ ɾDatadogͰͷSLI/SLO؅ཧ
77. DatadogͰͷSLI/SLO੔උ ɾDatadogͰͷSLI/SLO؅ཧ ɹɾαʔϏεͷ඼࣭ΛఆΊɺҡ࣋͢Δ ɹɹɾϨΠςϯγͳͲͷύϑΥʔϚϯεόδΣοτ ɹɹɾ5XXΤϥʔͳͲͷΤϥʔόδΣοτ
78. ·ͱΊ
79. ·ͱΊ ɾ1೥ؒͰΞΫηε਺5ഒʹαʔϏε͕੒௕ ɾͦΕʹ൐͍ϞχλϦϯάͷվળ͕ٸ຿ʹ ɾDatadog APMͷಋೖͰΦϒβʔόϏϦςΟΛ࣮ݱ ɾDatadog͕༷ʑͳ՝୊ղܾΛޙԡ͠ ɾࠓޙ΋ܧଓతͳαʔϏεվળʹDatadogΛ׆༻
80. ʲએ఻ʳ
81. ҰਓͰ΋ଟ͘ҰԁͰ΋ଟ͘ɺ ૝͍ͱ͓͕ۚΊ͙ΔੈքΛͭ͘Δɻ