Share
Embed
Transcript
48 slides
-
Japan Datadog User Group Meetup#1 ABEJA, Inc. Shogo Muranushi Suggested
Topicͷ࣭ʹՄೳͳݶΓ͑ͯΈͨ -
@rwle1212 3 ໊લ ɿ ଜओ ޛʢΉΒ͵͠ ͠ΐ͏͝ʣ ձࣾ ɿ ABEJA,
Inc. ۀ ɿΠϯϑϥΤϯδχΞ -> ϓϩμΫτΦʔφʔ -> SRE -
@rwle1212 11 ࣦഊྫ1: ݁ہCloudWatchΛݟ͍ͯͨ ALB -> Container -> EC2 ->
RDSͱ͍͏ϦΫΤετͷྲྀΕͰμογϡϘʔυΛ࡞ͬ ͯΈͨɻ μογϡϘʔυͷେ͖ܾ͞·͍ͬͯΔɺͭάϥϑ͕ଟ͗ͯ͢ෳࡶʹͳΔͷ ͰɺεϚʔτʹগͳΊʹάϥϑΛஔͨ͠ɻ ͔͠͠ো࣌ʹใ͕Γͳ͍ͷͰɺ݁ہCloudWatchΛோΊ͍ͯͨ͜ͱʹؾ͍ ͨɻ ͔ͦ͜ΒඞཁͳάϥϑΛ͍ͯ͘͠ͱͲΜͲΜෳࡶʹͳΓɺଞͷϝϯόʔʮԿΛ Ͳ͏ݟͨΒྑ͍ͷ͔Θ͔Βͳ͍ʯঢ়ଶʹͳͬͨɻ ݁ہCloudWatchΛோΊ͍ͯͨ͜ͱʹؾ͍ͨɻ -
@rwle1212 14 ղܾྫ NotebookΛͬͯΈͨ • Jupyter Notebook Likeͳ͍ํ͕Ͱ͖ ͯɺίϝϯτͱάϥϑΛॻ͚Δ •
ಛఆͷোʹରͯ͠ௐͨ͜ͱɺਪଌ ͳͲͷίϝϯτΛॻ͖ͳ͕Βه͕ ͤΔ • ϩάͷάϥϑԽ༰қ • Issueʹը૾ΛషΓ͚ΔΑΓεϜʔ ζʹ·ͱΊΒΕΔͷͰڞ༗ָ͕͘͢͝ -
@rwle1212 19 ՝ۚཧʹؔ͢Δେٕখٕ ಛʹେٕແ͠ɻখωλ͕͋Δͱͨ͠Β • Multi OrgڥBilling͕ू͞Εͯ͠·͏ɻʮLast Month’s UsageʯͰར༻ྔʢྉ ۚͰͳ͍ʣ͕Orgຖʹग़ྗ͞ΕΔͷͰ҈͍ͯ͠ΔɻӦۀ͞Μʹݴ͑Billing
͚ͯ͘ΕΔ͔ʁखؒͳͷͰͦ͜·Ͱͬͯͳ͍ • AWSΠϯςάϨʔγϣϯͰͨ͘͞ΜͷαʔϏεͷϝτϦΫεΛऔΕΔΑ͏ʹ͢Δ ͱɺCloudWatch ͷ GetMetricData API ͕ΊͪΌͪ͘Όݺͼग़͞ΕΔͷͰ՝͕ۚ େ͖͘૿͑Δɻ·ͣඞཁͳαʔϏε͚ͩߜͬͯऔΓࠐΉͷ͕٢ɻ SQSͰࢮͨ͠ܦݧ༗Γ -
@rwle1212 24 ʲཁʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔ • Datadog͔ΒPrometheusΓ͑ • ݩʑDatadogΛ͍ͬͯͨɻ$18/host͕ߴ͔ͬͨ͠ɺKubernetesͷࢹͷσϑΝΫ τʹͳΓͭͭ͋ΔPrometheusΛࢼ͢తؚΊ͍ͯ࢝Ίͨɻ • ܦաɻPrometheusࣗମͷো͕νϥϗϥ
• OSSͰใগͳ͍߹ௐࠪʹ͕͔͔࣌ؒΔɻଟ͍࣌िʹ1,2ճ΄Ͳ͕ൃੜ ͠1,2ਓ͕࣌ؒ/ि΄ͲऔΒΕ͍ͯͨ • ελʔτΞοϓਓ͕͍ͳ͍ɻຊۀͷ։ൃʹूத͍ͨ͠ • πʔϧͷ͓कΓʹ͕࣌ؒऔΒΕ͍ͯͨɻPrometheusͷӡ༻Λ֎෦ʹҕୗ͍ͨ͠ɻ ӡ༻Λ֎෦ʹҕୗ͢Δͱ͍͘ΒʹͳΔ͔͔Βͳ͍͚ͲɺࠓͷنΛDatadogʹม ͑Δͱ10ສ/݄͘Β͍ͩͬͨɻɾɾDatadogͰΑ͘Ͷʁͱࢥ͍࢝ΊΔ -
@rwle1212 25 ʲཁʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔ • ϩάͷऩूͲ͏͢Δ • CloudWatch LogsΛ͍͕ͬͯͨݕࡧੜօແɻʢ࣌ʣPrometheusϩάऩू ͢Δػೳͳ͍ɻσϑΝΫτతʹ ElasticSearch
+ Kibana ͩΖ͏ • ͔͠͠ɺͦͷ࣌Ͱϩάྔ͕5ԯ/݄ɻঃʑʹରͷΞϓϦνʔϜΛ૿ͦ͏ͱࢥ ͏ͱϲ݄Ͱഒʑʹ૿͍͑ͯ͘͜ͱ͕ఆ͞Ε͍ͯͨ • ͦΕΛElasticSearchͰड͚ͳ͕Βӡ༻͢Δ͜ͱΛఆ͢Δͱɺख͕औΒΕͦ͏ײ͕ ͳ͔ͬͨ • DatadogϩάͷαʔϏεΛఏڙ࢝͠Ί͍ͯͨͷͰͬͯΈΔ͜ͱʹ -
@rwle1212 26 ʲཁʳͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔ • ϩάͲ͏͢Δ͔ͷ݁ • Facets ͕ڧྗͰɺಋೖָ͕ɺίετίϯτϩʔϧ͘͢͠ɺԿ͔͋Εશϩά Λ͑Δɻͦͷ্Ͱɺϩάऩू͢Δ্ͰΠϯϑϥΛؾʹ͠ͳͯ͘ྑ͍ͷ͕ؾʹೖ ΓɺঃʑʹϩάΛ
Datadog Logs ʹدͤ·ͨ͠ • ͦͯ͠ɺϞχλϦϯά Promethes ͔Β Datadog Ҡߦ • ͳͥPrometheusΛࣙΊͯDatadogΛ࠾༻ͨ͠ͷ͔ɺͷ·ͱΊ • ϩάϞχλϦϯά ͦΕ͕ࣄۀͷίΞͰ͋ΕͪΌΜͱߏங͠ӡ༻͢Δ͜ͱࣗ ମԀΘͳ͍ͷͰ͕͢ɺίΞͰͳ͍ͳΒ࣌ؒΛׂ͖ͨ͘ͳ͍ -
@rwle1212 New RelicɺMackerel ྆ํͱݕূ͍ͯ͠ͳ͍ͷͰൺֱʹͳΒͳ͍Ͱ͕͢ɺݕূ͠ͳ͔ͬͨ ཧ༝ͷҰͭͱͯ͠ɺDatadogͰʮMonitoring Modern Infrastructureʯͱ͍͏ه ࣄ͕ެ։͞Ε͍ͯ·͢ɻ ͜͜ʹॻ͔Ε͍ͯΔ༰͕ͱͯཧʹ͔ͳ͍ͬͯͨͷͰɺ͜ͷΑ͏ͳࢥʹ ΑͬͯαʔϏε͕ઃܭ͞ΕࠓޙαʔϏεΛՃ͍ͯ͘͠ΜͩΖ͏ͳͱࢥ͍ɺ
࠾༻ͨ͠എܠ͋Γ·͢ɻ 28 ଞͷࢹSaaSʢ͕ࣗΔൣғͰZabbixɾ PrometheusɾNew RelicɾMackerelʣͱͷൺֱ -
@rwle1212 30 Datadogͷڭ׆ಈ ʢͲͷׂͷΤϯδχΞ·Ͱ͍ͬͯΔ͔ʁʣ • ޮՌ͋ͬͨ͜ͱ • Dev/Stg/ProdͷશΞϓϦͷϩάΛશ෦ूͨ͠ɻ1ͭͷWebαΠτΛݟʹ ͍͘͜ͱͰશ෦ͷϩά͕ݟΕΔঢ়ଶʹͨ͠ɻࠓ·ͰڥຖʹݟΔॴ͕ ͔ΕͯͨΓɺग़ͯͨΓग़ͯͳ͔ͬͨΓ
• ͔ͦ͜ΒঃʑʹDatadogͷ͍ํΛೝࣝ͠ɺMonitorsNotebooksɺAPM ʹϝϯόʔͷڵຯ͕͕ͬͯߦͬͨ -
@rwle1212 31 Datadogͷڭ׆ಈ ʢͲͷׂͷΤϯδχΞ·Ͱ͍ͬͯΔ͔ʁʣ • ޮՌ͋·Γͳ͔ͬͨ͜ͱ • Datadogͷ͍ํษڧձɻʮʔɻ͜Μͳ͜ͱͰ͖ΔΜͩʯͱ͍͏͚ͩͰखΛ ಈ͔͞ͳ͍ͷͰฉ͍ͯΕΔ •
ϩάɺϝτϦΫεΛՄೳͳݶΓूͯ͠ɺඞཁʹۦΒΕͯࣗͰௐͯ ͯ͠Β͏ํ͕͕֮͑ྑͦ͞͏ͳؾ͕ͨ͠ɻݴ͏ͯ͘͠ͳ͍ͷͰ • ͋Δఔͨ͠Βʮ͓Εͷ͔Μ͕͍͖͑ͨ͞ΐ͏ͷDatadogڞ༗ձʯΛ֤ϝ ϯόʔͰখωλతʹΔͷѱ͘ͳ͔ͬͨ -
@rwle1212 32 Datadogͷڭ׆ಈ ʢͲͷׂͷΤϯδχΞ·Ͱ͍ͬͯΔ͔ʁʣ • ͲͷׂͷΤϯδχΞ·Ͱ͍ͬͯΔ͔ • ͍ͬͯΔͷΠϯϑϥɺόοΫΤϯυ • ϑϩϯτΤϯυͷE2Eࢹܥগ͠ߴ͍ͷͰ͍ͬͯͳ͍
• Real User MonitoringɺSynthetic Monitoring (Browser) -
@rwle1212 34 Ξϥʔτͷ௨Ϩϕϧཧ ʢฐࣾwarn / error / fatal ͳͲʣ •
WarnͱErrorͷΈ • ʮαʔϏεʹӨڹ͕Ͱ͍ͯΔϞϊɺͰ͍ͯΔՄೳੑ͕͋ΔϞϊʯΛΞϥʔτ ͱͯ͠ઃఆ͍ͯ͠Δ • ྫʣCPUϝϞϦͷ༻Λࢹͱ͔͍ͯ͠ͳ͍ɻͦΕΒͷӨڹʹΑͬ ͯALBͰ5xx͕සൃ͢ΕΞϥʔτ͕ඈͿΑ͏ʹ͍ͯ͠ΔɻDynamoDBͷ εϩοτϦϯάͱ͔ -
@rwle1212 38 Graph by Logs • ϩάΛάϥϑԽͰ͖Δ • facetsʹొͨ͠ΛGroup Byʹࢦఆͨ͠Γɺ
CountʹೖΕΔ͜ͱͰ͖Δ • ࠨਤͷྫͰɺOrganization NameຖͷϦΫΤ ετฏۉԠ࣌ؒͳͲΛදࣔɻίϯςφID ຖʹݟͯภΓ͕ແ͍͔ݟͨΓͯ͠Δ • γεςϜܥ͚ͩͰͳ͘ΞϓϦͷར༻ঢ়گੳ ʹ༻ՄೳͳͷͰɺΞϓϦଆͰ͖ͳΛग़ ͤ৭ʑͳ؍ͰੳͰ͖Δ • ᮢΛઃ͚ͯΞϥʔτൃ๒Ͱ͖ΔͷͰ1ͭͷγ εςϜͰϝτϦΫεɾϩάͷࢹΛཧͰ͖Δ -
@rwle1212 39 Graph by Logs • େ͖͍มߋΛͨ࣌͠εςʔλείʔυͷׂ߹ Λ֬ೝ • ઃఆํ๏
• Group By: @http.status_code • Display: Areas • Stacked as: % of overall count: * -
@rwle1212 40 Pipelines • JSON͡Όͳ͍ϩάΛύʔεͨ͠ΓɺDatadog ্Ͱʮ͜ͷϑΟʔϧυεςʔλεѻ͍͢Δʯ ͱ͍͏Α͏ͳϚοϐϯάΛߦ͏ • NginxRedisͳͲҰൠతͳϛυϧΣΞͷϩά IntegrationΛೖΕΕউखʹύʔεͯ͘͠Ε
ΔͷͰࣗͰgrokॻ͘ඞཁ͕ͳ͘ศར • ԼਤͷΑ͏ʹܭࢉࣜΛೖΕͨΓɺผ໊ʹม͑ͨ Γ͢Δ͜ͱͰ͖Δ -
@rwle1212 41 Indexes • ϩάऩूͷআ֎ϑΟϧλ͕ॻ͚ΔɻIndexed LogsͰ՝ۚ͞Εͳ͘ͳΔ • 10%͚ͩऔಘ͢Δͱ͍͏͜ͱՄೳɻΫΤϦͰ ͔ͳΓߜΓࠐΉ͜ͱ͕Ͱ͖Δ •
ϩάΛૹ৴͢ΔAgent/ClientଆͰίϯτϩʔϧ ͢Δ͜ͱՄೳ͕ͩɺઃఆϑΝΠϧͷ͕ ໘ͳͷͰ͍ͬͨΜશ෦Datadogʹ͛ͯɺ IndexsͰϩάΛམͱ͢ͱ͍͏͍ํ͕Φεε ϝɻʢ҆Ձ͚ͩͲIngested Logsͷ՝ۚൃੜ͢ ΔͷͰҙʣ -
@rwle1212 42 Archives • Datadogͷϩάܖظؒʢ7ͱ͔15ͱ ͔ʣ͔͠ݟΕͳ͍ • աڈϩάظؒอଘ͍ͨ͠߹ArchivesͰ S3GCSʹஷΊΔ͜ͱ͕Ͱ͖ΔͷͰɺAthena BigQueryͰݟΔ͜ͱͰ͖Δ
• Fluentdͱ͔ͰS3ʹૹΔ͜ͱͰ͖Δ͚Ͳɺ Datadogͷ߹ͱΓ͋͑ͣऩू͢ΔઃఆϑΝ ΠϧΛૹ৴͢Δ͚ͩͰྑ͍ͷͰɺઃఆϑΝΠϧ ͷΒ·͖ߋ৽͕ͳָ͍ -
@rwle1212 44 datadog logsʹ͍ͭͯ༻ײࣄྫ ΞϓϦͷϩάΛDatadog LogsʹೖΕ͍ͯΔͷͰɺ΄΅ຖ౷ܭతʹݟͯҟৗ͕ແ͞ ͦ͏͔βοͱݟ͍ͯΔɻ ඞཁ͋ΒਂງΓ͢ΔͨΊʹ Graph Ͱ৭ʑͳ؍Ͱੳͨ͠Γɺه࣌͢ɺܦ
ա؍͢ΔNotebooksΛ࡞ͬͯܧଓతʹ༷ࢠΛݟͨΓ͍ͯ͠Δɻ -
@rwle1212 47 LogsઃఆͷϕετϓϥΫςΟε • ͱΓ͋͑ͣ • શ෦ͷϩάΛ1Օॴʹू͢Δɻѹతޮ্͕Δ • Pipeline •
ύʔεͰ͖Δ͚ͲɺͰ͖ΔݶΓJSONͰૹ৴ͯࣗ͠ಈతʹύʔεͯ͠Β͏ • εςʔλεͱ͔Ϛοϐϯάͯ͠ɺϏϡʔϫʔͷ੨ɾԫɾ৭දࣔͰҰྎવʹ͢Δ • Indexes • ίετݮ͢ΔͨΊʹݟͳ͍ϩάআ֎͢Δ • Archives • ೦ͷͨΊաڈϩάS3/GCSʹอଘ͢Δ • S3/GCSଆͰҰఆظؒޙʹࣗಈআ͢ΔΑ͏ʹϥΠϑαΠΫϧΛೖΕΔ