Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring at AbemaTV

nghialv
October 21, 2017

Monitoring at AbemaTV

nghialv

October 21, 2017
Tweet

More Decks by nghialv

Other Decks in Programming

Transcript

  1. "CPVUNF w -&7"//()*" ΪΞʣ w ϕτφϜग़਎ w (JUIVC!OHIJBMW w 5XJUUFS!OHIJBMW

    w ೥৽ଔͱͯ͠$ZCFS"HFOUʹೖࣾ w "NFCB 'SFTIੜ์ૹʂͰJ04ΤϯδχΞ w ͔Β"CFNB57ͷόοΫΤϯυΤϯδχΞ
  2. 4UBDLESJWFSͷΈͷ࣌ظ w ຊ։ہ͔Β൒೥͙Β͍ w ($&Πϯελϯε w ϚγϯϝτϦΫε $16 .FNPSZ /FUXPSLʜ

     w 7.্ʹಈ͍͍ͯΔ.POHP%# 3FEJTͷϝτϦΫε w ($1αʔϏε w 1VC4VC 4UPSBHF -PBE#BMBODJOH ʜ 4UBDLESJWFS"HFOU .POHP%# 7. NPOHPECWN
  3. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF
  4. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF

    w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  5. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  6. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  7. Կނ1SPNFUIFVT 1SPNFUIFVT BQJTFSWFS 1PE 1PE TFSWJDFEJTDPWFSZ TDSBQFNFUSJDT .BTUFSOPEF 8PSLFSOPEF 1PE

    1PE w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w "MFSUNBOBHFS͕ศར ˔
  8. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ ύϑΥʔϚϯεͷ໘ JOTUBODFLTBNQMFTTFDPOEɹ 4PVOE$MPVEͷ࣮੷ UJNFTFSJFTQFSUBSHFU TTDBQFJOUFSWBM JOTUBODF͕ UBSHFUT·ͰରԠͰ͖Δ
  9. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ w QSPNFUIFVTDMJFOU@HPMBOH w $PVOUFS w (BVHF w 4VNNBSZ w )JTUPHSBN w طଘͷ&YQPSUFS΋ଟ͍ w OPEFFYQPSUFS w NPOHPEC SFEJT NZTRM ʜ
  10. Կނ1SPNFUIFVT w ,VCFSOFUFTͱ਌࿨ੑ͕ߴ͍ w ੑೳ͕ྑ͍ w 1SPN2-͕࢖͍΍͍͢ w &YQPSUFSΛॻ͖΍͍͢ w

    "MFSUNBOBHFS͕ศར ˔ w 1SPNFUIFVT͕ൃੜͨ͠Ξϥʔτͷ؅ཧ w ॏෳ͍ͯ͠ΔΞϥʔτͷղআ w ΞϥʔτͷάϧʔϓԽ w ௨஌ͷҰ࣌తࢭͳఀࢭ w ৭ʑͳ௨৴ઌͷରԠ w 4MBDL &NBJM )5518FCIPPL
  11. 1SPNFUIFVTͷಋೖͷॳ࣌ظ w Ϋϥελ಺ʹͭͷ1SPNFUIFVTαʔό w ΞϓϦέʔγϣϯͷϝτϦΫεͷΈ w 1SPNFUIFVTͷ$POpHΛ
 $POpH.BQʹॻ͘ w λʔήοταʔϏεͷTFSWJDFZBNMʹ

    BOOPUBUJPO௥Ճ (SBGBOB "MFSUNBOBHFS 1SPNFUIFVT (BUFXBZ &YQPSUFS .FEJBQSPYZ &YQPSUFS 6TFS &YQPSUFS $PNNFOU &YQPSUFS 8BUDINBO &YQPSUFS ʜ $POpH.BQ FUDE apiVersion: v1 kind: Service metadata: name: abema-demo annotations: prometheus.io/scrape: "true" prometheus.io/port: "12345" prometheus.io/path: “/metrics"
  12. ϝτϦΫε w H31$ w SFRVFTU FSSPS EVSBUJPO w IUUQ w

    SFRVFTU FSSPS EVSBUJPO w HPMBOH w HPSPVUJOF IFBQ TUBDL HBSCBHFDPMMFDUJPO w QSPDFTT w DQVUJNF NFNPSZTJ[F
  13. 4MPXRVFSZ໰୊ w (SBGBOBଆͰෳࡶͳ2VFSZΛॻ͘ͱɺ
 ଴͕ͪ࣌ؒ௕͍ w 1SPNFUIFVTͰ3FDPSEJOH3VMFTΛ௥Ճ w ࣄલܭࢉΛߦ͍ɺ
 ৽͍͠UJNFTFSJFTͱͯ͠อଘ 100

    * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) (SBGBOB service:abema_http_request_5xx_percent:rate2m = 100 * sum by (service) ( rate(abema_http_requests_total{status=~"^5..$"}[2m]) ) / sum by (service) ( rate(abema_http_requests_total[2m]) ) 1SPNFUIFVT service:abema_http_request_5xx_percent:rate2m (SBGBOB
  14. /PEFϝτϦΫε w /PEFFYQPSUFS w $16 .FNPSZ %JTL /FUXPSL ʜ w

    ,TOPEF w %BFNPO4FUͱ֤ͯ͠OPEFʹ
 ҰͭͷQPEΛಈ͔͢ w ($&OPEF w 1BDLFSͰNBDIJOFJNBHFͷ
 Ϗϧυ࣌ʹೖΕΔ OPEFEBTICPBSE
  15. $$6ϝτϦΫε w ಉ࣌઀ଓ਺ͷϝτϦΫε w ֤$IBOOFM 04 4USFBNJOH1SPUPDPM 
 3FTPMVUJPO 1MBZFS

    ʜ w ΞΫηεϩά͔ΒϦΞϧλΠϜܭࢉ w )ZQFS-PH-PHΞϧΰϦζϜ w )PSJ[POUBM1PE"VUPTDBMFSͰ
 DDVDPMMFDUPSΛࣗಈεέʔϦϯά .FEJBQSPYZ 1VC4VC DDVDPMMFDUPS DDVFYQPSUFS DDVSFEJT 1SPNFUIFVT DDVDPMMFDUPS .FEJBQSPYZ MPH #JH2VFSZ
  16. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVT
  17. 1SPNFUIFVTTIBSEJOH w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔

    OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF
  18. 1SPNFUIFVTTIBSEJOH TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT TFSWJDF$ NFUSJDT TFSWJDF% NFUSJDT QSPNFUIFVT

    QSPNFUIFVT w ϝτϦΫελΠϓ w ϝτϦΫεछྨͰ1SPNFUIFVTΛ෼͚Δ w λʔήοτ w औಘର৅Ͱ1SPNFUIFVTΛ෼͚Δ ˔
  19. )"1SPNFUIFVT w ܦҢ w 1SPNFUIFVTͷϝτϦΫε͔ΒNJDSPTFSWJDFTΛࣗಈεέʔϦϯά w ಉ͡$POpHɺಉ͡λʔήοτϦετͰෳ਺1SPNFUIFVTΛߏங w NFNPSZ΍EJTLͷσʔλ͸΄΅ಉ͡ w

    σʔλͷࠩ෼ͷٵऩ w Ξϥʔτ w ීஈͷΞϥʔτ͸Ұఆͷ࣌ؒͰ൑ఆ w (SBGBOB w εςΟοΩʔηογϣϯ TFSWJDF" NFUSJDT TFSWJDF# NFUSJDT QSPNFUIFVT QSPNFUIFVT QSPNFUIFVT
  20. ӡ༻ w $POpHϑΝΠϧΛॻ͘ͷ͕େม w TIBSEJOHͰྔ͕ଟ͘ͳΓɺઃఆ͕ෳࡶ w SFMBCFMJOH͕෼͔Γʹ͍͘ w ৽ϝτϦΫεͷ௥Ճ΍EBTICPBSETͷमਖ਼͸
 νʔϜϝϯόʔͦΕͧΕͰ΍Γ͍ͨ

    w DPOpH΍EBTICPBSEΛߋ৽͍ͨ͠৔߹ɺ
 Ϧελʔτ͕ඞཁ OPEF FYQPSUFS OPEFNFUSJDT TFSWJDF" HPNFUSJDT IUUQNFUSJDT TFSWJDF# HPNFUSJDT HSQDNFUSJDT QSPNFUIFVTHP QSPNFUIFVTHSQD QSPNFUIFVTIUUQ QSPNFUIFVTOPEF
  21. 1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ
 TFSWJDFNPOJUPSΛهड़
 w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ

    EBTICPBSEΛ؂ࢹɺϦϩʔυ TFSWJDF" MBCFMT UZQFIUUQ TFSWJDF# MBCFMT UZQFIUUQ QSPNFUIFVTIUUQ TFSWJDFNPOJUPS QSPNUIFVTIUUQ TFMFDUPS NBUDI-BCFMT UZQFIUUQ $POpH 8BUDI (FOFSBUF QSPNFUIFVTPQFSBUPS
  22. 1SPNFUIFVT0QFSBUPS w $POpHϑΝΠϧΛࣗಈੜ੒ w DPOpHͷ୅ΘΓʹɺ
 TFSWJDFNPOJUPSΛهड़
 w ϗοτϦϩʔυ w 4JEFDBSίϯςφʔͰDPOpHɺ

    EBTICPBSEΛ؂ࢹɺϦϩʔυ SFMPBEFS QSPNFUIFVTIUUQ 6QEBUF 3FMPBE 8BUDI 1SPNFUIFVT1PE 3FMPBE $POpH QSPNFUIFVTPQFSBUPS
  23. "MFSUNBOHFS w /PEF@)JHI$166TBHF w /PEF@)JHI.FNPSZ6TBHF w 1PE@)JHI.FNPSZ6TBHF w )UUQ3FRVFTU@)JHI&SSPS3BUF w

    )UUQ3FRVFTU@)JHI-BUFODZ w (31$@)JHI&SSPS3BUF w 3FEJT@)JHI&SSP3BUF w ʜ "-&35(31$@)JHI&SSPS3BUF *'NFUIPEBCFNB@HSQD@SFRVFTU@JOUFSOBM@FSSPS@QFSDFOUSBUFN '03N -"#&-4\ TFWFSJUZXBSOJOH ^ "//05"5*0/4\ EFTDSJQUJPO\\MBCFMTTFSWJDF^^@\\MBCFMTNFUIPE^^@IBTBOFSSPSSBUF
 PG\\QSJOUGaGaWBMVF^^GPSN  ^ HSQDSVMFT TMBDL΁௨஌
  24. ໰୊ w LTͷඞཁͳZBNMϑΝΠϧͷॏෳ w ؀ڥ EFW QSE  w QSPNFUIFVTTIBSE

    w BQQMZॱ൪ w QFSTJTUFOUEJTLDPOpHNBQTUBUFGVMTFUTFSWJDF w EFQMPZNFOUҎ֎ͷSPMMCBDL %3: %0/`53&1&"5:0634&-'
  25. )FMN w ,VCFSOFUFTQBDLBHFNBOBHFS w )FMN$IBSUΛ࡞੒ w ඞཁͳϑΝΠϧͷςϯϓϨʔτͷఆٛ w )FMNͰΠϯετϧ w

    WBMVFTZBNMϑΝΠϧͰม਺Λઃఆ helm install --name prometheus-grpc -f dev-values.yaml ./charts/prometheus
  26. 1SPNWJ[ w 1SPNWJ[ w 1SPNFUIFVT͔ΒͷσʔλΛऔಘɺ
 άϥϑσʔλΛੜ੒ w (SBQI"1*Λఏڙ w 1SPNWJ[GSPOU

    w /FUqJYͷWJ[DFSBMΛج͍ͮͯɺ
 τϥϑΟοΫͷϨϯμϦϯά promviz Retrieval API Cache Storage HDD/SSD promviz-front Prometheus Prometheus Prometheus