Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ELKstackとAthenaの素敵な関係

February 10, 2017

 ELKstackとAthenaの素敵な関係

2017/02/10 JAWS-UG#9で発表した資料になります

February 10, 2017
Tweet

More Decks by 遊

Other Decks in Technology

Transcript

  1. & - , T U B D L ͱ "

    U I F O B ͷ  ૉ ఢ ͳ ؔ ܎ JAWS-UGԣ඿ #9 - Operational Excellence 2017/02/10 ੢ଜ ༡
  2. ˞ ຊ εϥ Π υ ͸  & - ,

    T U B D L ͱ " N B [ P O " U I F O B Λ  ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ ࿩ Ͱ ͸ ͋ Γ · ͤ Μ  ͢ ͍ · ͤ Μ ʂ  ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ࿩ ͋ Γ · ͤ Μ
  3. & - , T U B D L Ͱ ͷ

    ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ  " N B [ P O  " U I F O B ͕  औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ  ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩
  4. ͔ ͍ ͠ Ό ͍ Μ  ͤ ͍ ΂

    ͭ ɿ ͓ ͱ ͜  Ϩ ϕ ϧ ɿ    ) 1         . 1       ࣗ ݾ ঺ հ ࣾ ձ ਓ ྺ         d        ɹ % $ Ͱ ؂ ࢹ Φϖ Ϩ ʔ λ ʔ         d        ɹ ๭ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ         d  / 0 8 ɹ ɹ  ג ࣜ ձ ࣾ " 1 1 - * # 05  "8 4 ྺ    ໿  ೥ ͳ · ͑ ɹ ɹ ɿ ੢ ଜ ɹ ༡  ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO
  5. & - , T U B D L • elastic

    search + log stash + kibana = elk stack • L ͷ෦෼ʢσʔλ౤ೖʣ͸fluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ
 εϥΠυ಺Ͱ͸૯শͱͯ͠࢖ͬͯ·͢ɻ
  6. " N B [ P O  " U I

    F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)
 Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠
 ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ
  7. " N B [ P O  " U I

    F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)
 Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠
 ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢
  8. F M B T U J D T F B

    S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλ͸ϑΝΠϧͱͯ͠΋ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3΁backup • ྔͷଟ͍ϩάʢ1೔10GBҎ্ʣஷ·Δϩά͸3೔Ͱindexຖ࡟আ • Ҏલɺ௕ظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ஗͘ͳͬͯ͠·ͬͨ • ͦ΋ͦ΋ͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ΋༰ྔΛऔΒͳͯ͘͸͍͚ͳ͍ͨΊ௕ظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτ͸ͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨ΍ͭ΋Ұिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓ͸rundeck͞ΜͰຖ೔ࣗಈ࣮ߦ؅ཧɻS3ʹอଘɻ
  9. ࣄ ྫ  ໰ ͍ ߹ Θ ͤ ௐ ࠪ

    ͷ ҝ ɺ  ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍
  10. ͦ Ε ͳ Β L J C B O B

    Ͱ ݟ Ε · ͬ ͤ ʂ
  11. ͦ Ε ͳ Β L J C B O B

    Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
  12. T O B Q T I P U ΋ ͏

    ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
  13. \ EBUFz  UISFBElŠl  MFWFM*/'0  IBOEMFSlŠl  64&3@*%lŠl

     WFSTJPOlŠl  4&44*0/@*%lŠl  OB[P  NFTTBHFl63-IUUQ  1BSBNFUFS\dུd^  IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ
  14. $3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@  AEBUFAUJNFTUBNQ  AUISFBEATUSJOH  AMFWFMATUSJOH  AIBOEMFSATUSJOH 

    A64&3@*%ATUSJOH  AWFSTJPOATUSJOH  A4&44*0/@*%ATUSJOH  AOB[PATUSJOH  ANFTTBHFATUSJOH  AIPTUOBNFATUSJOH  308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4  TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞੒
  15. ࣄ ྫ   4  ͷ ΞΫ η ε

    ϩ ά ͕ ݟ ͨ ͍
  16. $3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU  #VDLFU0XOFSTUSJOH  #VDLFUTUSJOH  3FRVFTU%BUF5JNFTUSJOH  3FNPUF*1TUSJOH 

    3FRVFTUFSTUSJOH  3FRVFTU*%TUSJOH  0QFSBUJPOTUSJOH  ,FZTUSJOH  3FRVFTU63*@PQFSBUJPOTUSJOH  3FRVFTU63*@LFZTUSJOH  3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH  )551TUBUVTTUSJOH  &SSPS$PEFTUSJOH  #ZUFT4FOUTUSJOH  0CKFDU4J[FTUSJOH  5PUBM5JNFTUSJOH  5VSO"SPVOE5JNFTUSJOH  3FGFSSFSTUSJOH  6TFS"HFOUTUSJOH  7FSTJPO*ETUSJOH  308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4  TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?>  <?> aa<  aa> <?>  <?>  <?>  <?>  <?> aaa <?>  <?>  c<?> aaa c<>  <?>  <?>  <?>  <?>  <?>  <?>  a<?a> a  <?>  -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞੒
  17.  SFNPUFJQSFRVFTUFS DPVOU   BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS    

    BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS    BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS  dུd  BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ݁Ռ
  18. · ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ
 AthenaͰݟΔͨΊʹ~ΛS3ʹ͸͘Α͏ʹͯ͠…ͱ͔Ͱ͸ͳ͘ɺ
 ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹର৅ͱݴ͑Δɻ
 AWSͷαʔϏε(S3 /

    CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩά΋ɺ ௐ͚ࠪͩͳΒ͜ΕͰे෼ɻ • elasticsearch͔Β࡟আͯ͠΋S3ʹϩάϑΝΠϧ͋Ε͹ௐࠪ͸OK
 ௕ظؒݟΕΔ༷ʹ͢ΔͨΊʹ
 αʔόεϖοΫΛ্͓͛ͯ͘ඞཁ͸ແ͘ͳͬͨɻ
 ˠ ຊ౰ʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ
 ɹ→ snapshot͸ͪΌΜͱऔΖ͏+͋Δఔ౓ظؒ࣋ͬͯͯ΋Α͔ͬͨ(൓ল)
  19. & - , T U B D L Ͱ ͷ

    ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ  " N B [ P O  " U I F O B ͕  औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ  ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩ Ͱ ͠ ͨ