Upgrade to Pro — share decks privately, control downloads, hide ads and more …

mockmockの大量のログをいい感じに捌きたい

 mockmockの大量のログをいい感じに捌きたい

Fukuoka.dev #1

Keita Mohri

July 16, 2019
Tweet

More Decks by Keita Mohri

Other Decks in Technology

Transcript

  1.   ‣ 'VTJD$P -UE ‣ ൃҊऀϓϩμΫτΦʔφʔ ‣ *P5ܥͷडୗҊ݅΋ ‣

    "84 403"$0. (P 3VCZ ‣ ޷͖ͳ"84αʔϏε,JOFTJT ໟརܒଠʢ΋͏Γ͚͍ͨʣ
  2. ϩάॲཧػߏ   PublicSubnet PrivateSubnet PrivateSubnet S3 Bucket (Lot) S3

    Bucket (Binary) Web Web Worker Worker Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞
  3.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ PrivateSubnet Mock Mock Mock Mock Mock ϩάॲཧػߏ
  4.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ
  5.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ
  6.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ
  7.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ
  8. ϩάॲཧػߏ   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket

    (Binary) Web Web Worker Worker ίϯιʔϧͰ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock mockىಈ Mock Mock [ mock ] Go੡όΠφϦ͔Βཱͯͨ Ծ૝σόΠε໾ϓϩηε [ tower ] Go੡όΠφϦ͔Βཱͯͨ mock؅ཧϓϩηε
  9.   PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)

    Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ
  10. ‣ ֤NPDLͷৄࡉͳϩάΛऔΓ͍ͨ  ૹ৴σʔλૹ৴ઌϓϩτίϧFUD  ϩάͷαΠζ͸ૹ৴σʔλαΠζʹΑͬͯେ͖͘มಈ w ,#ҎԼd਺े,# ‣ ֤NPDLͷঢ়ଶΛϦΞϧλΠϜʹߋ৽͍ͨ͠

     3%4΁ͷߋ৽ ‣ ࠷େNPDLՔಇ਺͸࣮੷Ͱສ୆  Քಇ਺ͷͱ͖΋͋Ε͹͍͖ͳΓສʹͳΔ͜ͱ΋͋Δ ‣ Քಇ਺ʹΑͬͯඅ༻΋εέʔϧ͢ΔΑ͏ʹ͍ͨ͠  NPDLNPDLࣗମ΋ैྔ՝ۚ ཁ݅  
  11. ͜͏ͳΓ·ͨ͠   S3 Bucket Lambda PrivateSubnet Worker Worker S3

    Bucket ૹ৴ຖʹϩάΛ ॻ͖ग़͢ 1෼ຖʹશmockͷ ϩάΛूܭ S3΁ͷputͰ Lambda͕ൃՐ mockͷঢ়ଶΛ HTTPͰ఻͑Δ mockຖʹϩάΛ όϥͯ͠อଘ
  12. ‣   NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘  ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕࠷௿ݶ ‣   ϩάͷ఻ୡʹ΍ͨΒ͕͔͔࣌ؒΔ

     ϩάͷ಺༰͕ඞཁ࠷௿ݶ  ҟৗܥ·Ͱߟ͑Δͱϩάͷू໿ॲཧ͕ෳࡶʹʜ  XPSLFSαʔόʔʹNPDLͷঢ়ଶΛ఻͑Δͱ͜Ζ͕ಉظతࣦഊ࣌ͷ࠶ૹॲཧͱ͔ʜ  ͳΜ͔͔ͬ͜ѱ͍ʜ ݁Ռ  
  13. ‣ ैདྷͷ%ZOBNP%#Ͱ͸ɺΩϟύγςΟϢχοτ ಡΈॻ͖ੑೳ Λ
 ͋Β͔͡Ίઃఆ͓ͯ͘͠ඞཁ͕͋ͬͨ  8$6 8SJUF$BQBDJUZ6OJU  

    3$6 3FBE$BQBDJUZ6OJU  ‣ ΩϟύγςΟϢχοτͷΦʔτεέʔϦϯά͸ઃఆͰ͖͕ͨɺ
 ͙͢ʹεέʔϧ͢ΔΘ͚Ͱ͸ͳ͍ ‣ 0O%FNBOEϞʔυͰ͸ɺ࢖ͬͨΩϟύγςΟϢχοτʹରͯ͠
 ैྔ՝ۚͯ͘͠ΕΔʂ  ͋Β͔͡ΊΩϟύγςΟϢχοτΛઃఆ͓ͯ͘͠ඞཁ͕ͳ͍ʂ %ZOBNP%#0O%FNBOEര஀ʂ  
  14. ͜͏ͳΓ·ͨ͠   DynamoDB (On-Demand) Lambda SNS SQS ֤mock͕௚઀ DynamoDBʹϩάΛ

    ॻ͖ࠐΉ PrivateSubnet Worker Worker DynamoDB Streamsʹ ΑͬͯLambdaΛൃՐ ඇಉظʹσʔλΛ ߋ৽Ͱ͖ΔΑ͏ʹ
  15. ‣   NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘  ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ  ৄࡉϩά͕ݟΕΔ  ϩάॲཧػߏ͕ൺֱత؆୯

    ‣   ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕େ͖͘ͳͬͨʜ ‣   %ZOBNP%#0O%FNBOE͸ΰϦΰϦ࢖͏ͱߴ͍ʜ ݁Ռ  
  16. ͜͏ͳΓ·ͨ͠   DynamoDB (Provisioned) Lambda SNS SQS DynamoDBʹ͸εςʔλεߋ৽༻ͷϩά EFSʹ͸׬શͳϩάΛॻ͖ࠐΉ

    PrivateSubnet Worker Worker DynamoDB Streamsʹ ΑͬͯLambdaΛൃՐ EFS ΩϟύγςΟ͸ mockىಈ࣌ʹௐ੔ PublicSubnet Web Web ඇಉظʹσʔλΛ ߋ৽Ͱ͖ΔΑ͏ʹ
  17. ‣   NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘  ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ  ৄࡉϩά͕ݟΕΔ  ϩάॲཧػߏ͕ൺֱత؆୯

     %ZOBNP%#ͷඅ༻͕࠷దԽ͞Εͨ ‣   ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕΋ͬͱେ͖͘ͳͬͨʜ  Ұఆ࣌ؒܦաޙʹ&'4͔ΒσʔλΛ࡟আ͢Δͷ͕खؒ ‣   &'4΋*0Λ૿΍ͦ͏ͱ͢ΔͱͦΕͳΓʹඅ༻͕ʜ ݁Ռ  
  18. ‣ NPDLαʔόʔʹԿ͔ΛΠϯετʔϧ͢ΔͷΛڐ༰͠Α͏  શମతʹ͜ͳΕ͖ͯͨͷͰɺࠓͳΒ͍͚Δؾ͕͢Δ ‣ ࠓͦ͜,JOFTJTͱ޲͖߹͓͏  γϟʔυͷѻ͍͕େมͳͷͰࠓ·Ͱආ͚͖ͯͨ ‣ ૣ͘΋&'4͔Β୤٫͍ͨ͠ؾ࣋ͪ

     ωοτϫʔΫଳҬͷϘτϧωοΫʹͳΓ͕ͪ  ͓ۚͰղܾͰ͖Δ͚Ͳ͓ۚ͸େࣄ  ϩάͷͨΊʹNPDL͔ΒϲॴʹσʔλΛૹ͍ͬͯΔͷ΋ΠϠͩ ୈ̐ੈ୅ํ਑  
  19. ‣ ,JOFTJT%BUB4USFBNTʹσʔλΛ౤͛ࠐΉϥΠϒϥϦ ‣ ͋Δఔ౓σʔλΛ·ͱΊͯૹͬͯ͘ΕΔ  γϟʔυ͋ͨΓͷॻ͖ࠐΈ੍ݶΛΫϦΞ͠΍͘͢ͳΔ w .#ඵ w ϨίʔυඵίϨ

     ωοτϫʔΫଳҬతʹ΋΍͞͠Ί  -BNCEB࣮ߦճ਺తʹ΋΍͞͠Ί ‣ ૹ৴͕ඇಉظʹͳΔ ‣ qVFOUEͱ૊Έ߹Θͤͯ࢖͏ͱ΍Γ΍͍͢  BXTqVFOUQMVHJOLJOFTJT "NB[PO,JOFTJT1SPEVDFS-JCSBSZ ,1-  
  20. ͜͏ͳΔ༧ఆ   Lambda SNS SQS KPLͰσʔλΛ ·ͱΊͯૹ৴ PrivateSubnet Worker

    Worker PublicSubnet Web Web Kinesis
 Data Streams CloudWatch Logs
  21. ‣   ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ  ৄࡉϩά͕ݟΕΔ  ϩάॲཧػߏ͕ൺֱత؆୯  %ZOBNP%#Λ࢖͏ඞཁ͕ͳ͘ͳͬͨ

     ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕খ͘͞ͳͬͨ ‣   NPDLαʔόʔʹqVFOUEΛΠϯετʔϧͪ͠ΌͬͨͰ΋·͍͔͊ͬʂ  γϟʔυ਺ͷ੍ޚͲ͏͠Α͏ ߟ͑த ݁Ռ ͨͿΜ  
  22. ‣ ߴ଎Ͱεέʔϥϒϧͳ׬શϚωʔδυܕͷ࣌ܥྻσʔλϕʔε  ॻ͖ࠐΈΫΤϦετϨʔδʹΑΔैྔ՝ۚ  *0ͷੑೳ ੍ݶ ͸·ͩग़͍ͯͳ͍ʜ ŠƄŞž 

    σʔλอଘظ͕ؒઃఆͰ͖ΔͷͰ࡟আॲཧ΋ඞཁͳ͍ʁ ‣ ϓϨϏϡʔਃ੥ͯ͠Δ͚Ͳ·ͩདྷͳ͍ʜ ظ଴ͷ৽੕"NB[PO5JNFTUSFBN  
  23. ͜͏ͳΔ͔΋ʁ   Lambda SNS SQS KPLͰσʔλΛ ·ͱΊͯૹ৴ PrivateSubnet Worker

    Worker PublicSubnet Web Web Timestream Kinesis
 Data Streams CloudWatch Logsʹ
 ୅Θͬͯ࢖͑Δʁ
  24. ‣ େྔͷNPDL͔Βൃੜ͢ΔϩάΛॲཧ͢ΔػߏͷมભΛݟ͖ͯͨ ‣ ͳʹ͛ʹશ෦TFSWFSMFTT  අ༻΋NPDLͷՔಇ਺ʹԠͯ͡εέʔϧ͢Δ ෦෼͕ଟ͍  ‣ ࣗ෼Ͱ͸࡞Δͷ͕೉͍͠ػߏ͕ͨ͘͞Μ༻ҙ͞Ε͍ͯͯ


    ؾܰʹࢼߦࡨޡ͕Ͱ͖Δͷ͸Ϋϥ΢υͳΒͰ͸  ύζϧͷΑ͏ʹ͍Ζ͍Ζ૊Έ߹Θ͍ͤͯ͘ͷ͸ָ͍͠ʂ ‣ େن໛ͳσʔλΛѻ͏ͱ͖ʹ࠷ޙʹϘτϧωοΫʹͳΔͷ͸΍ͬͺΓ*0 ·ͱΊ