mockmockの大量のログをいい感じに捌きたい

ͷ େྔͷϩάΛ͍͍ײ͡ʹࡹ͖͍ͨ 'VLVPLBEFW ໟརܒଠ

‣ 'VTJD$P -UE ‣ ൃҊऀϓϩμΫτΦʔφʔ ‣ *P5ܥͷडୗҊ݅΋ ‣
"84 403"$0. (P 3VCZ ‣ ޷͖ͳ"84αʔϏε,JOFTJT ໟརܒଠʢ΋͏Γ͚͍ͨʣ

ࠓ೔͓࿩͍ͨ͜͠ͱ

ͷલʹʜ

ͬͯͳʹʁ

*P5։ൃऀ޲͚ ٙࣅσʔλੜ੒αʔϏε

*P5ͷ࿩

Ϟϊ͕ Thing

Πϯλʔωοτܦ༝Ͱ Thing

αʔόʹσʔλΛૹΓ Backend Thing D a t a

ղੳͨ͠Γͯ͠ Backend Thing D a t a

Կ͔͢Δ Backend Thing D a t a

*P5ͷ࿩ ͓ΘΓ

*P5 ։ൃ͠·͢

ςετ͍ͨ͠ Backend Thing D a t a

σʔλૹͬͯ΄͍͠ Backend Thing D a t a

Ϟϊʹͩͬͯ ౎߹͸͋Δ

͑͐ͬɺ ։ൃʹ൒೥΋͔͔Δͷʂʁ

͑͐ͬɺ ϗϯτʹյ͞ͳ͍ͱ Τϥʔσʔλग़ͤͳͷʂʁ

ෛՙࢼݧ༻ʹ୆ ཉ͍͚͠Ͳʜ ݸສԁʂʁ

ෛՙࢼݧ༻ʹ୆ ༻ҙ͚ͨ͠Ͳʜ ୭͕ૢ࡞͢Δͷ͜Εʂʁ

ςετʹࠔΔ

ͦ͏ͩʂ

γϛϡϨʔλʔΛ࡞Ζ͏ʂ Backend Simulator D a t a

΍͹͍ɺ ϦϦʔεʹؒʹ߹Θͳ͍ʂ

೉͗ͯ͢͠ ʓʓ͞Μ ࡞ͬͨਓ ͔͠ ಈ͔ͤͳ͍ʜ

ٱ͠ͿΓʹಈ͔ͨ͠Β શવಈ͔ͳ͍ʜ

ͦ͜Ͱ

ࢥ͍Ͳ͓ΓʹͳΔϞϊ Backend D a t a

͍ͭͰ΋ ཉ͍͠σʔλΛ ཉ͍͠ྔ͚ͩ ૹͬͯ͘ΕΔʂ

͏Ε͍͠

ը໘Πϝʔδ ࢥ͍௨ΓͷϑΥʔϚοτͰ+40/σʔλΛੜ੒ Ծ૝σόΠεͷεςʔλεͱͦͷมԽΛ೚ҙʹઃఆ ͞·͟·ͳ࣌ܥྻσʔλੜ੒ػೳ

ͷͬ͘͟Γͨ͠࢓૊Έ

ϩάॲཧػߏ PublicSubnet PrivateSubnet PrivateSubnet S3 Bucket (Lot) S3
Bucket (Binary) Web Web Worker Worker Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞

PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary)
Web Web Worker Worker WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ PrivateSubnet Mock Mock Mock Mock Mock ϩάॲཧػߏ

Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ

Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ

Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ

Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ

ϩάॲཧػߏ PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket
(Binary) Web Web Worker Worker ίϯιʔϧͰ mockΛૢ࡞ mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock mockىಈ Mock Mock [ mock ] Go੡όΠφϦ͔Βཱͯͨ Ծ૝σόΠε໾ϓϩηε [ tower ] Go੡όΠφϦ͔Βཱͯͨ mock؅ཧϓϩηε

ࠓ೔͓࿩͍ͨ͜͠ͱ վΊͯʂ

Web Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ Քಇঢ়گɾϩάૹ৴ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ ϩάॲཧػߏ

͜ͷ͋ͨΓʹ൓ө͠ͳ͍ͱ͍͚ͳ͍

‣ ֤NPDLͷৄࡉͳϩάΛऔΓ͍ͨ ૹ৴σʔλૹ৴ઌϓϩτίϧFUD ϩάͷαΠζ͸ૹ৴σʔλαΠζʹΑͬͯେ͖͘มಈ w ,#ҎԼd਺े,# ‣ ֤NPDLͷঢ়ଶΛϦΞϧλΠϜʹߋ৽͍ͨ͠
3%4΁ͷߋ৽ ‣ ࠷େNPDLՔಇ਺͸࣮੷Ͱສ୆ Քಇ਺ͷͱ͖΋͋Ε͹͍͖ͳΓສʹͳΔ͜ͱ΋͋Δ ‣ Քಇ਺ʹΑͬͯඅ༻΋εέʔϧ͢ΔΑ͏ʹ͍ͨ͠ NPDLNPDLࣗମ΋ैྔ՝ۚ ཁ݅

͋ͳͨͳΒͲ͏͢Δʁ

ୈ̍ੈ୅

‣ NPDLαʔόʔʹ͸Կ΋Πϯετʔϧ͠ͳ͍ ؂ࢹͱ͔ΞοϓάϨʔυͱ͔͕໘౗ ‣ ϩάॲཧػߏʹωοτϫʔΫଳҬΛۃྗׂ͔ͳ͍ ϝΠϯػೳͷٙࣅσʔλૹ৴ʹ࢖͍͍ͨ ୈੈ୅ํ਑

͜͏ͳΓ·ͨ͠ S3 Bucket Lambda PrivateSubnet Worker Worker S3
Bucket ૹ৴ຖʹϩάΛ ॻ͖ग़͢ 1෼ຖʹશmockͷ ϩάΛूܭ S3΁ͷputͰ Lambda͕ൃՐ mockͷঢ়ଶΛ HTTPͰ఻͑Δ mockຖʹϩάΛ όϥͯ͠อଘ

‣ NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕࠷௿ݶ ‣ ϩάͷ఻ୡʹ΍ͨΒ͕͔͔࣌ؒΔ
ϩάͷ಺༰͕ඞཁ࠷௿ݶ ҟৗܥ·Ͱߟ͑Δͱϩάͷू໿ॲཧ͕ෳࡶʹʜ XPSLFSαʔόʔʹNPDLͷঢ়ଶΛ఻͑Δͱ͜Ζ͕ಉظతࣦഊ࣌ͷ࠶ૹॲཧͱ͔ʜ ͳΜ͔͔ͬ͜ѱ͍ʜ ݁Ռ

ୈ̎ੈ୅

‣ NPDLαʔόʔʹ͸Կ΋Πϯετʔϧ͠ͳ͍ ΍ͬͺΓ؂ࢹͱ͔ΞοϓάϨʔυͱ͔͕໘౗ ‣ ϩάॲཧػߏʹωοτϫʔΫଳҬΛ͋Δఔ౓ׂ͜͏ ଍Γͳ͔ͬͨΒ͓ۚͷྗʹཔͬͯੜ͖͍ͯ͜͏ ‣ ΍ͬͺΓ%ZOBNP%#͔ͳ͊ʜ
Ͱ΋ΩϟύγςΟͷ؅ཧ໘౗ͩͳ͊ʜ ୈ̎ੈ୅ํ਑

%ZOBNP%#0O%FNBOEര஀ʂ

‣ ैདྷͷ%ZOBNP%#Ͱ͸ɺΩϟύγςΟϢχοτ ಡΈॻ͖ੑೳ Λ  ͋Β͔͡Ίઃఆ͓ͯ͘͠ඞཁ͕͋ͬͨ 8$6 8SJUF$BQBDJUZ6OJU
3$6 3FBE$BQBDJUZ6OJU ‣ ΩϟύγςΟϢχοτͷΦʔτεέʔϦϯά͸ઃఆͰ͖͕ͨɺ  ͙͢ʹεέʔϧ͢ΔΘ͚Ͱ͸ͳ͍ ‣ 0O%FNBOEϞʔυͰ͸ɺ࢖ͬͨΩϟύγςΟϢχοτʹରͯ͠  ैྔ՝ۚͯ͘͠ΕΔʂ ͋Β͔͡ΊΩϟύγςΟϢχοτΛઃఆ͓ͯ͘͠ඞཁ͕ͳ͍ʂ %ZOBNP%#0O%FNBOEര஀ʂ

͜͏ͳΓ·ͨ͠ DynamoDB (On-Demand) Lambda SNS SQS ֤mock͕௚઀ DynamoDBʹϩάΛ
ॻ͖ࠐΉ PrivateSubnet Worker Worker DynamoDB Streamsʹ ΑͬͯLambdaΛൃՐ ඇಉظʹσʔλΛ ߋ৽Ͱ͖ΔΑ͏ʹ

‣ NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘ ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ ৄࡉϩά͕ݟΕΔ ϩάॲཧػߏ͕ൺֱత؆୯
‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕େ͖͘ͳͬͨʜ ‣ %ZOBNP%#0O%FNBOE͸ΰϦΰϦ࢖͏ͱߴ͍ʜ ݁Ռ

ୈ̏ੈ୅

‣ NPDLαʔόʔʹ͸Կ΋Πϯετʔϧ͠ͳ͍ ΍ͬͺΓ؂ࢹͱ͔ΞοϓάϨʔυͱ͔͕໘౗ ‣ ϩάॲཧػߏʹωοτϫʔΫଳҬΛ͋Δఔ౓ׂ͜͏ ଍Γͳ͔ͬͨΒ͓ۚͷྗʹཔͬͯੜ͖͍ͯ͜͏ ‣ %ZOBNP%#0O%FNBOE͸୤٫͠Α͏
͓ۚʹ΋ݶք͕͋ΔΑͶ ୈ̏ੈ୅ํ਑

‣ ΩϟύγςΟ͕ৗʹେ͖͘มಈ͢Δ৔߹͸༗ޮ ‣ NPDLNPDLͰ΋มಈ͸͢Δ͕ɺ֤NPDL͕ॻ͖ࠐΉྔ͸Ұఆ ؤுΕ͹1SPWJTJPOFE͕࢖͑ΔͷͰ͸ʁ ΰϦΰϦ%ZOBNP%#0O%FNBOE ΋ͣͬ͠ͱҰఆͰ࢖͍ଓ͚ͨΒʜ
0O%FNBOE 1SPWJTJPOFE ར༻͍ͯ͠ΔΩϟύγςΟͷ໿ഒΛ ϓϩϏδϣϯ͍ͯ͠Δঢ়ଶ ʹ

‣ ىಈ࣌ʹඞཁͳΩϟύγςΟΛ༧ଌͰ͖Ε͹࣮ݱ͢Δ ૹ৴σʔλαΠζʹΑͬͯϩάͷαΠζ͕େ͖͘มಈ ૹ৴σʔλαΠζΛ༧ଌ͢Δͷ͸ࠔ೉ w ૹ৴σʔλͷ෦෼͚ͩผͷػߏΛ࢖ͬͯอଘ͢Ε͹Α͍ͷͰ͸ʁ ‣ ΩϟύγςΟ͸૿΍͢ํ޲ʹ͸ճ਺੍ݶ͕ແ͍͚ͲɺݮΒ͢ํ޲ʹ
͸ճ਺੍ݶ͕͋Δ NPDLͷՔಇ୯Ґ͝ͱʹΩϟύγςΟΛࢉग़ͯ͠௥Ճ DSPOͰ࣌ؒຖʹՔಇ୯ҐͷΩϟύγςΟͷ߹ܭ஋Λग़ͯ͠ద༻ %ZOBNP%#0O%FNBOE͔Βͷ୤٫

‣ ෳ਺ͷαʔόʔ͔ΒϚ΢ϯτͰ͖Δڞ༗ετϨʔδ ωοτϫʔΫϑΝΠϧγεςϜͱͯ͠Ϛ΢ϯτ ‣ ༰ྔ͕ࣗಈͰεέʔϧ͢Δ ‣ εϧʔϓοτ੍͕ޚՄೳ όʔετεϧʔϓοτϑΝΠϧγεςϜͷαΠζʹ߹Θͤͯεέʔϧ
ϓϩϏδϣϯυεϧʔϓοτεϧʔϓοτΛ௚઀ઃఆ ‣ ෳ਺ͷ";͔ΒΞΫηεՄೳ "NB[PO&MBTUJD'JMF4ZTUFN &'4

͜͏ͳΓ·ͨ͠ DynamoDB (Provisioned) Lambda SNS SQS DynamoDBʹ͸εςʔλεߋ৽༻ͷϩά EFSʹ͸׬શͳϩάΛॻ͖ࠐΉ
PrivateSubnet Worker Worker DynamoDB Streamsʹ ΑͬͯLambdaΛൃՐ EFS ΩϟύγςΟ͸ mockىಈ࣌ʹௐ੔ PublicSubnet Web Web ඇಉظʹσʔλΛ ߋ৽Ͱ͖ΔΑ͏ʹ

‣ NPDLαʔόʔʹԿ΋Πϯετʔϧ͠ͳ͍͍ͯ͘ ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ ৄࡉϩά͕ݟΕΔ ϩάॲཧػߏ͕ൺֱత؆୯
%ZOBNP%#ͷඅ༻͕࠷దԽ͞Εͨ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕΋ͬͱେ͖͘ͳͬͨʜ Ұఆ࣌ؒܦաޙʹ&'4͔ΒσʔλΛ࡟আ͢Δͷ͕खؒ ‣ &'4΋*0Λ૿΍ͦ͏ͱ͢ΔͱͦΕͳΓʹඅ༻͕ʜ ݁Ռ

ୈ̐ੈ୅  ͜Ε͔Β

‣ NPDLαʔόʔʹԿ͔ΛΠϯετʔϧ͢ΔͷΛڐ༰͠Α͏ શମతʹ͜ͳΕ͖ͯͨͷͰɺࠓͳΒ͍͚Δؾ͕͢Δ ‣ ࠓͦ͜,JOFTJTͱ޲͖߹͓͏ γϟʔυͷѻ͍͕େมͳͷͰࠓ·Ͱආ͚͖ͯͨ ‣ ૣ͘΋&'4͔Β୤٫͍ͨ͠ؾ࣋ͪ
ωοτϫʔΫଳҬͷϘτϧωοΫʹͳΓ͕ͪ ͓ۚͰղܾͰ͖Δ͚Ͳ͓ۚ͸େࣄ ϩάͷͨΊʹNPDL͔ΒϲॴʹσʔλΛૹ͍ͬͯΔͷ΋ΠϠͩ ୈ̐ੈ୅ํ਑

‣ ,JOFTJT%BUB4USFBNTʹσʔλΛ౤͛ࠐΉϥΠϒϥϦ ‣ ͋Δఔ౓σʔλΛ·ͱΊͯૹͬͯ͘ΕΔ γϟʔυ͋ͨΓͷॻ͖ࠐΈ੍ݶΛΫϦΞ͠΍͘͢ͳΔ w .#ඵ w ϨίʔυඵίϨ
ωοτϫʔΫଳҬతʹ΋΍͞͠Ί -BNCEB࣮ߦճ਺తʹ΋΍͞͠Ί ‣ ૹ৴͕ඇಉظʹͳΔ ‣ qVFOUEͱ૊Έ߹Θͤͯ࢖͏ͱ΍Γ΍͍͢ BXTqVFOUQMVHJOLJOFTJT "NB[PO,JOFTJT1SPEVDFS-JCSBSZ ,1-

‣ ͦͷ໊ͷ௨ΓϩάΛ؅ཧͯ͘͠ΕΔ΍ͭ $MPVE8BUDI-PHT*OTJHIUTͰϩάσʔλΛ෼ੳ͢Δ͜ͱ΋Մೳ อଘظؒ΋ઃఆͰ͖Δʂ ‣ Ұճݕ౼͚ͯͨ͠Ͳ੍ݶ಺༰Λצҧ͍͍ͯͯ͠Ϙπʹʜʂ ‣ ϩά౤ೖස౓ͷ੍ݶ؇࿨͕Ͳ͜·Ͱ͍͚Δ͔ʜɻ
σϑΥϧτͰ͸݅ඵͷτϥϯβΫγϣϯ BQOPSUIFBTU "NB[PO$MPVE8BUDI-PHT

͜͏ͳΔ༧ఆ Lambda SNS SQS KPLͰσʔλΛ ·ͱΊͯૹ৴ PrivateSubnet Worker
Worker PublicSubnet Web Web Kinesis  Data Streams CloudWatch Logs

‣ ϩάͷ఻ୡ͕΄΅ϦΞϧλΠϜ ৄࡉϩά͕ݟΕΔ ϩάॲཧػߏ͕ൺֱత؆୯ %ZOBNP%#Λ࢖͏ඞཁ͕ͳ͘ͳͬͨ
ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳҬ͕খ͘͞ͳͬͨ ‣ NPDLαʔόʔʹqVFOUEΛΠϯετʔϧͪ͠ΌͬͨͰ΋·͍͔͊ͬʂ γϟʔυ਺ͷ੍ޚͲ͏͠Α͏ ߟ͑த ݁Ռ ͨͿΜ

ୈੈ୅  ͋ΘΑ͘͹

ظ଴ͷ৽੕"NB[PO5JNFTUSFBN

‣ ߴ଎Ͱεέʔϥϒϧͳ׬શϚωʔδυܕͷ࣌ܥྻσʔλϕʔε ॻ͖ࠐΈΫΤϦετϨʔδʹΑΔैྔ՝ۚ *0ͷੑೳ ੍ݶ ͸·ͩग़͍ͯͳ͍ʜ ŠƄŞž
σʔλอଘظ͕ؒઃఆͰ͖ΔͷͰ࡟আॲཧ΋ඞཁͳ͍ʁ ‣ ϓϨϏϡʔਃ੥ͯ͠Δ͚Ͳ·ͩདྷͳ͍ʜ ظ଴ͷ৽੕"NB[PO5JNFTUSFBN

͜͏ͳΔ͔΋ʁ Lambda SNS SQS KPLͰσʔλΛ ·ͱΊͯૹ৴ PrivateSubnet Worker
Worker PublicSubnet Web Web Timestream Kinesis  Data Streams CloudWatch Logsʹ  ୅Θͬͯ࢖͑Δʁ

·ͱΊ

‣ େྔͷNPDL͔Βൃੜ͢ΔϩάΛॲཧ͢ΔػߏͷมભΛݟ͖ͯͨ ‣ ͳʹ͛ʹશ෦TFSWFSMFTT අ༻΋NPDLͷՔಇ਺ʹԠͯ͡εέʔϧ͢Δ ෦෼͕ଟ͍ ‣ ࣗ෼Ͱ͸࡞Δͷ͕೉͍͠ػߏ͕ͨ͘͞Μ༻ҙ͞Ε͍ͯͯ 
ؾܰʹࢼߦࡨޡ͕Ͱ͖Δͷ͸Ϋϥ΢υͳΒͰ͸ ύζϧͷΑ͏ʹ͍Ζ͍Ζ૊Έ߹Θ͍ͤͯ͘ͷ͸ָ͍͠ʂ ‣ େن໛ͳσʔλΛѻ͏ͱ͖ʹ࠷ޙʹϘτϧωοΫʹͳΔͷ͸΍ͬͺΓ*0 ·ͱΊ

ΛҰॹʹ ։ൃͯ͘͠ΕΔ஥ؒΛืूதʂ ͝ڵຯ͋Δํ͸ໟར·Ͱʂ Go Ruby AWS IoT ΋ͬͱ͏·͘΍ΕΔʂ ͓΋͠Ζͦ͏ʂ in
Fukuoka

mockmockの大量のログをいい感じに捌きたい

mockmockの大量のログをいい感じに捌きたい

More Decks by Keita Mohri

Other Decks in Technology

Featured

Transcript