Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IoTデバイスの疑似データ送信システムにおける サーバーレスなログ処理機構の試行錯誤

Keita Mohri
December 14, 2019

IoTデバイスの疑似データ送信システムにおける サーバーレスなログ処理機構の試行錯誤

ServerlessDays Fukuoka 2019

Keita Mohri

December 14, 2019
Tweet

More Decks by Keita Mohri

Other Decks in Technology

Transcript

  1.  ‣ 'VTJD$P -UE ‣ ϓϦϯγύϧΤϯδχΞNPDLNPDLνʔϜϦʔμʔ ‣ "84 403"$0. (P

    3VCZ ‣ ਺࣌ؒͰ׬શཧղʂΘΓͱΰπ͍,VCFSOFUFTϋϯζΦϯʂʂ w IUUQTRJJUBDPN,UB.JUFNTDFDEEGEE ໟརܒଠʢ΋͏Γ͚͍ͨʣ
  2.  PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary) Web

    Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ PrivateSubnet RDS
  3.  PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary) Web

    Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ PrivateSubnet RDS
  4.  PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary) Web

    Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ mockϓϩηε ىಈ PrivateSubnet RDS
  5.  PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary) Web

    Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockϓϩηε ىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ PrivateSubnet mockՔಇঢ়گ Λ௨஌ RDS ϩάॲཧػߏ ʁ Log Storage ϩάૹ৴ ϩάอଘ
  6.  PublicSubnet PrivateSubnet S3 Bucket (Lot) S3 Bucket (Binary) Web

    Web Worker Worker mockૢ࡞JobΛ ൃߦ ઃఆϑΝΠϧΛ Ξοϓϩʔυ mockαʔόʔʹ ૢ࡞ࢦࣔ ઃఆϑΝΠϧ όΠφϦΛDL PrivateSubnet Mock Mock Mock Mock Mock Mock mockϓϩηε ىಈ WebίϯιʔϧͰ ಈ࡞Λઃఆ mockΛૢ࡞ PrivateSubnet mockՔಇঢ়گ Λ௨஌ RDS ϩάॲཧػߏ ʁ Log Storage ϩάૹ৴ ϩάอଘ Mock Mock [ mock ] Go੡όΠφϦ͔Βཱͯͨ Ծ૝σόΠε໾ϓϩηε [ tower ] Go੡όΠφϦ͔Βཱͯͨ mock؅ཧϓϩηε
  7.  <*/165> ֤NPDLϓϩηε͕ ٙࣅσʔλૹ৴ͷͨͼʹ ϩάΛੜ੒ɾૹ৴ <065165> Կ͔͠ΒͷετϨʔδʹ ੜϩάΛอଘ <065165> ϩάΛղੳͯ͠

    NPDLͷՔಇঢ়ଶΛ௨஌ PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock Mock ϩάॲཧػߏ ʁ Log Storage
  8.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋
  9.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋ NPDLͷσʔλૹ৴ִؒ͸ ࠷খͰඵ
  10.  ૹ৴ͨٙ͠ࣅσʔλͷ αΠζʹΑͬͯ ϩάͷαΠζ͸େ͖͘มಈ ,#ະຬd਺े,# PrivateSubnet Worker Worker PrivateSubnet Mock

    Mock Mock Mock Mock Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋ NPDLͷσʔλૹ৴ִؒ͸ ࠷খͰඵ
  11.  ૹ৴ͨٙ͠ࣅσʔλͷ αΠζʹΑͬͯ ϩάͷαΠζ͸େ͖͘มಈ ,#ະຬd਺े,# NPDLͷՔಇঢ়ଶΛ ϦΞϧλΠϜʹ൓ө͍ͨ͠ PrivateSubnet Worker Worker

    PrivateSubnet Mock Mock Mock Mock Mock Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋ NPDLͷσʔλૹ৴ִؒ͸ ࠷খͰඵ
  12.  ૹ৴ͨٙ͠ࣅσʔλͷ αΠζʹΑͬͯ ϩάͷαΠζ͸େ͖͘มಈ ,#ະຬd਺े,# NPDLͷՔಇঢ়ଶΛ ϦΞϧλΠϜʹ൓ө͍ͨ͠ PrivateSubnet Worker Worker

    PrivateSubnet Mock Mock Mock Mock Mock Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋ NPDLͷσʔλૹ৴ִؒ͸ ࠷খͰඵ Ұఆظ͕ؒܦաͨ͠Β ϩάΛফ͍ͨ͠
  13.  ૹ৴ͨٙ͠ࣅσʔλͷ αΠζʹΑͬͯ ϩάͷαΠζ͸େ͖͘มಈ ,#ະຬd਺े,# NPDLͷՔಇ਺ʹΑͬͯ අ༻΋εέʔϧ͍ͤͨ͞ NPDLͷՔಇঢ়ଶΛ ϦΞϧλΠϜʹ൓ө͍ͨ͠ PrivateSubnet

    Worker Worker PrivateSubnet Mock Mock Mock Mock Mock Mock ϩάॲཧػߏ ʁ Log Storage NPDLͷՔಇ਺͸ ࠷େͰສ୆ Քಇ਺͔Β ͍͖ͳΓສʹͳΔ͜ͱ΋ NPDLͷσʔλૹ৴ִؒ͸ ࠷খͰඵ Ұఆظ͕ؒܦաͨ͠Β ϩάΛফ͍ͨ͠
  14.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock S3 Bucket S3 Bucket Lambda ૹ৴ຖʹϩάΛ ॻ͖ग़͢ 1෼ຖʹશmockͷ ϩάΛूܭ S3΁ͷputͰ Lambda͕ൃՐ mockͷঢ়ଶΛ HTTPͰ఻͑Δ mockຖʹϩάΛ όϥͯ͠อଘ
  15. ୈੈ୅݁Ռ  (PPE #BE ‣ NPDLαʔόʔʹԿ΋Πϯετʔϧ ͠ͳ͍͍ͯ͘ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳ Ҭ͕࠷௿ݶ

    ‣ ϩάͷ఻ୡʹ͕͔͔࣌ؒΔ ‣ ϩάͷू໿෼ղॲཧ͕͋ΔͨΊ
 ϩάͷαΠζ͕େ͖͘Ͱ͖ͳ͍ ‣ ҟৗܥ·Ͱߟ͑ΔͱUPXFSͰͷϩά ू໿ॲཧ͕ෳࡶʹͳΔ ‣ 4͔ΒظؒࢦఆͰϩάΛऔΓʹ͍͘ ‣ XPSLFSʹNPDLঢ়ଶΛ఻͑Δͱ͜ Ζ͕ಉظత w ࣦഊ࣌ͷ࠶ૹॲཧΛͤͶ͹
  16. ୈੈ୅݁Ռ  (PPE #BE ‣ NPDLαʔόʔʹԿ΋Πϯετʔϧ ͠ͳ͍͍ͯ͘ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳ Ҭ͕࠷௿ݶ

    ‣ ϩάͷ఻ୡʹ͕͔͔࣌ؒΔ ‣ ϩάͷू໿෼ղॲཧ͕͋ΔͨΊ
 ϩάͷαΠζ͕େ͖͘Ͱ͖ͳ͍ ‣ ҟৗܥ·Ͱߟ͑ΔͱUPXFSͰͷϩά ू໿ॲཧ͕ෳࡶʹͳΔ ‣ 4͔ΒظؒࢦఆͰϩάΛऔΓʹ͍͘ ‣ XPSLFSʹNPDLঢ়ଶΛ఻͑Δͱ͜ Ζ͕ಉظత w ࣦഊ࣌ͷ࠶ૹॲཧΛͤͶ͹ ͳΜ͔μα͍ʂ
  17. ‣ NPDLαʔόʔʹ͸Կ΋Πϯετʔϧ͠ͳ͍  ΍ͬͺΓ؂ࢹͱ͔ΞοϓάϨʔυͱ͔͕໘౗ ‣ ϩάॲཧػߏʹωοτϫʔΫଳҬΛ͋Δఔ౓ׂ͜͏  ଍Γͳ͔ͬͨΒ͓ۚͷྗʹཔͬͯੜ͖͍ͯ͜͏ ‣ 4͔Β%ZOBNP%#ʹม͑ͯΈΑ͏

     %ZOBNP%#ͳΒ֤NPDL͔Β௚઀ϩάΛೖΕͯ΋େৎ෉ͦ͏  ΫΤϦ͕ॻ͚ΔͷͰอଘͨ͠ϩά͕औಘ͠΍͍͢  Ͱ΋ΩϟύγςΟͷ؅ཧ͕໘౗ʜ ୈ̎ੈ୅ํ਑ 
  18. ‣ ैདྷͷ%ZOBNP%#Ͱ͸ɺΩϟύγςΟϢχοτ ಡΈॻ͖ੑೳ Λ
 ͋Β͔͡Ίઃఆ͓ͯ͘͠ඞཁ͕͋ͬͨ  8$6 8SJUF$BQBDJUZ6OJU  

    3$6 3FBE$BQBDJUZ6OJU  ‣ ΩϟύγςΟϢχοτͷΦʔτεέʔϦϯά͸ઃఆͰ͖Δ͕ɺ
 ॠؒతʹεέʔϧ͢ΔΘ͚Ͱ͸ͳ͍ ෼ऑ͔͔Δ  ‣ 0O%FNBOEϞʔυͰ͸ɺ࢖ͬͨΩϟύγςΟϢχοτʹରͯ͠
 ैྔ՝ۚͯ͘͠ΕΔʂ  ͋Β͔͡ΊΩϟύγςΟϢχοτΛઃఆ͓ͯ͘͠ඞཁ͕ͳ͍ʂ %ZOBNP%#0O%FNBOEര஀ʂ 
  19.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock DynamoDB (On-Demand) Lambda SNS SQS ֤mock͕௚઀ DynamoDBʹϩάΛ ॻ͖ࠐΉ Worker͔ΒσʔλΛ औΓʹߦ͘͜ͱͰ ඇಉظʹσʔλΛߋ৽ DynamoDB Streamsʹ ΑͬͯLambdaΛൃՐ ੜϩά͸DynamoDBΛࢀর TTLͰҰఆظؒܦաޙʹ࡟আ
  20. ୈੈ୅݁Ռ  (PPE #BE ‣ NPDLՔಇঢ়ଶͷ఻ୡ͕΄΅ϦΞϧ λΠϜ ‣ ػߏ͕ൺֱత؆୯ʹͳͬͨ ‣

    ϩάͷू໿෼ղॲཧ͕ͳ͘ͳͬͨ ͷͰɺେ͖Ίͷϩά΋ૹΕΔΑ͏ʹ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫ
 ଳҬ͕େ͖͘ͳͬͨ ‣ %ZOBNP%#0O%FNBOE͸
 ΰϦΰϦ࢖͏ͱߴ͍ʜ
  21. ୈੈ୅݁Ռ  (PPE #BE ‣ NPDLՔಇঢ়ଶͷ఻ୡ͕΄΅ϦΞϧ λΠϜ ‣ ػߏ͕ൺֱత؆୯ʹͳͬͨ ‣

    ϩάͷू໿෼ղॲཧ͕ͳ͘ͳͬͨ ͷͰɺେ͖Ίͷϩά΋ૹΕΔΑ͏ʹ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫ
 ଳҬ͕େ͖͘ͳͬͨ ‣ %ZOBNP%#0O%FNBOE͸
 ΰϦΰϦ࢖͏ͱߴ͍ʜ #JLLVSJ ‣ ࢥͬͨҎ্ʹॠؒతʹεέʔϧ͢Δ w ਺෼Ͱ<8$6>Ͱ΋0,
  22.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock DynamoDB (Provisioned) Lambda SNS SQS DynamoDBʹ͸ mockঢ়ଶߋ৽༻ͷϩά EFSʹ͸׬શͳϩάΛ ॻ͖ࠐΉ ΩϟύγςΟ͸ mockىಈ࣌ʹௐ੔ EFS
  23. ୈੈ୅݁Ռ  (PPE #BE ‣ %ZOBNP%#ͷඅ༻͕࠷దԽ͞Εͨ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫ
 ଳҬ͕΋ͬͱେ͖͘ͳͬͨ ‣

    Ұఆ࣌ؒܦաޙʹ&'4͔ΒσʔλΛ ࡟আ͢Δͷ͕खؒ ‣ &'4΋*0Λ૿΍ͦ͏ͱ͢ΔͱͦΕͳ Γʹඅ༻͕ʜ
  24. ୈੈ୅݁Ռ  (PPE #BE ‣ %ZOBNP%#ͷඅ༻͕࠷దԽ͞Εͨ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫ
 ଳҬ͕΋ͬͱେ͖͘ͳͬͨ ‣

    Ұఆ࣌ؒܦաޙʹ&'4͔ΒσʔλΛ ࡟আ͢Δͷ͕खؒ ‣ &'4΋*0Λ૿΍ͦ͏ͱ͢ΔͱͦΕͳ Γʹඅ༻͕ʜ :BSBLBTIJ ‣ ςετ؀ڥͰ8$6໭Βͣߴֹ੥ٻ
  25. ‣ NPDLαʔόʔʹԿ͔ΛΠϯετʔϧ͢ΔͷΛڐ༰͠Α͏  શମతʹ͜ͳΕ͖ͯͨͷͰɺࠓͳΒ͍͚Δؾ͕͢Δ ‣ ࠓͦ͜,JOFTJTͱ޲͖߹͓͏  γϟʔυ͕ෳ਺ʹͳΔͱͪΐͬͱେม͚ͩͲ ‣ ૣ͘΋&'4͔Β୤٫͍ͨ͠ؾ࣋ͪ

     ωοτϫʔΫଳҬͷϘτϧωοΫʹͳΓ͕ͪ  ͓ۚͰղܾͰ͖Δ͚Ͳ͓ۚ͸େࣄ  ϩάͷͨΊʹNPDL͔ΒϲॴʹσʔλΛૹ͍ͬͯΔͷ΋͔ͬ͜ѱ͍ ୈ̐ੈ୅ํ਑ 
  26. ‣ ,JOFTJT%BUB4USFBNTʹσʔλΛ౤͛ࠐΉϥΠϒϥϦ ‣ ͋Δఔ౓σʔλΛ·ͱΊͯૹͬͯ͘ΕΔ  γϟʔυ͋ͨΓͷॻ͖ࠐΈ੍ݶΛΫϦΞ͠΍͘͢ͳΔ w .#ඵ w ϨίʔυඵίϨ

     ωοτϫʔΫଳҬతʹ΋΍͞͠Ί  -BNCEB࣮ߦճ਺తʹ΋΍͞͠Ί ‣ ૹ৴͕ඇಉظʹͳΔ ‣ qVFOUEͱ૊Έ߹Θͤͯ࢖͏ͱ΍Γ΍͍͢  BXTqVFOUQMVHJOLJOFTJT "NB[PO,JOFTJT1SPEVDFS-JCSBSZ ,1- 
  27.  PrivateSubnet Worker Worker PrivateSubnet Mock Mock Mock Mock Mock

    Mock Lambda SNS SQS Kinesis
 Data Streams KPLͰσʔλΛ ·ͱΊͯૹ৴ CloudWatch Logs ΞΧ΢ϯτ͝ͱʹLogGroup mock͝ͱʹLogStream
  28. ୈੈ୅݁Ռ  (PPE #BE ‣ &'4͔Βͷ୤٫ ‣ ϩάॲཧػߏʹׂ͘ωοτϫʔΫଳ Ҭ͕খ͘͞ͳͬͨ *LLB

    ‣ ͍ͭʹNPDLαʔόʔʹΠϯετʔ ϧͨ͠ ‣ ,1-ͰϩάΛू໿ͯ͠ૹΔͨΊɺ NPDLঢ়ଶͷ఻ୡϥά͕ŧŔŕŪũƄſ
  29. 

  30. 

  31. ‣ େྔͷNPDL͔Βൃੜ͢ΔϩάΛॲཧ͢ΔػߏͷมભΛݟ͖ͯͨ ‣ ͳʹ͛ʹશ෦TFSWFSMFTT  අ༻΋NPDLͷՔಇ਺ʹԠͯ͡εέʔϧ͢Δ ෦෼͕ଟ͍  ‣ ࣗ෼Ͱ͸࡞Δͷ͕೉͍͠ػߏ͕ͨ͘͞Μ༻ҙ͞Ε͍ͯͯ


    ؾܰʹࢼߦࡨޡ͕Ͱ͖Δͷ͸Ϋϥ΢υͳΒͰ͸  ύζϧͷΑ͏ʹ͍Ζ͍Ζ૊Έ߹Θ͍ͤͯ͘ͷ͸ָ͍͠ʂ ‣ େن໛ͳσʔλΛѻ͏ͱ͖ʹ࠷ޙʹϘτϧωοΫʹͳΔͷ͸΍ͬͺΓ*0 ·ͱΊ