ITS を支える 情報集約基盤 アーキテクチャ / ITS Tech Study #02

1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Hiroyoshi HOUCHI Office of Technology Development System Management Unit
 DeNA Co., Ltd. *54
   Λࢧ͑Δ
 ৘ใू໿ج൫
 ΞʔΩςΫνϟ *54
   5FDI
   4UVEZ
   
   "84
   *P5 1
2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࣗݾ঺հ ์஍
   ޺Ղ
   !IJYJ@IZJ
   w ೥݄d
   ݱΦʔϓϯϓϥοτϑΥʔϜγεςϜࣄۀ෦
    4PDJBM
   "1*
   ։ൃ
    ࣾ಺޲͚
   .PCJMF
   #BB4
   ։ൃϦʔμ
    ڠۀҊ݅
   ։ൃ
   
   Πϯϑϥ
    "OE"QQ
   Ϋϥ΢υΞʔΩςΫνϟઃܭ
   w ೥݄d
   ΦʔτϞʔςΟϒࣄۀຊ෦
    Ϋϥ΢υγεςϜΞʔΩςΫτɾڞ௨ج൫
   w ೥݄d
   γεςϜຊ෦
    શࣾԣஅΫϥ΢υγεςϜΞʔΩςΫτ
   Λ΍Δͭ΋Γ 
 ΦʔτϞʔςΟϒ΋Ҿ͖ଓ͖ 2
3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊ೔ͷൃද಺༰ 3
4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54
   Λࢧ͑Δ৘ใू໿ج൫ΞʔΩςΫνϟ ⁃ ं྆ొ࿥ػೳ ⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ৘ใͷऩू ⁃ ं྆ͦͷ΋ͷͷ৘ใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷ෼ੳ৘ใͷऩू ⁃ ं྆৘ใͷఏڙ ⁃ ं྆৘ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔ਺Λ༻͍ͨं྆৘ใͷఏڙ ⁃ ෼ੳγεςϜ΁ͷ৘ใఏڙ 4
5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. *54
   Λࢧ͑Δ৘ใू໿ج൫ΞʔΩςΫνϟ ⁃ ं྆ొ࿥ػೳ ⁃ ं྆ɾαʔόʹର͢ΔೝূೝՄ ⁃ ৘ใͷऩू ⁃ ं྆ͦͷ΋ͷͷ৘ใͷऩू ⁃ ΤοδίϯϐϡʔςΟϯάͷ෼ੳ৘ใͷऩू ⁃ ं྆৘ใͷఏڙ ⁃ ं྆৘ใͷϦΞϧλΠϜऔಘػೳͷఏڙ ⁃ δΦؔ਺Λ༻͍ͨं྆৘ใͷఏڙ ⁃ ෼ੳγεςϜ΁ͷ৘ใఏڙ 5
6. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࢀߟࢿྉ λΫϕϧʹ͍ͭͯ ⁃ AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ 
 ٕज़෦෼ʹ͍ͭͯ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ݕࡧΩʔϫʔυʮ AWS IoT DeNA ʯ(ൃද౰೔ Google ݕࡧ࣌఺) 6
7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ൃද෦෼ͷΞʔΩςΫνϟਤ 7 3. ෼ੳ৘ใऩू 2. ं྆৘ใऩू 1. ෼ੳγεςϜ౷߹
8. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 8 8
9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 9 { "topic": "pub/sub topic", "data": { "hoge": "fuga" } } ※ format ʹ͍ͭͯ͸গ͠ lambda Ͱٵऩͯ͠Δ෦෼͋Γ
10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 10 • όοϑΝϦϯά͞Εͨ৘ใΛ Lambda Ͱऔಘ • σʔλϑϩʔ্ Kinesis Data Streams → Lambda
 ࣮ࡍ͸ Lambda → Kinesis Data Streams
11. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹ 11 • topic Ͱάϧʔϐϯά • Cloud Pub/Sub ΁͸
 ·ͱΊͯˍฒྻ Publish [ { "topic": "pub/sub topic", "data": { "hoge": "fuga" } }, …… ]
12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1. ෼ੳγεςϜ౷߹෦ʹ͍ͭͯ • ޻෉఺ • جຊతʹ Network ͷ४උ + I/O ͕࣌ؒͱͯ΋͔͔Δ෦෼ • ୯ൃ࣮ߦͰ͸ͳͯ͘ɺόοϑΝϦϯά͢Δ͜ͱʹΑͬͯ Lambda ͷ࣮ߦճ਺ + ૯࣮ߦ࣌ؒΛେ෯ʹݮΒ͢͜ͱ(=ίετݮ)͕Ͱ͖Δ (※ ं྆୆਺ rps͙Β͍͘Δ ) • ΄΅΄΅ϦΞϧλΠϜʹ৘ใΛૹΔ͜ͱ͕Ͱ͖Δ • ߟྀ఺ • Kinesis Data Streams ͷγϟʔυ਺ • ผख๏ • ϦΞϧλΠϜੑΛ΋͏গ͠ଛͳ͍͍ͬͯͷͰ͋Ε͹ S3 Λ༻͍ͭͭɺ
 ఆظతʹ GCS ʹૹ৴͢Δ͜ͱ΋Մ • ͦ͏ͨ͠৔߹ɺ΋ͬͱίετΛݮΒ͢͜ͱ͕Ͱ͖Δ 12
13. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 13
14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ͷલͷɺࣄલ৘ใ… • ं͔྆Βͷૹ৴ϑΥʔϚοτ͸ bson (TechCon #33) • ࣄલʹߦ͏ं྆ొ࿥(TechCon #25)ʹΑΓҎԼͷΑ͏ʹͳΔ • ThingName ͸ূ໌ॻID • ThingShadow ʹ͸ҎԼͷ৘ใؚ͕·Ε͍ͯΔ (TechCon #29) 14 \
    
    
    TUBUF
    \
    
    
    
    
    SFQPSUFE
    \
    
    
    
    
    
    
    EFWJDF
    \
    
    
    
    
    
    
    
    
    !JE
    999999
    
    
    
    
    
    
    ^
    
    
    
    
    ^
    
    
    ^
    ^
15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 15
16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 16 Topic p/@s.b 
 (providers/taxibell/@status.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ {"@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407"}
17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 17 SQL SELECT aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as state.reported.status FROM p/@s.b Action ҎԼ΁ͷ Republish $$aws/things/${principal()}/shadow/update
18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 18 ݩͷσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" } }, "status": {"hoge":"fuga" } } ೖྗσʔλ { "state": { "reported": { "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407" } } ݁Ռσʔλ { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": “1528437407”, “hoge”: “fuga” } }
19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 19 Topic $aws/things/+/shadow/update/documents Payload { "current": { "state": { "reported": { "device": { "@id": "XXXXXX" }, "status": { "@pos": { "lat": "35.4429108606794", "lon": "139.642686919283" }, "@pos_ts": "1528437407", "@ts": "1528437407", "hoge":"fuga" } } , "desired": {} } }, "previous": {ུ} }
20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू 20 SQL SELECT {'data': {'device': current.state.reported.device, 'reported': current.state.reported.status}, 'topic': ‘${topicname}'} FROM '$aws/things/+/shadow/update/documents' Action Kinesis
21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2. ं྆৘ใऩू෦ʹ͍ͭͯ • ޻෉఺ • ৘ใͷ౷߹ • ThingShadow ͷػೳΛ༗ޮ׆༻͢Δ͜ͱͰɺࠩ෼͚ͩͰ͸ͳ͘ɺͦͷ࣌ͷं྆ ͷ͢΂ͯͷঢ়ଶΛ౉͢͜ͱ͕Մೳ • ΫϥΠΞϯτ͸ࠩ෼͚ͩͳͷͰ௨৴ྔతʹ༏͍͠ • ෼ੳ؀ڥʹͯ৘ใͷิ׬͕ඞཁͳ͘ͳΔ • ඞཁͳ෦෼ͷΈͷ Lambda ʹ཈͑Δ • Lambda Ͱ ThingShadow ͷߋ৽·Ͱ͢Δ͜ͱ͸Մೳ • ҰํͰ ThingShadow ͷؾ෼Ͱ࣮ߦ͕࣌ؒ૿͑ͯ͠·͏ʹίετ͕૿Ճ͢ΔՄೳ ੑ͕͋Δ 21
22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 22
23. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 23 Topic p/@a.b 
 (providers/taxibell/@analytics.bson) Payload ҎԼͷΑ͏ͳ json ͷ bson දݱ { "@t": "${ࣝผࢠ}", "@d" { "accuracy": "100", "raw": { "hoge": "huga" }, "res": { "hoge": "huga" } } } ※ ͜͜Ͱͷ @t ͸ type ͷུ
24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩू 24 SQL SELECT “${topicname} as topic, aws_lambda(“${bson2json lambda arn}”, { “bson”: encode(*, “base64”)) as data, get_thing_shadow(principal(), “${role arn}”) as thing FROM p/ @a.b Action Kinesis
25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 3. ෼ੳ৘ใऩूʹ͍ͭͯ • ޻෉఺ • ৘ใͷิ׬ • ୯७ͳ෼ੳͷΈͰ͸ͳ͘ɺݱࡏͷं྆ͷ৘ใ΋ؚΊͯૹ৴ • ෼ੳ؀ڥʹͯ৘ใͷิ׬͕ඞཁͳ͘ͳΔ • ඞཁͳ෦෼ͷΈͷ Lambda • Thing Shadow ͷऔಘͰ͢Β SQL 25
26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ޻෉఺ • Ͱ͖ΔݶΓϚωʔδϝϯτػೳΛ࢖͏ • Lambda ΋Ϛωʔδϝϯτ͕ͩɺ΋ͬͱ͍͍΋ͷ͸΋ͬͱ͋Δ • ཪଆͰಈ͍͍ͯΔ༷ʑͳγεςϜΛ༗ޮ׆༻͢Δ • ThingShadow ͕ߋ৽͞Εͨͱ͖ʹ͸৭ʑ Publish ͞ΕΔɻ • ࠓճ͸ update/documents ͷΈ. update/delta ΋৭ʑເ͕޿͕Δ • ৽͍͠ػೳ΋ϝϦοτΛߟ͑ͯ࢖͏ • SQL ͷؔ਺ͷҰͭͰ͋Δ aws_lambda ͸ൺֱత৽͍͠ػೳɻ • අ༻࡟ݮʹେ͍ʹ໾ཱͬͯ͘ΕͯΔ (͸͡Ί͸͍Βͳ͍ͳʔͬͯࢥͬͯͨ) 26
27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ຊεϥΠυ • ৘ใू໿ج൫γεςϜ಺ͷ
 ৘ใΛ෼ੳ͢Δ·Ͱͷ෦෼ͷൃදΛͤͯ͞௖͖·ͨ͠ • ͦͷଞ෦෼ʹڵຯ͋Γ·ͨ͠ΒҎԼΛࢀর͍ͩ͘͞ ⁃ AWS Summit 2018 Tokyo - DeNA ΦʔτϞʔςΟϒʹ͓͚Δ AWS ׆༻ࣄྫ ⁃ DeNA TechCon 2018 – AWS IoTΛ༻͍ͨDeNAΦʔτϞʔςΟϒΞʔΩςΫ νϟ ⁃ ݕࡧΩʔϫʔυʮAWS IoT DeNAʯ ※ ΞʔΩςΫνϟͷத਎͸೔ʑਐԽ͓ͯ͠Γ·͢ 27
28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ࣦഊஊ 28
29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (1) • Lambda ࢖͍·͘Γ • ੲ͸౷ҰԽΛؾʹͯ͠ɺશ෦ Lambda ͰॲཧΛͯͨ͠ • ं͔྆Βͷೖྗ͕ ਺ઍ rps. • தΛશ෦ lambda ʹ͢ΔͱඵؒສϨϕϧͷlambda͕࣮ߦ͞Ε͍ͯͨɻ • ߴ͍ɻ 29
30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (2) • ThingShadow ͷ index ࢖͍·͘Γ • ݕࡧͰ͖Δͷ͸ඇৗʹศརͩͬͨ (͜ͷ৚݅ͷं͚ͩநग़͍ͨ͠ͱ͔) • ਺ઍ rps ͷ index ͷߋ৽͕͔͔Δ • ߴ͍ɻ • ஗͘ͳΔʢ͸ͣʣ 30
31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix. ੲͷࣦഊ఺ (3) • Admin API (؅ཧܥͷAPI) ࢖͍·͘Γ • εϧʔϓοτΛٻΊͨ API ͡Όͳ͍ͷʹɺ਺ઍ rps Ͱόϯόϯୟ͍ͯ·ͨ͠ • έʔε͕࡞੒͞ΕΔͱ͍͏ܦݧΛ͠·ͨ͠ 31