OPENREC.tv におけるライブ動画およびメッセージ配信基盤の全貌 / 20170601_aws_devday_tokyo_openrec

2017/6/1　14:20 ～ 15:00　プリンスホール
[CyberZ] OPENREC.tvにおけるライブ動画およびメッセージ配信基盤のリプレース全貌
http://www.awssummit.tokyo/devday/?id=type#D3T8-3

1. OPENREC.tv ʹ͓͚Δ ϥΠϒಈը&ϝοηʔδ ഑৴ج൫ͷશ๴ AWS DevDay Tokyo 2017
2. Today’s Contents ຊηογϣϯΛ௨ͯ͡ʮϥΠϒʯʹٻΊΒΕΔΞʔΩςΫςΟϯά Λ͝঺հ͠·͢ ▸ OPENREC.tv ͱ͸ ▸ ಈը഑৴ͷجຊ ▸ ಈը഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ νϟοτͷجຊ ▸ ϦΞϧλΠϜϝοηʔδ഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ αʔϏεͷ੒௕ΛՃ଎ͤ͞ΔͨΊʹ
3. whoami ኍ੉ ଠ࿠ / Taro Hirose ▸ OPENREC.tv / CyberZ, Inc. ▸ Backend Engineer ▸ id: @uorat ▸ http://uorat.hatenablog.com
4. OPENREC.tv αʔϏε঺հ
5. OPENREC.tv ήʔϜʹಛԽͨ͠ಈը഑৴ϝσΟΞ
7. e-sports େձ΍ΦϦδφϧ൪૊΋഑৴
8. Service’s Growth DAU͸13ഒɾMAU͸110ສਓΛಥഁ http://gamebiz.jp/?p=180677
9. RAGE vol.4 GRAND FINAL 2017.06.10 (Sat) 10:00 - 19:30
10. Today’s Contents ຊηογϣϯΛ௨ͯ͡ʮϥΠϒʯʹٻΊΒΕΔΞʔΩςΫςΟϯά Λ͝঺հ͠·͢ ▸ OPENREC.tv ͱ͸ ▸ ಈը഑৴ͷجຊ ▸ ಈը഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ νϟοτͷجຊ ▸ ϦΞϧλΠϜϝοηʔδ഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ αʔϏεͷ੒௕ΛՃ଎ͤ͞ΔͨΊʹ
11. Today’s Contents ຊηογϣϯΛ௨ͯ͡ʮϥΠϒʯʹٻΊΒΕΔΞʔΩςΫςΟϯά Λ͝঺հ͠·͢ ▸ OPENREC.tv ͱ͸ ▸ ಈը഑৴ͷجຊ ▸ ಈը഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ νϟοτͷجຊ ▸ ϦΞϧλΠϜϝοηʔδ഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ αʔϏεͷ੒௕ΛՃ଎ͤ͞ΔͨΊʹ
12. Today’s Contents ຊηογϣϯΛ௨ͯ͡ʮϥΠϒʯʹٻΊΒΕΔΞʔΩςΫςΟϯά Λ͝঺հ͠·͢ ▸ OPENREC.tv ͱ͸ ▸ ಈը഑৴ͷجຊ ▸ ಈը഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ νϟοτͷجຊ ▸ ϦΞϧλΠϜϝοηʔδ഑৴ج൫ͷΞʔΩςΫνϟͱมભ ▸ αʔϏεͷ੒௕ΛՃ଎ͤ͞ΔͨΊʹ Live Streaming Real-Time
 Messaging
13. Live Streaming ಈը഑৴ͷجຊ
14. ಈը഑৴ େ͖͘ 2 छͷ༻్͕͋Δ ▸ ΦϯσϚϯυ഑৴ ▸ ࢹௌऀ͕ ݟ͍ͨ࣌ ʹಈըΛࢹௌ͢Δ͜ͱ͕Ͱ͖Δ഑৴ํ๏ ( abbr. VOD ) ▸ On-Demand = ʮཁٻʹԠͯ͡ʯ ▸ ૣૹΓ΍ר͖໭͠ͳͲͷ࠶ੜ੍ޚ͕Մೳ ▸ e.g. ౤ߘಈըɺΞʔΧΠϒಈը / λΠϜγϑτ ▸ ϥΠϒ഑৴ ▸ ϦΞϧλΠϜ ʹ഑৴͞Ε͍ͯΔಈըΛࢹௌ͢Δ͜ͱ͕Ͱ͖Δ഑৴ํ๏ ▸ ϦΞϧλΠϜࢹௌͳͷͰɺجຊతʹө૾ͷ࠶ੜ੍ޚ͸ߦ͑ͳ͍͜ͱ͕ଟ͍
15. ಈը഑৴ ഑৴ํࣜ΋େ͖͘ 2 छ͋Δ ▸ Progressive Download ▸ Ұͭͷ·ͱ·ͬͨϝσΟΞσʔλΛμ΢ϯϩʔυͤ͞Δํࣜ ▸ ϓϨΠϠʔ͸μ΢ϯϩʔυͰ͖ͨͱ͜Ζ͔Β࠶ੜ͢Δ ▸ γϯϓϧ͕ͩ՝୊͋Γ ▸ ϥΠϒ഑৴ඇରԠ ▸ ϝσΟΞϑΝΠϧͦͷ΋ͷΛ௚઀μ΢ϯϩʔυͤ͞ΔͨΊίϯςϯπอޢͷ؍఺Ͱ೉͋Γ ▸ Streaming ▸ ϝσΟΞσʔλΛࡉ͔͘෼ׂͯ͠ɺॱ࣍సૹ / μ΢ϯϩʔυͤ͞Δํࣜ ▸ Progressive Download ͱࣅͨΑ͏ʹɺμ΢ϯϩʔυͱฒߦͯ͠࠶ੜ͢Δ ▸ ࢹௌதʹσʔλ͕ੜ੒͞Εଓ͚ΔϥΠϒ഑৴ʹରԠՄೳ ▸ ໪࿦ΦϯσϚϯυ഑৴΋Մೳ
16. ಈը഑৴ ഑৴ํࣜ΋େ͖͘ 2 छ͋Δ ▸ Progressive Download ▸ Ұͭͷ·ͱ·ͬͨϝσΟΞσʔλΛμ΢ϯϩʔυͤ͞Δํࣜ ▸ ϓϨΠϠʔ͸μ΢ϯϩʔυͰ͖ͨͱ͜Ζ͔Β࠶ੜ͢Δ ▸ γϯϓϧ͕ͩ՝୊͋Γ ▸ ϥΠϒ഑৴ඇରԠ ▸ ϝσΟΞϑΝΠϧͦͷ΋ͷΛ௚઀μ΢ϯϩʔυͤ͞ΔͨΊίϯςϯπอޢͷ؍఺Ͱ೉͋Γ ▸ Streaming ▸ ϝσΟΞσʔλΛࡉ͔͘෼ׂͯ͠ɺॱ࣍సૹ / μ΢ϯϩʔυͤ͞Δํࣜ ▸ Progressive Download ͱࣅͨΑ͏ʹɺμ΢ϯϩʔυͱฒߦͯ͠࠶ੜ͢Δ ▸ ࢹௌதʹσʔλ͕ੜ੒͞Εଓ͚ΔϥΠϒ഑৴ʹରԠՄೳ ▸ ໪࿦ΦϯσϚϯυ഑৴΋Մೳ VOD / LIVE ͍ͣΕ΋ Streaming ഑৴͕τϨϯυ
17. Broadcaster / Viewer Live Streaming ഑৴ʹ͍ͭͯ ഑৴ऀଆͱࢹௌऀଆʹཱͬͯݟͯΈΔ Broadcaster Media Server Viewer
18. Live Streaming: Broadcaster Side RTMP w/ H.264/AAC ͷ૊Έ߹Θ͕ͤओྲྀ ▸ RTMP = Real-Time Messaging Protocol ▸ ࣋ଓ઀ଓܕϓϩτίϧ ▸ ಈըɾԻ੠σʔλΛ෼ׂͯ͠సૹ ▸ Τϯίʔυ ▸ ΫϥΠΞϯτଆͰө૾ & Ի੠σʔλΛѹॖ ▸ ө૾ & Ի੠σʔλͷѹॖํࣜ͸༷ʑ͋Δ ▸ ಈըίʔσοΫ: H.264/MPEG-4 AVC, H.265/HEVC, … ▸ Ի੠ίʔσοΫ: MP3, AAC, WMA, …
19. Live Streaming: Broadcaster Side Α͘࢖ΘΕΔ഑৴ιϑτ΢ΣΞ ▸ RTMP = Real-Time Messaging Protocol ▸ ࣋ଓ઀ଓܕϓϩτίϧ ▸ ಈըɾԻ੠σʔλΛ෼ׂͯ͠సૹ ▸ Τϯίʔυ ▸ ΫϥΠΞϯτଆͰө૾ & Ի੠σʔλΛѹॖ ▸ ө૾ & Ի੠σʔλͷѹॖํࣜ͸༷ʑ͋Δ ▸ ಈըίʔσοΫ: H.264/MPEG-4 AVC, H.265/HEVC, … ▸ Ի੠ίʔσοΫ: MP3, AAC, WMA, … OBS Wirecast
20. Live Streaming: Viewer Side HTTP ετϦʔϛϯάܕ͕ओྲྀ ▸ ετϦʔϛϯάܕ ▸ ઐ༻ϓϨΠϠʔ΍ϓϥάΠϯ͕ඞཁͱͳΔ ▸ RTMP w/ Adobe Flash ▸ RTSP/RTP w/ Windows Media, ReadMedia, QuickTime …etc ▸ MMS w/ Windows Media ▸ େྔϢʔβʔ΁ͷ഑৴ʹ͸େྔͷ Edge Server ͕ඞཁͱͳΔ ▸ HTTP ετϦʔϛϯάܕ ▸ ෼ׂ͞ΕͨܰྔͳϝσΟΞϑΝΠϧΛ HTTP Ͱ഑৴ ▸ ϒϥ΢β / σόΠεඪ४Ͱ࠶ੜՄೳʢͳ͜ͱ͕ଟ͍ʣ ▸ HLS ▸ MPEG-DASH ▸ HDS ▸ Smooth Streaming ▸ Ұൠతͳ Caching ٕज़ΛԠ༻Մೳ ▸ ݴ͍׵͑Ε͹ͨͩͷ HTTP ϑΝΠϧμ΢ϯϩʔυ
21. Live Streaming: Viewer Side HTTP ετϦʔϛϯάܕ͕ओྲྀ ▸ ετϦʔϛϯάܕ ▸ ઐ༻ϓϨΠϠʔ΍ϓϥάΠϯ͕ඞཁͱͳΔ ▸ RTMP w/ Adobe Flash ▸ RTSP/RTP w/ Windows Media, ReadMedia, QuickTime …etc ▸ MMS w/ Windows Media ▸ େྔϢʔβʔ΁ͷ഑৴ʹ͸େྔͷ Edge Server ͕ඞཁͱͳΔ ▸ HTTP ετϦʔϛϯάܕ ▸ ෼ׂ͞ΕͨܰྔͳϝσΟΞϑΝΠϧΛ HTTP Ͱ഑৴ ▸ ϒϥ΢β / σόΠεඪ४Ͱ࠶ੜՄೳʢͳ͜ͱ͕ଟ͍ʣ ▸ HLS ▸ MPEG-DASH ▸ HDS ▸ Smooth Streaming ▸ Ұൠతͳ Caching ٕज़ΛԠ༻Մೳ ▸ ݴ͍׵͑Ε͹ͨͩͷ HTTP ϑΝΠϧμ΢ϯϩʔυ Amazon CloudFront
22. Live Streaming: Viewer Side Mobile first, Multi Device.
23. Live Streaming: Viewer Side Adaptive Bitrate Streaming ▸ Ϣʔβʔͷճઢঢ়گʹΑͬͯը࣭Λࣗಈ ௐ੔͢Δٕज़ ▸ ଳҬ෯͕ણࡉͳ Mobile Ͱ͸ඞਢ
24. Live Streaming: Viewer Side Adaptive Bitrate Streaming ▸ ഑৴༻ϝσΟΞϑΝΠϧΛ਺छྨͷը࣭ Ͱ༻ҙ͓ͯ͘͠ 1080p 720p 360p
25. Live Streaming: Viewer Side Adaptive Bitrate Streaming ▸ ଳҬ෯͕ڱ͍࣌͸௿ը࣭Λ࠾୒ 1080p 720p 360p
26. Live Streaming: Viewer Side Adaptive Bitrate Streaming ▸ ଳҬ෯ʹԠͯ͡ը࣭Λ੾Γସ͑ 1080p 720p 360p
27. Media Server Transcoding Live Streaming: Transcoding ड৴ͨ͠ϝσΟΞσʔλΛ഑৴༻σʔλʹม׵ 1080p 720p 360p 1080p 720p 360p 1080p 720p 360p HLS MPEG-DASH Smooth Streaming
28. Media Server Transcoding Live Streaming: Transcoding Middleware for Live Transcoding 1080p 720p 360p 1080p 720p 360p 1080p 720p 360p HLS MPEG-DASH Adobe Media Server LIVE Smooth Streaming
29. Live Streaming ҰൠతͳϥΠϒಈը഑৴ͷΞʔΩςΫνϟ Broadcaster Media Server (Ingest / Transcode) Viewer CDN
30. Live Streaming: New Generation ࠷ۙͰ͸ WebRTC Λ༻͍ͨಈը഑৴΋
31. Live Streaming: New Generation ࠷ۙͰ͸ WebRTC Λ༻͍ͨಈը഑৴΋
32. Time-Shifting ࢹௌऀ͕ ݟ͍ͨ࣌ ʹಈըΛࢹௌ͢Δ͜ͱ͕Ͱ͖ΔΑ͏ʹ͢Δ͜ͱ Broadcaster Media Server (Ingest / Transcode) Viewer CDN rtmp http http(s)
33. Time-Shifting Live Streaming >> Stop Broadcaster Media Server (Ingest / Transcode) Viewer CDN media file (e.g. mp4)
34. Time-Shifting Live Streaming >> Stop >> Archiving Broadcaster Media Server (Ingest / Transcode) Viewer CDN media file (e.g. mp4) transcode
35. Time-Shifting Live Streaming >> Stop >> Archived = VOD Broadcaster Media Server (Ingest / Transcode) Viewer CDN media file (e.g. mp4) http(s) http
36. ಈը഑৴ͷجຊݪଇ ʮ͓͕͔͔ۚΔʯ ▸ Processing ▸ τϥϯείʔσΟϯάʹཁ͢Δύϫϑϧͳ Processing Resource ▸ Memory ▸ τϥϯείʔσΟϯά࣌ͷ Buffer ▸ ഑৴ Caching ▸ Network ▸ ςΩετσʔλͱൺ΂ͯαΠζͷେ͖͍ϝσΟΞσʔλ ▸ ෆಛఆଟ਺ͷࢹௌऀ ▸ Storage ▸ ςΩετσʔλͱൺ΂ͯαΠζͷେ͖͍ϝσΟΞσʔλͷӬଓԽ
37. OPENREC.tv Case of
38. Case: OPENREC.tv OPENREC.tv ͷϥΠϒ഑৴ͷཁ݅ ▸ ߴը࣭ɾ௿஗Ԇ ▸ ϢʔβʔυϦϒϯͷ഑৴ ▸ ಉ࣌഑৴਺΋഑৴࣌ؒ΋഑৴ऀ࣍ୈ ▸ ू٬ྗ΋഑৴ऀ࣍ୈ ▸ ∴ ෛՙ͕ಡΈͮΒ͍ ▸ ಉ࣌ࢹௌऀ਺ ແ੍ݶ ▸ ͍ΘΏΔ “࿮” ͸ແ͍ ▸ શϢʔβʔ౳͘͠ϥΠϒࢹௌͰ͖Δ͜ͱ ▸ શϥΠϒ഑৴ ΞʔΧΠϒ ▸ ͍ΘΏΔ “Time Shifting”, VODԽ ▸ ΞʔΧΠϒಈըΛެ։͢Δ͔Ͳ͏͔͸഑৴ऀͷࣗ༝
39. Case: OPENREC.tv Architecture: Wowza on EC2 (c4, g2), CloudFront, S3, ElasticTranscoder Aurora LIVE API https LIVE CORE Broadcastrer ELB EC2 RDS EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution rtmp Broadcastrer rtmp Viewer HLS (https) LIVE ARCHIVE CloudFront Distribution S3 bucket PutObjects Broadcastrer VOD PutObjects CloudFront Distribution S3 bucket ElasticTranscoder
40. Case: OPENREC.tv Processing Layer Aurora LIVE API https LIVE CORE Broadcastrer ELB EC2 RDS EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution rtmp Broadcastrer rtmp Viewer HLS (https) LIVE ARCHIVE CloudFront Distribution S3 bucket PutObjects Broadcastrer VOD PutObjects CloudFront Distribution S3 bucket ElasticTranscoder ▸ GPU Acceleration ▸ w/ g2 instance ▸ ಉ࣌഑৴਺ʹԠ֦ͨ͡ுॖୀ੍ޚ ▸ w/ EC2 API
41. Case: OPENREC.tv CPU ࢖༻཰ ͕ 50 ʙ 70% ఔ౓ʹ཈͑ΒΕΔΑ͏ʹ഑৴Λ෼ࢄ Aurora LIVE API https LIVE CORE Broadcastrer ELB EC2 RDS EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution rtmp Broadcastrer rtmp Viewer HLS (https) LIVE ARCHIVE CloudFront Distribution S3 bucket PutObjects Broadcastrer VOD PutObjects CloudFront Distribution S3 bucket ElasticTranscoder ▸ GPU Acceleration ▸ w/ g2 instance ▸ ಉ࣌഑৴਺ʹԠ֦ͨ͡ுॖୀ ੍ޚ ▸ w/ EC2 API
42. Case: OPENREC.tv Network Layer Aurora LIVE API https LIVE CORE Broadcastrer ELB EC2 RDS EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution rtmp Broadcastrer rtmp Viewer HLS (https) LIVE ARCHIVE CloudFront Distribution S3 bucket PutObjects Broadcastrer VOD PutObjects CloudFront Distribution S3 bucket ElasticTranscoder All transfer in AWS ▸ CDN Caching ʹΑΔ Origin ΁ͷ Request ݮ ▸ CDN Edge - Origin ؒͷ Latency ▸ Origin → CDN ؒͷ σʔλసૹඅ༻ ▸ CDN σʔλసૹඅ༻ͷ Volume Discount
43. Replaces
44. 2014 - 2015 ϓϨΠಈը࿥ը SDK ࣌୅ ▸ εϚϗಛԽͷϓϨΠಈըڞ༗αʔϏε ▸ iOS / Android ͔Β ࿥ը / ฤू / ౤ߘ ▸ ࠾༻ ▸ ౰࣌ͷ؊ͷ SDK ։ൃʹ஫ྗ
45. 2015 - 2016 ಈը౤ߘ → LIVE ഑৴΁ ▸ খ࢝͘͞ΊΔ ▸ ϦʔυλΠϜ࡟ݮͷͨΊ Zencoder Live Λ׆༻ ▸ େ͖͘ҭͯΔ ▸ 2016 ೥ʹೖΔͱ LIVE ʹ஫ྗ ▸ ߴը࣭ɾ௿஗Ԇɾ௿අ༻ʹ޲͖߹͏ LIVE
46. 1st Replace in 2016 Mid Replace Transcoder Aurora LIVE API https LIVE CORE Broadcastrer ELB EC2 RDS EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution rtmp Broadcastrer rtmp Viewer HLS (https) LIVE ARCHIVE CloudFront Distribution S3 bucket PutObjects Broadcastrer VOD PutObjects CloudFront Distribution S3 bucket ElasticTranscoder
47. 2016 Mid - Later ΞʔΧΠϒػೳͷ՝୊ײ ▸ VODԽʹཁ͢ΔϦʔυλΠϜ ▸ ϥΠϒऴྃޙʹύοέʔδϯά͞Εͨ mp4 Λ ETS Ͱ ࠶ Transcode ͯ͠ S3 ʹӬଓԽ ▸ ഑৴͕24࣌ؒʹ΋ͳΔͱ໨΋౰ͯΒΕͳ͍ ▸ Elastic Transcoder ͷඅ༻૿ ▸ ΞʔΧΠϒର৅͸ શϥΠϒ഑৴ ▸ ެ։/ඇެ։͸ ഑৴ऀͷݖར ▸ ྉۚ͸ ಈը਺ × ഑৴࣌ؒ ͷֻ͚߹Θͤ
48. 2nd Replace in 2016 Later Extract by Wowza MPEG-TS Ingestion Listner API Broadcaster Media Server (Ingest / Transcode) Viewer CDN http(s) On Memory Extract plugin rtmp http
49. 2nd Replace in 2016 Later Extract by Wowza MPEG-TS Ingestion Listner API ▸ Wowza Streaming Engine v4.5.0 or later ▸ ϥΠϒ഑৴தϝϞϦ্ʹ͋Δ HLS Λநग़ ▸ ଈ࣌ΞʔΧΠϒԽΛ࣮ݱ ▸ ΞʔΧΠϒԽͷͨΊͷ࠶ Transcoding අ ༻Λେ෯࡟ݮ ▸ Elastic Transcoder ͸౤ߘಈը΍ɺෆ҆ఆͳϥΠϒಈ ըͷม׵ʹҾ͖ଓ͖ར༻
50. 2017 Early ن໛ͷ֦େ ▸ ಉ࣌ϥΠϒ഑৴਺ ૿Ճ ▸ 2016 Mid: ʙ ਺े ▸ 2017 Mid: ʙ ਺ඦ ▸ ͜Εʹ൐͍ αʔό਺΋૿Ճ ▸ ϩʔϦϯά σϓϩΠ / ϝϯςφϯεͷ൥Θ͠͞ ▸ ϥΠϒ഑৴Λड͚෇͚ͨ Wowza αʔόΛϥΠϒऴྃ·Ͱམͱͤͳ͍ ▸ ഑৴ऀ͔Βͷ઀ଓ͸࣋ଓ઀ଓ RTMP ▸ ϢʔβʔओಋͷϥΠϒ഑৴͸ऴ͕ྃ࣌ؒಡΊͳ͍ LIVE CORE EIP Wowza on EC2 CloudFront Distribution EIP Wowza on EC2 CloudFront Distribution
51. Next 3rd Replace Wowza αʔόΛشൃԽ Amazon ECR (Container Registory) Developer ECS Cluster Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container)
52. Next 3rd Replace Expire >> Drain >> Stop Container Amazon ECR (Container Registory) Developer ECS Cluster Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container)
53. Next 3rd Replace Expire >> Drain >> Stop Container Amazon ECR (Container Registory) Developer ECS Cluster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container)
54. Next 3rd Replace Expire >> Drain >> Stop Container Amazon ECR (Container Registory) Developer ECS Cluster Container Instance Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container)
55. Next 3rd Replace ഑৴਺ʹԠͯ͡ Container ͔Β Instance ͔Β EIP/DNS Record ·Ͱ AutoScale Amazon ECR (Container Registory) Developer ECS Cluster RunTask Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Container Instance
56. Next 3rd Replace ϦϦʔε࡞ۀ͸ `docker image push` ͷΈ Amazon ECR (Container Registory) Developer docker image push ECS Cluster Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Container Instance
57. Next 3rd Replace ͋ͱ͸৽Πϝʔδ͕উखʹਁಁ͢ΔͷΛ଴͚ͭͩ Amazon ECR (Container Registory) Developer docker image push ECS Cluster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) RunTask
58. Next 3rd Replace Container Scheduler ▸ EC2/ECS Auto Scaling ͸૬ੑ͕ѱ͍ ▸ RTMP = ৗ࣌઀ଓ ▸ ෛՙ͕௿ͯ͘΋഑৴͍ͯ͠Ε͹ॖୀͰ͖ͳ͍ ▸ “഑৴ঢ়گ” ͱ͍͏ಠࣗࢦඪʹج͍ͮͯ εέʔϧͤ͞Δεέδϡʔϥʔ͕ඞཁ
59. Container Scheduler 1. ▸ Container Scheduler for Amazon ECS ▸ re:Invent 2016 Ͱެ։͞Εͨ golang ੡ OSS ▸ ECS Cluster ༻ͷΧελϜεέδϡʔϥΛ࣮૷Մೳ ▸ ECS Cluster ͷΠϕϯτݕ஌ ▸ ECS Cluster ͷঢ়ଶ௥੻ ▸ ΧελϜεέδϡʔϥʔͷ࣮ߦ ▸ REST API ͷఏڙ
60. Container Scheduler 2. Amazon ECS Event Stream + Lambda ▸ Amazon ECS ͷλεΫͱίϯςφΠϯελϯεͷঢ়ଶߋ৽Λ hook ʹ Lambda function Λ࣮ߦ Amazon ECS CloudWatch Events Lambda Event Stream Event
61. Next 3rd Replace 3. CloudWatch Events (scheduled/1min) + Lambda + API ECS Cluster Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Container Instance Aurora LIVE API ELB EC2 RDS CloudWatch Events scheduled 1 minute ec2::RunInstances ecs::RunTask ecs::StopTask ec2::StopInstances …
62. Next 3rd Replace 3. CloudWatch Events (scheduled/1min) + Lambda + API ECS Cluster Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Broadcaster Container Instance Task (Container) Container Instance Aurora LIVE API ELB EC2 RDS CloudWatch Events scheduled 1 minute ec2::RunInstances ecs::RunTask ecs::StopTask ec2::StopInstances … Coming soon…
63. Messaging Chat
64. OPENREC.tv ʹ͓͚Δ Chat Offline - Online ؒͷۭؒڞ༗ ▸ ձ৔ͷྟ৔ײΛ Online Ͱٙࣅମݧ ▸ e.g. e-sports Πϕϯτ RAGE ▸ Communication ▸ ഑৴ऀ - ࢹௌऀ ؒ ▸ ࢹௌऀ - ࢹௌऀ ؒ
65. Case: Minecraft
66. OPENREC.tv ͷνϟοτͷཁ݅ ϦΞϧλΠϜϝοηʔδ഑৴ج൫ ▸ ϦΞϧλΠϜੑ ▸ Online - Offline ؒͷྟ৔ײͷڞ༗ ▸ ഑৴ऀ - ࢹௌऀ, ࢹௌऀ - ࢹௌऀ ؒͷॏཁͳର࿩खஈ ▸ ߴ֦ுੑ ▸ ಉ࣌ࢹௌऀ਺૿ʹ௥ैͰ͖ΔεέʔϥϏϦςΟ ▸ ໷ؒʹूத͢ΔϥΠϒ഑৴
67. OPENREC’s Chat in 2015 1st Architecture: Node.js (WebSocket) + Redis (Pub/Sub) + API (ELB / EC2) Chat API https WebSocket Cluster ELB EC2 RDS EIP Node.js on EC2 EIP Node.js on EC2 ElastiCache for Redis (Pub/Sub) wss wss Viewer
68. Chat API https WebSocket Cluster ELB EC2 RDS EIP Node.js on EC2 EIP Node.js on EC2 ElastiCache for Redis (Pub/Sub) wss wss Viewer OPENREC’s Chat in 2015 1st Architecture: Node.js (WebSocket) + Redis (Pub/Sub) + API (ELB / EC2) ▸ WebSocket ઀ଓΛࣗ࡞ API Ͱ෼ࢄ ▸ ELB (Classic) ͕ WebSocket ඇରԠͷͨΊ
69. Chat API https WebSocket Cluster ELB EC2 RDS EIP Node.js on EC2 EIP Node.js on EC2 ElastiCache for Redis (Pub/Sub) wss wss Viewer OPENREC’s Chat in 2015 1st Architecture: Node.js (WebSocket) + Redis (Pub/Sub) + API (ELB / EC2) ▸ API ͔Βฦ͞ΕΔ endpoint ʹ WebSocket ઀ଓཱ֬ ▸ Message Hub ʹ ElastiCache for Redis Λ࠾༻
70. OPENREC’s Chat After 1 year … Chat API https WebSocket Cluster ELB EC2 RDS EIP Node.js on EC2 EIP Node.js on EC2 ElastiCache for Redis (Pub/Sub) wss wss Viewer ൥Θ͕͠͞෾ग़ ▸ Not AutoScaling ▸ EIP + DNS record ▸ SSL αʔό ূ໌ॻߋ৽ Redis Cluster Λ෼ࢄ͍ͤͨ͞ ▸ Pub/Sub ૿
72. WebSocket Content Based Routing
73. OPENREC’s Chat Replaced Architecture Chat API https WebSocket Cluster ELB (CLB) EC2 RDS ELB (ALB) wss Viewer Aurora Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group
74. OPENREC’s Chat Replaced Architecture Chat API https WebSocket Cluster ELB (CLB) EC2 RDS ELB (ALB) wss Viewer Aurora Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group ▸ Contents Based Routing Λ׆༻ͨ͠ Sharding ▸ wss ઀ଓऴ୺ʹΑΔӡ༻ ෛՙݮ
75. OPENREC’s Chat Replaced Architecture Chat API https WebSocket Cluster ELB (CLB) EC2 RDS ELB (ALB) wss Viewer Aurora Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group Node.js Cluster w/ AutoScaling ElastiCache for Redis (Pub/Sub) Target Group ▸ AutoScaling ׆༻ʹΑΔӡ༻ෛՙݮ ▸ Middleware Version up ▸ Node.js ▸ Socket.IO
76. Results αʔϏεෛՙʹରͯ͠ॊೈͳScaling Λ࣮ݱ
77. Point of WebSocket Application w/ AutoScaling Scaling ʹ͓͍ͯز͔ͭͷ஫ҙ͕ඞཁ ▸ ૣΊʹεέʔϧΞ΢τͤ͞Δ ▸ WebSocket = ࣋ଓ઀ଓܕϓϩτίϧ ▸ ෛՙ͕ภΓ͗͢ͳ͍Α͏ʹૣΊʹεέʔϧΞ΢ τͤ͞Δͱྑ͍ ▸ ॏཁͳࢦඪ ▸ ಉ࣌઀ଓ਺ ▸ ϝοηʔδྲྀྔʢPub/Sub ස౓ʣ ▸ ϝοηʔδαΠζ
78. Point of WebSocket Application w/ AutoScaling Scaling Policy ▸ γϯϓϧʹ CPU Utilization ͕͓͢͢Ί ▸ ᮢ஋Λݫ͠Ίʹ ૣΊʹ Scale out ▸ Increase ͸හײʹ Decrease ͸؇͘ ▸ ϐʔΫଳ΍Πϕϯτ͕ಡΊΔ৔߹͸ Scheduled Policy ͕༗ޮ
79. Caution of ALB w/ WebSocket ▸ ༧ظͤ͵࠶઀ଓʹඋ͑Δ ▸ ALB ಺෦ϊʔυ͕ Scale up / out ͢ΔλΠϛϯάͰҰ੪੾அͱ࠶઀ଓ͕૸Δ ▸ onConnect, onDisconnect ͷίʔϧόοΫॲཧ͸ྲྀྔ੍ޚ͓ͯ͘͠ͱྑ͍ ▸ Stickness ༗ޮԽ ( Socket.IO ͷ৔߹) ▸ Handshake ॲཧ ʙ Upgrade ·Ͱͷ xhr-polling ௨৴࣌ʹඞཁ ▸ client ʹΑͬͯ͸ Upgrade ࣌ʹ Cookie ͕౉Βͣ Stickness ͕ޮ͔ͳ͍ࣄ͕͋Δ ▸ socket.io-client-swift, socket.io-client-java Ͱ֬ೝ ▸ جຊ Stickness Λ༗ޮԽ͠ɺϥΠϒϥϦʹԠͯ͡ forceWebSocket ͯ͠ճආ͢Δ
80. Requests to ALB ▸ Sticky Session ͷ Custom Cookie ࢦఆ ▸ CLB Ͱ͸Մೳ ▸ ALB ಺෦ͷεέʔϦϯάͷಈ࡞ΛγϡϛϨʔγϣϯ͍ͨ͠ ▸ ಺෦ϊʔυͷ Scale up / out / in ͕ൃੜͨ͠৔߹ͷɺΞϓϦέʔγϣϯ૚ʹ༩͑ΔӨڹ֬ೝ ▸ Leastconn Balancing Rule ▸ ࠷΋઀ଓ਺ͷগͳ͍όοΫΤϯυʹ༏ઌతʹ Balancing ͍ͨ͠ ▸ ಛʹ WebSocket ͷΑ͏ͳৗ࣌઀ଓܕ௨৴ͷෛՙΛۉ౳෼ࢄ͍ͨ͠
81. Summary
82. Grow larger, Start small OPENREC.tv
83. RUNNING LEAN ελʔτΞοϓʹ͓͚Δ3ͭͷεςʔδ ՝୊ / ղܾ ϑΟοτ ੡඼ / ࢢ৔ ϑΟοτ ֦େ ूதɿݕূʹΑΔֶश ࣮ݧɿϐϘοτ ूதɿ੒௕ ࣮ݧɿ࠷దԽ
84. RUNNING LEAN ֶशʹඞཁͳ ࠷΋ඞཁͳ͜ͱΛ΍Δ ՝୊ / ղܾ ϑΟοτ ੡඼ / ࢢ৔ ϑΟοτ ֦େ ूதɿݕূʹΑΔֶश ࣮ݧɿϐϘοτ ूதɿ੒௕ ࣮ݧɿ࠷దԽ ▸ ήʔϜϓϩϞʔγϣϯͷ໰୊ͱޮՌ޲্ͱ͍͏໋୊ ▸ ϓϨΠಈըڞ༗ͱ͍͏ݕূ ▸ εϚϗಛԽ / SaaS ׆༻ / SDK ूத
85. RUNNING LEAN ֶशʹඞཁͳ ࠷΋ඞཁͳ͜ͱΛ΍Δ ՝୊ / ղܾ ϑΟοτ ੡඼ / ࢢ৔ ϑΟοτ ֦େ ूதɿݕূʹΑΔֶश ࣮ݧɿϐϘοτ ूதɿ੒௕ ࣮ݧɿ࠷దԽ ▸ LIVE ഑৴ʹϐϘοτ ▸ SaaS ׆༻, 2 week Chat࣮૷, UI/UXूத
86. RUNNING LEAN Grow larger, Small Start ՝୊ / ղܾ ϑΟοτ ੡඼ / ࢢ৔ ϑΟοτ ֦େ ूதɿݕূʹΑΔֶश ࣮ݧɿϐϘοτ ूதɿ੒௕ ࣮ݧɿ࠷దԽ ▸ Ϣʔβʔٸ૿ ▸ LIVE & CHAT ج൫ͷ࠷దԽ
87. LEAN STARTUP ։ൃ - ܭଌ - ֶश ͷαΠΫϧΛ೗Կʹ଎͘ճͤΔ͔ IDEAS CODE DATA LEARN MEASURE BUILD
88. Why rebuild 4 times? “The Perfect is often the enemy of the good”
89. Why rebuild 4 times? “The Perfect is often the enemy of the good” Ϗδωε໨ඪΛୡ੒͢ΔͨΊͷ ޮ཰తͳಓےΛඳ͘͜ͱ