Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalable Microservice for Media Transcoding / T...
Search
atsushisakai
March 23, 2019
Programming
4
1.5k
Scalable Microservice for Media Transcoding / Transcoder
Rails Developers Meetup 2019の発表に利用した資料です。
atsushisakai
March 23, 2019
Tweet
Share
More Decks by atsushisakai
See All by atsushisakai
大規模 SaaS の技術的意思決定を支える三要素 / Three elements that support technical decision-making for large-scale SaaS
_atsushisakai
0
1.1k
一方、私はなぜEMでありつづけるか / TECH TRACK vol.1
_atsushisakai
0
450
子育てとEMと転職と
_atsushisakai
2
810
「家族アルバム みてね」を支える エンジニアリング組織/introduce-familyalbum-engineering-team
_atsushisakai
2
230
ミクシィのマネージャーは悩んでいる / mixi's manager is in trouble
_atsushisakai
16
14k
実践 Engineering Manager / practice engineering manager
_atsushisakai
19
8.1k
New Relic {FUTURE} STACK 3/14, 2019 / newrelic_future_stack
_atsushisakai
1
1.5k
実践 WorkManager / WorkManager
_atsushisakai
3
7.4k
How to use New Relic in Mitene Development Team
_atsushisakai
0
590
Other Decks in Programming
See All in Programming
英語文法から学ぶ、クリーンな設計の秘訣
newnomad
1
280
Kamal 2 – Get Out of the Cloud
aleksandrov
0
140
Devinのメモリ活用の学びを自社サービスにどう組み込むか?
itarutomy
0
1.9k
custom_lintで始めるチームルール管理
akaboshinit
0
190
Go1.24で testing.B.Loopが爆誕
kuro_kurorrr
0
170
AtCoder Heuristic First-step Vol.1 講義スライド
terryu16
3
1.1k
リアクティブシステムの変遷から理解するalien-signals / Learning alien-signals from the evolution of reactive systems
yamanoku
2
1.2k
地域ITコミュニティの活性化とAWSに移行してみた話
yuukis
0
180
PHPのガベージコレクションを深掘りしよう
rinchoku
0
260
S3静的ホスティング+Next.js静的エクスポート で格安webアプリ構築
iharuoru
0
210
なぜselectはselectではないのか
taiyow
2
320
マルチアカウント環境での、そこまでがんばらない RI/SP 運用設計
wa6sn
0
670
Featured
See All Featured
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Gamification - CAS2011
davidbonilla
81
5.2k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
177
52k
Thoughts on Productivity
jonyablonski
69
4.5k
Being A Developer After 40
akosma
90
590k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
129
19k
Visualization
eitanlees
146
16k
Designing for humans not robots
tammielis
251
25k
The Cult of Friendly URLs
andyhume
78
6.3k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
A designer walks into a library…
pauljervisheath
205
24k
Transcript
Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails
Developers Meetup 2019
ञҪ ಞ @_atsushisakai • ՈΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on
Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͡Ίͯ
Agenda ʮΈͯͶʯͰ͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝ͩͬͨͷ͔ʁ • Ͳ͏ͬͯղܾͨ͠ͷ͔ʁ
·ͣͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
ಈը(WIP)
͜ΕҰମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲͷΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞ΕͨಈըࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞ • ൢച͢ΔDVDʹऩͨ͠ΓɺΞϓϦʹ৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ͚ͯࢠҭͯͷࢥ͍ग़Λ࡞ʹ͢Δ •
͍ಈըΛͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢͳͲͷը૾ΛΓ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
ಈը(WIP)
1ඵಈըΛࣗಈͰେྔʹ࡞Δ
͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͚ͭΕΔ • ಈըͷ݁߹Ͱ͖Δ
• Ͱ͖ͳ͍͜ͱ • ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Իͱͷ߹ͱ͍͏ͷͰ͖ͳ͍
࡞Δ͔͠ͳ͍…
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%
͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ • ffmpegʹΑΔมॲཧΛྻʹ࿈݁ͨ͠ͷ • લͷॲཧͷΞτϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
Έͳ͞ΜͳΒͲ͏࣮͠·͔͢❓
࠷۪ͳ࣮
࠷۪ͳ࣮ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ͕ग़ͯ͘Δ • ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ৴ରͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷग़ͯ͘Δ
• CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͍ͬͨͳ͍ • ޮతͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮ ύΠϓϥΠϯΛಉ࣌ʹෳΒͤΔͱɺ͖ͬͱύϑΥʔϚϯεྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͋ΔఔΑ͘ͳΔ͔͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧͦ͠͏
None
Sidekiq Rubyͷ Job Queue γεςϜͰͱͯ༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ100Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ
• https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯগͳ͍ • ๛ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
Sidekiq
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔΛ࡞Δ • ෳͷ
Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
None
ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛͬͯBʙF͕ىಈ • BʙFύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓ɺ͜Ε…ʁʁ
https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
Ͳ͏ͬͯ࡞Δʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ Sidekiq BatchΛ͏͜ͱͰ
తͷύΠϓϥΠϯ͕࣮Ͱ͖ͦ͏‼
ύΠϓϥΠϯ࣮Ͱ͖Δ͔ʁ ׂͱཧʹ͍ۙ☺ • جຊతʹͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά؆୯ • ޮతͳϦτϥΠ͕ߦΘΕΔ
• SidekiqͷฒྻઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴԽ͍ͯ͘͠ • CPU͕Γͳ͚Εεέʔϧͤ͞Δ
Transcoder
ϚΠΫϩαʔϏεͱͯ͠ͷ࣮ • ผαʔϏε͔ΒTranscoderͷϦΫΤετAWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁ՌAWS SQSʹͤͯฦ͢ •
தؒϑΝΠϧՌAWS S3ʹอଘ͢Δ • αʔϏεؒΦϒδΣΫτ໊ͷΓͱΓͷΈ • ࣮ϑΝΠϧ͏ଆ͕औΓʹߦ͘
αʔϏεͷεέʔϥϏϦςΟ • ffmpegϞϦϞϦͱCPUΛ͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ
Transcoder γεςϜߏ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷΓऔΓͷՕॴͰখن ʹRails͕ಈ͘γεςϜ
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 1 2 3 4 5
6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2
3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2
3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜɾ࠷ऴՌΛอ
ଘ͢ΔS3όέοτ 1 2 3 4 5 6
Transcoder ͷΞʔΩςΫνϟ JobͱResult • JobɺSQSʹͤΔJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoderඇಉظʹॲཧΛ։࢝͢Δ •
ResultɺSQSʹͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ߹ʹErrorResultΛฦ͢
None
None
None
Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮ • = Sidekiq Batchͷ࣮ͦͷͷɻ͜ΕࣗମSidekiq Worker
• ෦ʹෳͷStepΛ࣋ͭ • Pipeline͝ͱʹՌ͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •
Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
Transcoder ͷΞʔΩςΫνϟ Worker • StepͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͯ͢Sidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊffmpegͷίϚϯυΛҰճ࣮ߦ͢ΔཻͰ࣮͢Δ
• ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧS3ʹอଘ͢Δ
Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
Pipeline Step Step Step Worker + Command
Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ
Sidekiq Batch࣮ͷίπ • Batch෦ͷWorkerґଘΛ͘ɺཻ͔ͭΛখ࣮͘͢͞Δɻ • εέʔϧ͍͢͠ɺϦτϥΠ͍͢͠ɺͳͲͷతɻ • ෳࡶͳBatchΛ࡞Δ߹ɺPipelineΛදݱ͢ΔίʔυͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueueͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
• Callbackॲཧ࣮ମSidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըมύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷՌΛग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ
• Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮͕ංେԽͯ͠ؾʹͳΒͳ͍
ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Εʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏ͬͯΔ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
None
ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Εʁʣ • ffmpegίϚϯυͷೖग़ྗ༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠ೖྗͷॱংॏཁ
• CommandͷجఈΫϥεΛ࡞ͬͯநԽ • Pipeline → Step → Worker ͱɺෳͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ͢
None
·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiqͱͯྑ͍ •
Sidekiq Proͬͱྑ͍ • Sidekiq BatchΛ͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • நతͳ͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉSidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
·ͱΊ
None
THANK YOU!!