Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalable Microservice for Media Transcoding / T...
Search
atsushisakai
March 23, 2019
Programming
4
1.5k
Scalable Microservice for Media Transcoding / Transcoder
Rails Developers Meetup 2019の発表に利用した資料です。
atsushisakai
March 23, 2019
Tweet
Share
More Decks by atsushisakai
See All by atsushisakai
大規模 SaaS の技術的意思決定を支える三要素 / Three elements that support technical decision-making for large-scale SaaS
_atsushisakai
0
1.2k
一方、私はなぜEMでありつづけるか / TECH TRACK vol.1
_atsushisakai
1
490
子育てとEMと転職と
_atsushisakai
2
850
「家族アルバム みてね」を支える エンジニアリング組織/introduce-familyalbum-engineering-team
_atsushisakai
2
240
ミクシィのマネージャーは悩んでいる / mixi's manager is in trouble
_atsushisakai
16
14k
実践 Engineering Manager / practice engineering manager
_atsushisakai
19
8.2k
New Relic {FUTURE} STACK 3/14, 2019 / newrelic_future_stack
_atsushisakai
1
1.5k
実践 WorkManager / WorkManager
_atsushisakai
3
7.5k
How to use New Relic in Mitene Development Team
_atsushisakai
0
600
Other Decks in Programming
See All in Programming
iOS 26にアップデートすると実機でのHot Reloadができない?
umigishiaoi
0
140
チームで開発し事業を加速するための"良い"設計の考え方 @ サポーターズCoLab 2025-07-08
agatan
1
450
The Evolution of Enterprise Java with Jakarta EE 11 and Beyond
ivargrimstad
0
180
AI Agent 時代のソフトウェア開発を支える AWS Cloud Development Kit (CDK)
konokenj
5
650
“いい感じ“な定量評価を求めて - Four Keysとアウトカムの間の探求 -
nealle
2
11k
AI駆動のマルチエージェントによる業務フロー自動化の設計と実践
h_okkah
0
210
ニーリーにおけるプロダクトエンジニア
nealle
0
890
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
670
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
Porting a visionOS App to Android XR
akkeylab
0
660
CDK引数設計道場100本ノック
badmintoncryer
2
360
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
240
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
49
14k
Unsuck your backbone
ammeep
671
58k
Building Applications with DynamoDB
mza
95
6.5k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
The Pragmatic Product Professional
lauravandoore
35
6.7k
BBQ
matthewcrist
89
9.7k
The Cult of Friendly URLs
andyhume
79
6.5k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Balancing Empowerment & Direction
lara
1
440
Docker and Python
trallard
45
3.5k
Designing for humans not robots
tammielis
253
25k
Transcript
Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails
Developers Meetup 2019
ञҪ ಞ @_atsushisakai • ՈΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on
Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͡Ίͯ
Agenda ʮΈͯͶʯͰ͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝ͩͬͨͷ͔ʁ • Ͳ͏ͬͯղܾͨ͠ͷ͔ʁ
·ͣͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
ಈը(WIP)
͜ΕҰମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲͷΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞ΕͨಈըࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞ • ൢച͢ΔDVDʹऩͨ͠ΓɺΞϓϦʹ৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ͚ͯࢠҭͯͷࢥ͍ग़Λ࡞ʹ͢Δ •
͍ಈըΛͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢͳͲͷը૾ΛΓ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
ಈը(WIP)
1ඵಈըΛࣗಈͰେྔʹ࡞Δ
͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͚ͭΕΔ • ಈըͷ݁߹Ͱ͖Δ
• Ͱ͖ͳ͍͜ͱ • ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Իͱͷ߹ͱ͍͏ͷͰ͖ͳ͍
࡞Δ͔͠ͳ͍…
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%
͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ • ffmpegʹΑΔมॲཧΛྻʹ࿈݁ͨ͠ͷ • લͷॲཧͷΞτϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
Έͳ͞ΜͳΒͲ͏࣮͠·͔͢❓
࠷۪ͳ࣮
࠷۪ͳ࣮ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ͕ग़ͯ͘Δ • ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ৴ରͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷग़ͯ͘Δ
• CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͍ͬͨͳ͍ • ޮతͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮ ύΠϓϥΠϯΛಉ࣌ʹෳΒͤΔͱɺ͖ͬͱύϑΥʔϚϯεྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͋ΔఔΑ͘ͳΔ͔͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧͦ͠͏
None
Sidekiq Rubyͷ Job Queue γεςϜͰͱͯ༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ100Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ
• https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯগͳ͍ • ๛ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
Sidekiq
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔΛ࡞Δ • ෳͷ
Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
None
ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛͬͯBʙF͕ىಈ • BʙFύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓ɺ͜Ε…ʁʁ
https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
Ͳ͏ͬͯ࡞Δʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ Sidekiq BatchΛ͏͜ͱͰ
తͷύΠϓϥΠϯ͕࣮Ͱ͖ͦ͏‼
ύΠϓϥΠϯ࣮Ͱ͖Δ͔ʁ ׂͱཧʹ͍ۙ☺ • جຊతʹͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά؆୯ • ޮతͳϦτϥΠ͕ߦΘΕΔ
• SidekiqͷฒྻઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴԽ͍ͯ͘͠ • CPU͕Γͳ͚Εεέʔϧͤ͞Δ
Transcoder
ϚΠΫϩαʔϏεͱͯ͠ͷ࣮ • ผαʔϏε͔ΒTranscoderͷϦΫΤετAWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁ՌAWS SQSʹͤͯฦ͢ •
தؒϑΝΠϧՌAWS S3ʹอଘ͢Δ • αʔϏεؒΦϒδΣΫτ໊ͷΓͱΓͷΈ • ࣮ϑΝΠϧ͏ଆ͕औΓʹߦ͘
αʔϏεͷεέʔϥϏϦςΟ • ffmpegϞϦϞϦͱCPUΛ͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ
Transcoder γεςϜߏ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷΓऔΓͷՕॴͰখن ʹRails͕ಈ͘γεςϜ
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 1 2 3 4 5
6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2
3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2
3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜɾ࠷ऴՌΛอ
ଘ͢ΔS3όέοτ 1 2 3 4 5 6
Transcoder ͷΞʔΩςΫνϟ JobͱResult • JobɺSQSʹͤΔJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoderඇಉظʹॲཧΛ։࢝͢Δ •
ResultɺSQSʹͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ߹ʹErrorResultΛฦ͢
None
None
None
Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮ • = Sidekiq Batchͷ࣮ͦͷͷɻ͜ΕࣗମSidekiq Worker
• ෦ʹෳͷStepΛ࣋ͭ • Pipeline͝ͱʹՌ͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •
Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
Transcoder ͷΞʔΩςΫνϟ Worker • StepͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͯ͢Sidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊffmpegͷίϚϯυΛҰճ࣮ߦ͢ΔཻͰ࣮͢Δ
• ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧS3ʹอଘ͢Δ
Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
Pipeline Step Step Step Worker + Command
Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ
Sidekiq Batch࣮ͷίπ • Batch෦ͷWorkerґଘΛ͘ɺཻ͔ͭΛখ࣮͘͢͞Δɻ • εέʔϧ͍͢͠ɺϦτϥΠ͍͢͠ɺͳͲͷతɻ • ෳࡶͳBatchΛ࡞Δ߹ɺPipelineΛදݱ͢ΔίʔυͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueueͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
• Callbackॲཧ࣮ମSidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըมύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷՌΛग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ
• Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮͕ංେԽͯ͠ؾʹͳΒͳ͍
ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Εʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏ͬͯΔ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
None
ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Εʁʣ • ffmpegίϚϯυͷೖग़ྗ༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠ೖྗͷॱংॏཁ
• CommandͷجఈΫϥεΛ࡞ͬͯநԽ • Pipeline → Step → Worker ͱɺෳͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ͢
None
·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiqͱͯྑ͍ •
Sidekiq Proͬͱྑ͍ • Sidekiq BatchΛ͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • நతͳ͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉSidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
·ͱΊ
None
THANK YOU!!