Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scalable Microservice for Media Transcoding / Transcoder
Search
atsushisakai
March 23, 2019
Programming
4
1.4k
Scalable Microservice for Media Transcoding / Transcoder
Rails Developers Meetup 2019の発表に利用した資料です。
atsushisakai
March 23, 2019
Tweet
Share
More Decks by atsushisakai
See All by atsushisakai
一方、私はなぜEMでありつづけるか / TECH TRACK vol.1
_atsushisakai
0
350
子育てとEMと転職と
_atsushisakai
1
690
「家族アルバム みてね」を支える エンジニアリング組織/introduce-familyalbum-engineering-team
_atsushisakai
2
200
ミクシィのマネージャーは悩んでいる / mixi's manager is in trouble
_atsushisakai
16
14k
実践 Engineering Manager / practice engineering manager
_atsushisakai
19
7.8k
New Relic {FUTURE} STACK 3/14, 2019 / newrelic_future_stack
_atsushisakai
1
1.4k
実践 WorkManager / WorkManager
_atsushisakai
3
7k
How to use New Relic in Mitene Development Team
_atsushisakai
0
530
みてねのスクラム開発 / scrum development style in mitene
_atsushisakai
3
3.2k
Other Decks in Programming
See All in Programming
Go製Webアプリケーションのエラーとの向き合い方大全、あるいはやっぱりスタックトレース欲しいやん / Kyoto.go #50
utgwkk
6
1.8k
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
680
禅の心を手に入れよ
eltociear
1
290
見た目から始める生産性向上
ikumatadokoro
10
1.3k
検証も兼ねて個人開発でHonoとかと向き合った話
hanetsuki
1
1.3k
初心者のためのRubyKaigi入門/RubyKaigi Introduction
a_matsuda
8
1.4k
雑に思考を整理する技術と効能
konifar
63
30k
Hanami and htmx
bkuhlmann
0
220
Node.js v22 で変わること
yosuke_furukawa
PRO
11
3.9k
"config" ってなんだ? / What is "config"?
okashoi
0
250
Kotlin Multiplatform at Stable and Beyond (Android Makers 2024)
zsmb
0
460
R言語の環境構築と基礎 Tokyo.R 112
bob3bob3
0
280
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
Teambox: Starting and Learning
jrom
128
8.4k
From Idea to $5000 a Month in 5 Months
shpigford
378
45k
A designer walks into a library…
pauljervisheath
201
23k
Navigating Team Friction
lara
179
13k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
11
1k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
Music & Morning Musume
bryan
41
5.6k
The Straight Up "How To Draw Better" Workshop
denniskardys
228
130k
Faster Mobile Websites
deanohume
300
30k
What's in a price? How to price your products and services
michaelherold
238
11k
The Mythical Team-Month
searls
216
42k
Transcript
Scalable Microservice for Media Transcoding ञҪ ಞ @_atsushisakai גࣜձࣾϛΫγΟ Rails
Developers Meetup 2019
ञҪ ಞ @_atsushisakai • ՈΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on
Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͡Ίͯ
Agenda ʮΈͯͶʯͰ͏ɺಈըฤूɾτϥϯείʔυϚΠΫϩαʔϏεͷ͓ɻ • ͳͥ࡞ͬͨͷ͔ʁ • ղܾ͍ͨ͠ཁ݅ɾ՝ͩͬͨͷ͔ʁ • Ͳ͏ͬͯղܾͨ͠ͷ͔ʁ
·ͣͪ͜ΒͷಈըΛ͝ཡ͍ͩ͘͞!
ಈը(WIP)
͜ΕҰମ…ʁ • ʮ1ඵಈըʯͱ͍͏ࢠͲͷΛৼΓฦΔػೳ • େྔʹΞοϓϩʔυ͞ΕͨಈըࣸਅΛૉࡐͱͯ͠μΠδΣετಈըΛ࡞ • ൢച͢ΔDVDʹऩͨ͠ΓɺΞϓϦʹ৴ͨ͠Γ͢Δ • ଟ༷ͳޮՌΛ͚ͯࢠҭͯͷࢥ͍ग़Λ࡞ʹ͢Δ •
͍ಈըΛͬͯɺผͷಈըͱ݁߹͢Δ • WatermarkΛ͚ͭΔʢͳͲͷը૾ΛΓ͚ͭΔʣ • ΤϑΣΫτΛ͚ͭͯܨ͛ΔʢΫϩεϑΣʔυͳͲʣ • BGMΛ͚ͭΔ
ಈը(WIP)
1ඵಈըΛࣗಈͰେྔʹ࡞Δ
͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͚ͭΕΔ • ಈըͷ݁߹Ͱ͖Δ
• Ͱ͖ͳ͍͜ͱ • ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Իͱͷ߹ͱ͍͏ͷͰ͖ͳ͍
࡞Δ͔͠ͳ͍…
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ
Ͳ͏ͬͯʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ ͍ΘΏΔύΠϓϥΠϯॲཧ%
͜͜Ͱݴ͏ύΠϓϥΠϯॲཧͱ • ffmpegʹΑΔมॲཧΛྻʹ࿈݁ͨ͠ͷ • લͷॲཧͷΞτϓοτ͕࣍ͷॲཧͷΠϯϓοτͱͳΔ
Έͳ͞ΜͳΒͲ͏࣮͠·͔͢❓
࠷۪ͳ࣮
࠷۪ͳ࣮ ffmpegΛॱ൪ʹ࣮ߦ͍ͯ͘͠εΫϦϓτΛͳΜΒ͔ͷݴޠͰॻ͍͍ͯ͘ • ύϑΥʔϚϯεͷ͕ग़ͯ͘Δ • ྻͷॲཧΛಈը1݅ͣͭʹରͯ͠ॱ൪ʹ͍ͬͯͨΒ͕࣌ؒແݶʹඞཁ • ٸʹ৴ରͷϢʔβʔ͕૿͑ͨΒͲ͏͢Δʁ • ίετͷग़ͯ͘Δ
• CPU/ϝϞϦ͕༨ͬͨঢ়ଶͰ͍ͬͨͳ͍ • ޮతͳϦτϥΠ͕Ͱ͖ͳ͍ • ్தͰࣦഊͨ͠Β࠷ॳ͔Β…
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮
ฒྻ࣮ߦ͢ΔΑ͏ʹ࣮ ύΠϓϥΠϯΛಉ࣌ʹෳΒͤΔͱɺ͖ͬͱύϑΥʔϚϯεྑ͘ͳΔ • ϚϧνϓϩηεϓϩάϥϛϯάΊͬͪΌͦ͠͏ • ઈରʹόάΒͤΔࣗ৴͕͋Γ·͢ • ύϑΥʔϚϯε͋ΔఔΑ͘ͳΔ͔͠Εͳ͍͚ͲνϡʔχϯάͰ͖ͳͦ͏ • Ҿ͖ଓ͖ϦτϥΠॲཧͦ͠͏
None
Sidekiq Rubyͷ Job Queue γεςϜͰͱͯ༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ100Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ
• https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯগͳ͍ • ๛ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
Sidekiq
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Pro Sidekiqʹ՝ۚ͢Δͱྑ͍͜ͱ͕͋Γ·͢ɻ https://sidekiq.org/products/pro.html • ϝʔϧαϙʔτΛड͚ΒΕΔ • Job͕ϩετ͠ͳ͍ͨΊͷReliabilityػೳ • ෳࡶͳϫʔΫϑϩʔΛ࡞Δ͜ͱ͕Ͱ͖ΔBatchػೳ
Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔΛ࡞Δ • ෳͷ
Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
None
ͬͱෳࡶͳBatchॲཧ • A͕ऴΘΔͷΛͬͯBʙF͕ىಈ • BʙFύϥϨϧʹىಈ͢Δ • BʙF͕ऴΘͬͨΒGΛىಈ͢Δ • … ͓ɺ͜Ε…ʁʁ
https://github.com/mperham/sidekiq/wiki/Really-Complex-Workflows-with-Batches
Ͳ͏ͬͯ࡞Δʁ Γग़͠ ϦαΠζͳͲͷ લॲཧ ෳͷಈըΛ ݁߹ Իָͱ߹ Ռ͕ʂ Sidekiq BatchΛ͏͜ͱͰ
తͷύΠϓϥΠϯ͕࣮Ͱ͖ͦ͏‼
ύΠϓϥΠϯ࣮Ͱ͖Δ͔ʁ ׂͱཧʹ͍ۙ☺ • جຊతʹͨͩͷSidekiq Workerͳͷ Ͱ͓खܰʹฒྻॲཧͰ͖Δ • ϓϩάϥϛϯά؆୯ • ޮతͳϦτϥΠ͕ߦΘΕΔ
• SidekiqͷฒྻઃఆΛ্͛ΔͱCPU͕ ༨͍ͬͯΔݶΓߴԽ͍ͯ͘͠ • CPU͕Γͳ͚Εεέʔϧͤ͞Δ
Transcoder
ϚΠΫϩαʔϏεͱͯ͠ͷ࣮ • ผαʔϏε͔ΒTranscoderͷϦΫΤετAWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁ՌAWS SQSʹͤͯฦ͢ •
தؒϑΝΠϧՌAWS S3ʹอଘ͢Δ • αʔϏεؒΦϒδΣΫτ໊ͷΓͱΓͷΈ • ࣮ϑΝΠϧ͏ଆ͕औΓʹߦ͘
αʔϏεͷεέʔϥϏϦςΟ • ffmpegϞϦϞϦͱCPUΛ͏ • ffmpegΛ࣮ߦ͢ΔWorker͕ͨ͘͞ΜQueueʹੵ·Ε͍ͯΔ࣌ʹίϯϐϡʔτϦ ιʔεΛΦʔτεέʔϧ͠ɺશମͷॲཧೳྗΛ্͛ΔʢAWS EC2ͷAutoScale ػೳͰ༰қʹରԠՄೳʣ
Transcoder γεςϜߏ • શͯAWSͷαʔϏεΛར༻͠ ͨઃܭ • DBͱͷΓऔΓͷՕॴͰখن ʹRails͕ಈ͘γεςϜ
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 1 2 3 4 5
6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 1 2
3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 1 2 3 4 5 6
Transcoder γεςϜߏ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔεSQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ
ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 1 2
3 4 5 6
Transcoder γεςϜߏ 5. Sidekiq ͷ Job ΛӬଓԽ͢Δ όοΫΤϯυͱͯ͠ͷRedis 6. தؒੜɾ࠷ऴՌΛอ
ଘ͢ΔS3όέοτ 1 2 3 4 5 6
Transcoder ͷΞʔΩςΫνϟ JobͱResult • JobɺSQSʹͤΔJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoderඇಉظʹॲཧΛ։࢝͢Δ •
ResultɺSQSʹͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ߹ʹErrorResultΛฦ͢
None
None
None
Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮ • = Sidekiq Batchͷ࣮ͦͷͷɻ͜ΕࣗମSidekiq Worker
• ෦ʹෳͷStepΛ࣋ͭ • Pipeline͝ͱʹՌ͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
Transcoder ͷΞʔΩςΫνϟ Step • Pipeline͕ఆٛͨ͠ෳͷWorkerͷଋͷ͜ͱΛ͜͏ݺΜͰ͍Δ • = Sidekiq Batchʹ͓͚ΔBatchͷ͜ͱ •
Step͝ͱʹCallbackΛΩοΫ͠ɺ࣍ͷStepʹҠΔ
Transcoder ͷΞʔΩςΫνϟ Worker • StepͷͻͱͭͣͭͷSidekiq Workerͷ͜ͱ • ͯ͢Sidekiq্Ͱඇಉظɾฒྻʹ࣮ߦ͞ΕΔ • جຊffmpegͷίϚϯυΛҰճ࣮ߦ͢ΔཻͰ࣮͢Δ
• ffmpegͷ࣮ߦ݁Ռͷग़ྗϑΝΠϧS3ʹอଘ͢Δ
Transcoder ͷΞʔΩςΫνϟ Command • WorkerͰ࣮ߦ͢ΔffmpegίϚϯυΛڞ௨ԽͷͨΊʹϥΠϒϥϦԽͨ͠ͷ • ୯ʹffmpegΛ࣮ߦ͠ɺͦͷ݁ՌΛϑΝΠϧͱͯ͠ड͚औΔ͚ͩ
Pipeline Step Step Step Worker + Command
Pipeline Step Step Step Worker + Command Sidekiq BatchͰղܾ
Sidekiq Batch࣮ͷίπ • Batch෦ͷWorkerґଘΛ͘ɺཻ͔ͭΛখ࣮͘͢͞Δɻ • εέʔϧ͍͢͠ɺϦτϥΠ͍͢͠ɺͳͲͷతɻ • ෳࡶͳBatchΛ࡞Δ߹ɺPipelineΛදݱ͢ΔίʔυͲΜͲΜංେԽ͢Δ • Callbackͷ࣮ߦQueueͪΌΜͱઃఆͨ͠ํ͕ྑ͍ɻ
• Callbackॲཧ࣮ମSidekiq Workerͱͯ͠ॳظԽ͞Εɺ࣮ߦ͞ΕΔɻ • େྔʹߴͳCallback͕࣮ߦ͞ΕΔͱRedisʹෛՙ͕ҟৗʹ͔͔ͬͯࢮ…
͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըมύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷՌΛग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ
• Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮͕ංେԽͯ͠ؾʹͳΒͳ͍
ͨ͠ͱ͜Ζ 1ʢ࣌ؒ͋Εʁʣ • ʮલͷStepͷग़ྗΛɺ࣍ͷStepͷೖྗͱ͢ΔʯΛͲ͏ͬͯΔ͔ʁ • Sidekiq BatchͷCallbackͰҾ͖ճ͍ͯ͘͜͠ͱͰղܾɻ
None
ͨ͠ͱ͜Ζ 2ʢ࣌ؒ͋Εʁʣ • ffmpegίϚϯυͷೖग़ྗ༷͕ಛघ͗͢Δ • ͻͱͭͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ෳͷϑΝΠϧΛInput͠ɺͻͱͭͷϑΝΠϧΛOutput͢Δ • ͔͠ೖྗͷॱংॏཁ
• CommandͷجఈΫϥεΛ࡞ͬͯநԽ • Pipeline → Step → Worker ͱɺෳͷInputΛॱংΛอͪͳ͕Β͍͍ײ ͡ʹCommandʹ͢
None
·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiqͱͯྑ͍ •
Sidekiq Proͬͱྑ͍ • Sidekiq BatchΛ͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • நతͳ͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉSidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ
·ͱΊ
None
THANK YOU!!