Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scalable Microservice for Media Transcoding / T...

Scalable Microservice for Media Transcoding / Transcoder

Rails Developers Meetup 2019の発表に利用した資料です。

Avatar for atsushisakai

atsushisakai

March 23, 2019
Tweet

More Decks by atsushisakai

Other Decks in Programming

Transcript

  1. ञҪ ಞ @_atsushisakai • Ո଒ΞϧόϜʮΈͯͶʯΛ։ൃ͍ͯ͠·͢ • SRE • Ruby on

    Rails • AWS • iOS/Android • Engineering Manager • iOSDC/DroidKaigi • RailsDM͸͸͡Ίͯ
  2. ͍͍ײ͡ʹͰ͖ΔϚωʔδυαʔϏεͳ͍Ͱ͔͢ʁ • ྫ͑͹ɺAWS ElasticTranscoder • Ͱ͖Δ͜ͱ • Watermark͸͚ͭΕΔ • ಈըͷ݁߹΋Ͱ͖Δ

    • Ͱ͖ͳ͍͜ͱ • ෼ׂͨ͠ΓɺΤϑΣΫτ͔͚ͨΓɺͱ͍͏͜ͱ͕Ͱ͖ͳ͍ • Ի੠ͱͷ߹੒ͱ͍͏ͷ΋Ͱ͖ͳ͍
  3. Sidekiq Ruby੡ͷ Job Queue γεςϜͰͱͯ΋༗໊ͳgemɻΦεεϝɻ ʮΈͯͶʯͰ͸100୆Ҏ্ͷSidekiq͕ಈ͘αʔόʔ͕ৗʹಈ͍͍ͯΔɻ • ύϑΥʔϚϯε͕ͱʹ͔͘ྑ͍ • Resque/DelayedJobͷ20ഒۙ͘ߴ଎

    • https://github.com/mperham/sidekiq#performance • όʔδϣϯΞοϓʹΑΓͲΜͲΜύϑΥʔϚϯε͕ྑ͘ͳΔ • ଞͷGemґଘ͕ͱͯ΋গͳ͍ • ๛෋ͳެࣜυΩϡϝϯτɺ׆ൃͳ։ൃ • ActiveJobͰAdapter͕ఏڙ͞Ε͍ͯΔ
  4. Sidekiq Batch • Sidekiq Pro ݶఆͷػೳ • ୯७ͳWorkerͷॲཧʹґଘؔ܎Λ࡞Δ • ෳ਺ͷ

    Worker ΛଋͶͯɺͦͷ Batch ͝ͱʹॲཧΛਐΊ͍ͯ͘ • ͻͱͭͷ Batch ͕ऴΘΔͱ Callback ͕࣮ߦ͞Εɺ࣍ͷॲཧʹਐΉɺͱ͍͏ ॲཧΛهड़͢Δ͜ͱ͕Ͱ͖Δ • ͨͱ͑ D ͕ࣦഊͯ͠΋ D ͚ͩϦτϥΠ ͯ͘͠ΕΔ A B C D
  5. ϚΠΫϩαʔϏεͱͯ͠ͷ࣮૷ • ผαʔϏε͔ΒTranscoder΁ͷϦΫΤετ͸AWS SQSΛܦ༝͢Δɻ • ඇಉظɾૄ݁߹ʹαʔϏεؒ࿈ܞ • ࣮ߦ݁Ռ΋AWS SQSʹ৐ͤͯฦ͢ •

    தؒϑΝΠϧ΍੒Ռ෺͸AWS S3ʹอଘ͢Δ • αʔϏεؒ͸ΦϒδΣΫτ໊ͷ΍ΓͱΓͷΈ • ࣮ϑΝΠϧ͸࢖͏ଆ͕औΓʹߦ͘
  6. Transcoder γεςϜߏ੒ 1. ଞͷαʔϏεͱͷೖग़ྗΠϯ λʔϑΣʔε͸SQSΛ༻͍Δ 2. SQSΛPolling͢ΔΠϯελϯ ε 3. SQS͔ΒऔΓग़࣮ͨ͠ߦґཔ

    ΛSidekiq BatchͰॲཧ͢Δ 4. RDSʹґཔ͞Εͨ಺༰Λอ ࣋ɾࢀর͢Δ ʢActiveRecordʣ 1 2 3 4 5 6
  7. Transcoder ͷΞʔΩςΫνϟ JobͱResult • Job͸ɺSQSʹ৐ͤΔJSONͷ͜ͱ • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛPolling͠ɺTranscoder͸ඇಉظʹॲཧΛ։࢝͢Δ •

    Result͸ɺSQSʹ৐ͤΔॲཧ݁ՌͷJSONͷ͜ͱ • ܕ͸ࣄલʹఆٛ͞Ε͍ͯΔ • SQSΛґཔͨ͠ଆ͕Pollingͯ݁͠ՌΛड͚ͱΔ • ݁Ռ͕Errorͷ৔߹ʹ͸ErrorResultΛฦ͢
  8. Transcoder ͷΞʔΩςΫνϟ Pipeline • JobͷೖྗΛड͚ɺ࠷ޙʹResultΛग़ྗ͢Δ·ͰͷશମͷॲཧΛ࣮૷ • = Sidekiq Batchͷ࣮૷ͦͷ΋ͷɻ͜Εࣗମ΋Sidekiq Worker

    • ಺෦ʹෳ਺ͷStepΛ࣋ͭ • Pipeline͝ͱʹ੒Ռ෺͕ఆٛ͞Ε͍ͯΔ • ʮΞϓϦ഑৴ʯ༻ͷಈըΛͭ͘ΔPipeline • ʮDVDʯ༻ͷಈըΛͭ͘ΔPipeline
  9. ͕࣌ؒͳͯ͘Ͱ͖ͳ͔ͬͨ͜ͱ • ಈըม׵ύϥϝʔλͷPresetػೳΛ࡞Γ͔ͨͬͨ • ઃఆϑΝΠϧΛ༻ҙ͓͚ͯ͠͹ɺJobͰPresetΛࢦఆ͢Δ͚ͩͰɺ೚ҙͷύϥ ϝʔλͰ͍ΖΜͳը࣭ɾαΠζͷ੒Ռ෺Λग़ͤΔΑ͏ʹ͔ͨͬͨ͠ • TV༻ɺεϚϗ༻ɺPC༻ͱ͔… • Pipelineͷࣗಈੜ੒

    • Α͋͘ΔCIͷઃఆͷΑ͏ʹyamlʹPipelineͷॲཧΛఆ͍ٛͯͬͨ͠Βίʔυ Λࣗಈੜ੒͞ΕɺউखʹPipeline͕ग़དྷ্͕ΔΑ͏ʹ͔ͨͬͨ͠ • Pipelineͷ࣮૷͕ංେԽͯ͠΋ؾʹͳΒͳ͍
  10. ·ͱΊ • εέʔϦϯάՄೳͳಈըฤू Microservice ΛSidekiq BatchΛ࢖͍͍ͬͯײ͡ ʹ࡞ͬͨ • Sidekiq͸ͱͯ΋ྑ͍ •

    Sidekiq Pro͸΋ͬͱྑ͍ • Sidekiq BatchΛ࢖͏͜ͱͰෳࡶͳύΠϓϥΠϯॲཧΛεέʔϧͰ͖Δ • ಈըॲཧҎ֎ʹ΋ɺूܭɾղੳͱ͔ɺෳࡶͳcron scriptͱ͔… • ந৅తͳ࿩͕ଟ͘ͳͬͯ͠·ͬͨͷͰɺৄࡉ͸Sidekiq BatchΛੋඇ͓ࢼͩ͘͠ ͍͞ʢ$950/yearʣ