Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Video Encoding Pipeline at The New Y...

Building a Video Encoding Pipeline at The New York Times

These slides were presented at the Streaming Media West conference in 2016. This talk is also a reference to the blog post "Using Microservices to Encode and Publish Videos at The New York Times" at The New York Times Open blog.

- Streaming Media West 2016: http://streamingmedia.com/Conferences/West2016/
- Open Blog: http://open.blogs.nytimes.com/2016/11/01/using-microservices-to-encode-and-publish-videos-at-the-new-york-times/

Flávio Ribeiro

November 02, 2016
Tweet

More Decks by Flávio Ribeiro

Other Decks in Technology

Transcript

  1. The Architecture of the old system Video CMS (StoryLog) NYTimes

    CMS (Scoop) Internal Encoding Cluster Internal Storage CDN
  2. The Flaws 1. The whole process relies on a filename

    2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs are required to publish a video.
  3. The Flaws 1. The whole process relies on a filename

    2. Several watch folders (hard to maintain) 3. 4. Unstable transcoding (proprietary software) 5. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4
  4. The Flaws 1. The whole process relies on a filename

    2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Production ID
  5. The Flaws 1. The whole process relies on a filename

    2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 5. 43380_1_saudi-documentary_wg_720p.mp4 Version
  6. The Flaws 1. The whole process relies on a filename

    2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Slug
  7. The Flaws 1. The whole process relies on a filename

    2. Several watch folders (hard to maintain) 3. Unstable transcoding (proprietary software) 4. 5. Hard to scale (cluster in-house) 43380_1_saudi-documentary_wg_720p.mp4 Resolution
  8. The Flaws 1. The whole process relies on a filename

    2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  9. The Flaws 1. Filenames 2. Several watch folders 3. U

    4. Hard to scale (cluster in-house)
  10. The Flaws 1. The whole process relies on a filename

    2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  11. The Flaws 1. The whole process relies on a filename

    2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMS to publish a video
  12. The Flaws 1. The whole process relies on a filename

    2. Several watch folders 3. Unstable transcoding 4. Hard to scale 5. Two CMSs to publish a video
  13. The Challenges 1. Fast encoding. News Clips are very time

    sensitive 2. Scalability 3. Reliability 4. Company-wide video transcoding solution 5. High-quality transcoding ◦ 360°, Drone footage, Live Action
  14. The Video Pipeline Distribution API Acquisition API Transcoding API Storage

    Media Factory Client Media Factory API CDN Database
  15. Media Factory Client Distribution API Acquisition API Transcoding API Storage

    Media Factory Client Media Factory API Database CDN
  16. Media Factory Client 1. EcmaScript 6 Library 2. Upload Files

    from Browser to S3 ◦ http://github.com/TTLabs/EvaporateJS 3. Trigger Jobs on Media Factory API 4. Used by other teams
  17. Acquisition API 1. Back-end integration to upload files ◦ External

    partners 2. Plan to support resumable uploads
  18. Database Media Factory Client Storage Media Factory API Distribution API

    CDN Acquisition API Transcoding API Media Factory API
  19. Media Factory API 1. All NYTimes business logics ◦ Interact

    with our internal APIs ◦ Define which presets will be used based on job type (Standard, 360) ◦ http://github.com/nytimes/video-presets
  20. Transcoding API 1. Integration with multiple encoding cloud services 2.

    Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification
  21. Transcoding API - Job Example { "provider": "elastictranscoder", "source": "s3://somebucket/folder/my_video.mov",

    "streamingParams": { "playlistFileName": "hls/playlist.m3u8", "segmentDuration": 5, "protocol": "hls" } "outputs": [ {"preset": "720p_mp4", "fileName": "my_video_720p.mp4"}, {"preset": "1080p_mp4", "fileName": "my_video_1080p.mp4"}, {"preset": "256p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "480p_hls", "fileName": "hls/my_video_480p.m3u8"}, {"preset": "720p_hls", "fileName": "hls/my_video_720p.m3u8"}, {"preset": "1080p_hls", "fileName": "hls/my_video_1080p.m3u8"}, {"preset": "2160p_hls", "fileName": "hls/my_video_2160p.m3u8"} ] }
  22. Transcoding API - Preset Example "providers": ["elastictranscoder", "elementalconductor", "encodingcom", "zencoder"],

    "preset": { "name": "sample_preset", "description": "This is an example preset", "container": "mp4", "profile": "Main", "profileLevel": "3.1", "rateControl": "VBR", "video": { "height": "720", "width": "1080", "codec": "h264", "bitrate": "1000000", "gopSize": "90", "gopMode": "fixed", "interlaceMode": "progressive" }, "audio": { "codec": "aac", "bitrate": "64000" }
  23. Transcoding API 1. Integration with multiple encoding cloud services 2.

    Reliability & Scalability 3. Define a common Job specification 4. Define a common Preset specification 5. http://nyti.ms/mediafactory
  24. The Distribution API Database Media Factory API Distribution API Acquisition

    API Media Factory Client Transcoding API Storage CDN
  25. Distribution API 1. Fast file transfer leveraging Aspera protocol 2.

    Plan to support distribution to our partners ◦ Youtube, Facebook, other partners 3. Plan to support to multiple CDNs
  26. Just a recap! Distribution API Acquisition API Transcoding API Storage

    Media Factory Client Media Factory API CDN Database
  27. Future 1. Video Quality ◦ Classify into groups ◦ VMAF,

    Reingest on the Media Factory API 2. Open Source ◦ Distribution API ◦ Acquisition API 3. Snickers (http://github.com/snickers/snickers)