Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let's build a video streaming app using Web tec...

Let's build a video streaming app using Web technologies

Arnelle Balane

November 24, 2022
Tweet

More Decks by Arnelle Balane

Other Decks in Programming

Transcript

  1. Google Developers Expert for Web Technologies I write about Web

    stuff on my blog, arnellebalane.com Arnelle Balane @arnellebalane
  2. Simple video delivery we have a video file uploaded and

    stored in some server or cloud storage played in the browser using <video> tag
  3. Simple video delivery <video controls> <source src="sample.webm" type="video/webm" /> <source

    src="sample.mp4" type="video/mp4" /> Sorry, your browser doesn't support HTML video. </video>
  4. ✅ Very easy to setup ❌ Little control over how

    the video is buffered ❌ Large videos are not ideal for mobile devices Simple video delivery
  5. Third party embeds we have a video file uploaded to

    third-party platforms like Youtube, Vimeo, etc. embedded into our pages using <iframe> tag
  6. ✅ Very easy to use ✅ No need to host

    the video files ourselves ✅ Efficient loading of video across devices ❌ Little control over the embedded content Third party embeds
  7. ✅ Full control on the content delivery and video player

    interface ❌ Our app has to do a lot Our own video streaming app
  8. Our own video streaming app we have a video file

    processed and stored in our custom video pipeline played through our custom video player
  9. Video processing we can also add alternate audio streams, e.g.

    for dubbed content 480p 720p 1080p English Tagalog French
  10. package create a manifest file with info about all the

    representations and segments Video processing manifest.mpd
  11. Video playback <video> src app loads manifest file to determine

    the names of the video and audio segments manifest.mpd
  12. Video playback <video> src app loads manifest file to determine

    the names of the video and audio segments manifest.mpd
  13. Video playback <video> src app populates the source buffer with

    data from the video/audio segments manifest.mpd
  14. 🎉 An international standard for adaptive bitrate streaming 🎉 Builds

    on top of existing HTTP infrastructure DASH Dynamic Adaptive Streaming over HTTP
  15. 👉 ffmpeg - a suite of libraries for handling multimedia

    files and streams Tools we’re going to use brew install ffmpeg
  16. 👉 MP4box - multimedia packaging library under the gpac multimedia

    framework Tools we’re going to use brew install gpac
  17. Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv':

    Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
  18. Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv':

    Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
  19. Demux audio stream ffmpeg -i sample.mkv -map 0:1 # gets

    the audio track by index (index 0 is the video) -c:a aac # audio encoder -b:a 640k # audio bitrate -ar 48000 # sampling rate (48 kHz) -ac 2 # audio channels (2 = stereo) -vn # exclude video track outputs/audio.m4a
  20. Demux and transcode video stream ffmpeg -i sample.mkv -c:v libx264

    # video encoder -r 24 # frame rate -x264opts 'keyint=48:min-keyint=48:no-scenecut' -vf scale=-2:480 # resize video to 480p -b:v 1050k # video bitrate -maxrate 1050k # video max bitrate -bufsize 2100k # buffer size -movflags faststart -profile:v main -preset ultrafast -an # exclude audio track outputs/video-480p-1050k.mp4
  21. Example bitrates and resolutions Bitrate (kbps) Resolution (px) 235 320

    x 240 375 384 x 288 560 512 x 384 1050 640 x 480 1750 720 x 480 3000 1280 x 720 5800 1920 x 1080 repeat previous command to transcode video into each desired bitrate and resolution
  22. After demux and transcode ~/dev/video-streaming-app 🐈 t . ├── outputs

    │ ├── audio.m4a │ ├── video-1080p-5800k.mp4 │ ├── video-480p-1050k.mp4 │ └── video-720p-3000k.mp4 └── sample.mkv
  23. Segment and package MP4Box -dash 4000 # break videos into

    4-second segments -frag 4000 -rap -segment-name 'segment-$RepresentationID$-' -fps 24 # frame rate outputs/video-480p-1050k.mp4#video:id=480p outputs/video-720p-3000k.mp4#video:id=720p outputs/video-1080p-5800k.mp4#video:id=1080p outputs/audio.m4a#audio:id=English:role=main -out dash/manifest.mpd
  24. Final DASH files ~/dev/video-streaming-app 🐈 ls dash manifest.mpd segment-720p-.mp4 segment-1080p-.mp4

    segment-720p-1.m4s segment-1080p-1.m4s segment-720p-10.m4s segment-1080p-10.m4s segment-720p-100.m4s segment-1080p-100.m4s segment-720p-101.m4s segment-1080p-101.m4s segment-720p-102.m4s segment-1080p-102.m4s segment-720p-103.m4s
  25. 👉 ffmpeg is pre-installed! cloud.google.com/functions/docs/reference/system-packages 👉 Be mindful of quotas

    and limits firebase.google.com/docs/firestore/quotas 👉 Other caveats When using Firebase
  26. 1⃣ Download and parse DASH manifest 2⃣ Initialize video element

    and JS objects 3⃣ Download and add segments to source buffer 4⃣ Switch between available video qualities What we’re going to do
  27. Download manifest file const manifestUrl = 'dash/manifest.mpd'; const manifestResponse =

    await fetch(manifestUrl); const manifestString = await manifestResponse.text(); Fetch on MDN: https://developer.mozilla.org/en-US/docs/Web/API/fetch 1⃣
  28. Parse manifest file const parser = new DOMParser(); const manifest

    = parser.parseFromString( manifestString, 'text/xml' ); DOMParser on MDN: https://developer.mozilla.org/en-US/docs/Web/API/DOMParser
  29. We need to extract these values… let segmentTmpl; // ☝

    'dash/segment-$RepresentationID$-$Number$.m4s' let initTmpl; // 'dash/segment-$RepresentationID$-.mp4' let mimeType; // 'video/mp4; codecs="avc1.424034"' let videoDuration; // 888.053 seconds let segmentDuration; // 4.011 seconds let segmentMax; // 222 let representations; // ['480p', '720p', '1080p']
  30. … and initialize these let representation = representations[0]; // '480p'

    let segmentNumber = 1; let video = document.querySelector('video'); let mediaSource; let sourceBuffer;
  31. Initialize video and JS objects mediaSource = new MediaSource(); mediaSource.addEventListener('sourceopen',

    () => { sourceBuffer = mediaSource.addSourceBuffer(mimeType); }); video.src = URL.createObjectURL(mediaSource); 2⃣
  32. Download initialization segment const initUrl = initTmpl .replace('$RepresentationID$', representation); const

    initResponse = await fetch(initUrl); const initBuffer = await initResponse.arrayBuffer(); sourceBuffer.appendBuffer(initBuffer);
  33. Download individual segments const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$',

    segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); 3⃣
  34. Download individual segments async function loadNextSegment() { const segmentUrl =

    segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); } 3⃣
  35. Video buffering strategies // Load everything at once, until final

    segment while (notYetAtFinalSegment()) { await loadNextSegment(); }
  36. Video buffering strategies // Load next segment only when almost

    running out of data video.addEventListener('timeupdate', async () => { if (almostRunningOutOfBufferedData()) { await loadNextSegment(); } });
  37. Video buffering strategies // Load correct segment when seeking the

    video video.addEventListener('seeking', async () => { segmentNumber = computeSegmentNumberFromTime( video.currentTime ); await loadNextSegment(); });
  38. Switch between video qualities controls.addEventListener('click', async (event) => { if

    (event.target.closest('button')) { representation = event.target.dataset.quality; await loadNextSegment(); } }); 4⃣