Share
Embed
Let's build a video streaming app using Web technologies
Arnelle Balane
November 24, 2022
Transcript
64 slides
-
GDE, Web Technologies Arnelle Balane @arnellebalane Let’s build a video
streaming app using Web technologies -
Google Developers Expert for Web Technologies I write about Web
stuff on my blog, arnellebalane.com Arnelle Balane @arnellebalane -
Simple video delivery we have a video file uploaded and
stored in some server or cloud storage played in the browser using <video> tag -
Simple video delivery <video controls> <source src="sample.webm" type="video/webm" /> <source
src="sample.mp4" type="video/mp4" /> Sorry, your browser doesn't support HTML video. </video> -
✅ Very easy to setup ❌ Little control over how
the video is buffered ❌ Large videos are not ideal for mobile devices Simple video delivery -
Third party embeds we have a video file uploaded to
third-party platforms like Youtube, Vimeo, etc. embedded into our pages using <iframe> tag -
Third party embeds <iframe width="560" height="315" src="https://www.youtube.com/embed/dQw4w9WgXcQ" title="YouTube video player"
frameborder="0" allowfullscreen ></iframe> -
✅ Very easy to use ✅ No need to host
the video files ourselves ✅ Efficient loading of video across devices ❌ Little control over the embedded content Third party embeds -
✅ Full control on the content delivery and video player
interface ❌ Our app has to do a lot Our own video streaming app -
Our own video streaming app we have a video file
processed and stored in our custom video pipeline played through our custom video player -
Video processing we can also add alternate audio streams, e.g.
for dubbed content 480p 720p 1080p English Tagalog French -
segment break the video and audio streams into small chunks
Video processing -
package create a manifest file with info about all the
representations and segments Video processing manifest.mpd -
Video playback <video> src app loads manifest file to determine
the names of the video and audio segments manifest.mpd -
Video playback <video> src app loads manifest file to determine
the names of the video and audio segments manifest.mpd -
Video playback <video> src app populates the source buffer with
data from the video/audio segments manifest.mpd -
🎉 An international standard for adaptive bitrate streaming 🎉 Builds
on top of existing HTTP infrastructure DASH Dynamic Adaptive Streaming over HTTP -
👉 ffmpeg - a suite of libraries for handling multimedia
files and streams Tools we’re going to use brew install ffmpeg -
👉 MP4box - multimedia packaging library under the gpac multimedia
framework Tools we’re going to use brew install gpac -
Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv':
Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s -
Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv':
Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s -
Demux audio stream ffmpeg -i sample.mkv -map 0:1 # gets
the audio track by index (index 0 is the video) -c:a aac # audio encoder -b:a 640k # audio bitrate -ar 48000 # sampling rate (48 kHz) -ac 2 # audio channels (2 = stereo) -vn # exclude video track outputs/audio.m4a -
Demux and transcode video stream ffmpeg -i sample.mkv -c:v libx264
# video encoder -r 24 # frame rate -x264opts 'keyint=48:min-keyint=48:no-scenecut' -vf scale=-2:480 # resize video to 480p -b:v 1050k # video bitrate -maxrate 1050k # video max bitrate -bufsize 2100k # buffer size -movflags faststart -profile:v main -preset ultrafast -an # exclude audio track outputs/video-480p-1050k.mp4 -
Example bitrates and resolutions Bitrate (kbps) Resolution (px) 235 320
x 240 375 384 x 288 560 512 x 384 1050 640 x 480 1750 720 x 480 3000 1280 x 720 5800 1920 x 1080 repeat previous command to transcode video into each desired bitrate and resolution -
After demux and transcode ~/dev/video-streaming-app 🐈 t . ├── outputs
│ ├── audio.m4a │ ├── video-1080p-5800k.mp4 │ ├── video-480p-1050k.mp4 │ └── video-720p-3000k.mp4 └── sample.mkv -
Segment and package MP4Box -dash 4000 # break videos into
4-second segments -frag 4000 -rap -segment-name 'segment-$RepresentationID$-' -fps 24 # frame rate outputs/video-480p-1050k.mp4#video:id=480p outputs/video-720p-3000k.mp4#video:id=720p outputs/video-1080p-5800k.mp4#video:id=1080p outputs/audio.m4a#audio:id=English:role=main -out dash/manifest.mpd -
Final DASH files ~/dev/video-streaming-app 🐈 ls dash manifest.mpd segment-720p-.mp4 segment-1080p-.mp4
segment-720p-1.m4s segment-1080p-1.m4s segment-720p-10.m4s segment-1080p-10.m4s segment-720p-100.m4s segment-1080p-100.m4s segment-720p-101.m4s segment-1080p-101.m4s segment-720p-102.m4s segment-1080p-102.m4s segment-720p-103.m4s -
👉 ffmpeg is pre-installed! cloud.google.com/functions/docs/reference/system-packages 👉 Be mindful of quotas
and limits firebase.google.com/docs/firestore/quotas 👉 Other caveats When using Firebase -
Google App Engine Put it in the cloud! ☁ DASH
files Google Cloud Storage -
1⃣ Download and parse DASH manifest 2⃣ Initialize video element
and JS objects 3⃣ Download and add segments to source buffer 4⃣ Switch between available video qualities What we’re going to do -
Download manifest file const manifestUrl = 'dash/manifest.mpd'; const manifestResponse =
await fetch(manifestUrl); const manifestString = await manifestResponse.text(); Fetch on MDN: https://developer.mozilla.org/en-US/docs/Web/API/fetch 1⃣ -
Parse manifest file const parser = new DOMParser(); const manifest
= parser.parseFromString( manifestString, 'text/xml' ); DOMParser on MDN: https://developer.mozilla.org/en-US/docs/Web/API/DOMParser -
We need to extract these values… let segmentTmpl; // ☝
'dash/segment-$RepresentationID$-$Number$.m4s' let initTmpl; // 'dash/segment-$RepresentationID$-.mp4' let mimeType; // 'video/mp4; codecs="avc1.424034"' let videoDuration; // 888.053 seconds let segmentDuration; // 4.011 seconds let segmentMax; // 222 let representations; // ['480p', '720p', '1080p'] -
… and initialize these let representation = representations[0]; // '480p'
let segmentNumber = 1; let video = document.querySelector('video'); let mediaSource; let sourceBuffer; -
Initialize video and JS objects mediaSource = new MediaSource(); mediaSource.addEventListener('sourceopen',
() => { sourceBuffer = mediaSource.addSourceBuffer(mimeType); }); video.src = URL.createObjectURL(mediaSource); 2⃣ -
Download initialization segment const initUrl = initTmpl .replace('$RepresentationID$', representation); const
initResponse = await fetch(initUrl); const initBuffer = await initResponse.arrayBuffer(); sourceBuffer.appendBuffer(initBuffer); -
Download individual segments const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$',
segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); 3⃣ -
Download individual segments async function loadNextSegment() { const segmentUrl =
segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); } 3⃣ -
Video buffering strategies // Load everything at once, until final
segment while (notYetAtFinalSegment()) { await loadNextSegment(); } -
Video buffering strategies // Load next segment only when almost
running out of data video.addEventListener('timeupdate', async () => { if (almostRunningOutOfBufferedData()) { await loadNextSegment(); } }); -
Video buffering strategies // Load correct segment when seeking the
video video.addEventListener('seeking', async () => { segmentNumber = computeSegmentNumberFromTime( video.currentTime ); await loadNextSegment(); }); -
Switch between video qualities <div class="controls"> <button data-quality="480p">480p</button> <button data-quality="720p">720p</button>
<button data-quality="1080p">1080p</button> </div> 4⃣ -
Switch between video qualities controls.addEventListener('click', async (event) => { if
(event.target.closest('button')) { representation = event.target.dataset.quality; await loadNextSegment(); } }); 4⃣