Let's build a video streaming app using Web technologies

Transcript

1. GDE, Web Technologies Arnelle Balane @arnellebalane Let’s build a video streaming app using Web technologies
2. Google Developers Expert for Web Technologies I write about Web stuff on my blog, arnellebalane.com Arnelle Balane @arnellebalane
3. VIDEO STREAMING APP
4. Part 1 of 3 Concepts
5. Simple video delivery we have a video file uploaded and stored in some server or cloud storage played in the browser using <video> tag
6. Simple video delivery <video src="sample.mp4"></video>
7. Simple video delivery <video controls> <source src="sample.webm" type="video/webm" /> <source src="sample.mp4" type="video/mp4" /> Sorry, your browser doesn't support HTML video. </video>
8. ✅ Very easy to setup ❌ Little control over how the video is buffered ❌ Large videos are not ideal for mobile devices Simple video delivery
9. Third party embeds we have a video file uploaded to third-party platforms like Youtube, Vimeo, etc. embedded into our pages using <iframe> tag
10. Third party embeds <iframe width="560" height="315" src="https://www.youtube.com/embed/dQw4w9WgXcQ" title="YouTube video player" frameborder="0" allowfullscreen ></iframe>
11. ✅ Very easy to use ✅ No need to host the video files ourselves ✅ Efficient loading of video across devices ❌ Little control over the embedded content Third party embeds
12. or we can do it 🙌 ourselves 🙌
13. ✅ Full control on the content delivery and video player interface ❌ Our app has to do a lot Our own video streaming app
14. Our own video streaming app we have a video file processed and stored in our custom video pipeline played through our custom video player
15. Video processing original video file
16. Video processing video demux separate video and audio streams audio
17. Video processing transcode create multiple representations of the video stream 480p 720p 1080p
18. Video processing we can also add alternate audio streams, e.g. for dubbed content 480p 720p 1080p English Tagalog French
19. segment break the video and audio streams into small chunks Video processing
20. package create a manifest file with info about all the representations and segments Video processing manifest.mpd
21. Video playback <video> src https://example.com/sample.mp4
22. Video playback <video> src MediaSource MediaSource on MDN: https://developer.mozilla.org/en-US/docs/Web/API/MediaSource
23. Video playback <video> src SourceBuffer SourceBuffer on MDN: https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer
24. Video playback <video> src app loads manifest file to determine the names of the video and audio segments manifest.mpd
25. Video playback <video> src app loads manifest file to determine the names of the video and audio segments manifest.mpd
26. Video playback <video> src app populates the source buffer with data from the video/audio segments manifest.mpd
27. 🎉 An international standard for adaptive bitrate streaming 🎉 Builds on top of existing HTTP infrastructure DASH Dynamic Adaptive Streaming over HTTP
28. Part 2 of 3 Process Media Content
29. 👉 ffmpeg - a suite of libraries for handling multimedia files and streams Tools we’re going to use brew install ffmpeg
30. 👉 MP4box - multimedia packaging library under the gpac multimedia framework Tools we’re going to use brew install gpac
31. Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv': Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
32. Inspect source video ffprobe sample.mkv Input #0, matroska,webm, from 'sample.mkv': Metadata: encoder : libebml v1.0.0 + libmatroska v1.0.0 creation_time : 2011-04-25T12:57:46.000000Z Duration: 00:14:48.03, start: 0.000000, bitrate: 10562 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709/unknown/unknown, progressive), 1920x818, SAR 1:1 DAR 960:409, 24 fps, 24 tbr, 1k tbn Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s
33. Demux audio stream ffmpeg -i sample.mkv -map 0:1 # gets the audio track by index (index 0 is the video) -c:a aac # audio encoder -b:a 640k # audio bitrate -ar 48000 # sampling rate (48 kHz) -ac 2 # audio channels (2 = stereo) -vn # exclude video track outputs/audio.m4a
34. Demux and transcode video stream ffmpeg -i sample.mkv -c:v libx264 # video encoder -r 24 # frame rate -x264opts 'keyint=48:min-keyint=48:no-scenecut' -vf scale=-2:480 # resize video to 480p -b:v 1050k # video bitrate -maxrate 1050k # video max bitrate -bufsize 2100k # buffer size -movflags faststart -profile:v main -preset ultrafast -an # exclude audio track outputs/video-480p-1050k.mp4
35. Example bitrates and resolutions Bitrate (kbps) Resolution (px) 235 320 x 240 375 384 x 288 560 512 x 384 1050 640 x 480 1750 720 x 480 3000 1280 x 720 5800 1920 x 1080 repeat previous command to transcode video into each desired bitrate and resolution
36. After demux and transcode ~/dev/video-streaming-app 🐈 t . ├── outputs │ ├── audio.m4a │ ├── video-1080p-5800k.mp4 │ ├── video-480p-1050k.mp4 │ └── video-720p-3000k.mp4 └── sample.mkv
37. Segment and package MP4Box -dash 4000 # break videos into 4-second segments -frag 4000 -rap -segment-name 'segment-$RepresentationID$-' -fps 24 # frame rate outputs/video-480p-1050k.mp4#video:id=480p outputs/video-720p-3000k.mp4#video:id=720p outputs/video-1080p-5800k.mp4#video:id=1080p outputs/audio.m4a#audio:id=English:role=main -out dash/manifest.mpd
38. Final DASH files ~/dev/video-streaming-app 🐈 ls dash manifest.mpd segment-720p-.mp4 segment-1080p-.mp4 segment-720p-1.m4s segment-1080p-1.m4s segment-720p-10.m4s segment-1080p-10.m4s segment-720p-100.m4s segment-1080p-100.m4s segment-720p-101.m4s segment-1080p-101.m4s segment-720p-102.m4s segment-1080p-102.m4s segment-720p-103.m4s
39. Put it in the cloud! ☁ DASH files
40. 👉 ffmpeg is pre-installed! cloud.google.com/functions/docs/reference/system-packages 👉 Be mindful of quotas and limits firebase.google.com/docs/firestore/quotas 👉 Other caveats When using Firebase
41. Google App Engine Put it in the cloud! ☁ DASH files Google Cloud Storage
42. 🎉 ✨🎉 Demo time!
43. Part 3 of 3 Playback Media Content
44. 1⃣ Download and parse DASH manifest 2⃣ Initialize video element and JS objects 3⃣ Download and add segments to source buffer 4⃣ Switch between available video qualities What we’re going to do
45. Download manifest file const manifestUrl = 'dash/manifest.mpd'; const manifestResponse = await fetch(manifestUrl); const manifestString = await manifestResponse.text(); Fetch on MDN: https://developer.mozilla.org/en-US/docs/Web/API/fetch 1⃣
46. Parse manifest file const parser = new DOMParser(); const manifest = parser.parseFromString( manifestString, 'text/xml' ); DOMParser on MDN: https://developer.mozilla.org/en-US/docs/Web/API/DOMParser
47. Parse manifest file manifest.querySelector('MPD') .getAttribute('maxSegmentDuration'); manifest.querySelectorAll('Representation');
48. We need to extract these values… let segmentTmpl; // ☝ 'dash/segment-$RepresentationID$-$Number$.m4s' let initTmpl; // 'dash/segment-$RepresentationID$-.mp4' let mimeType; // 'video/mp4; codecs="avc1.424034"' let videoDuration; // 888.053 seconds let segmentDuration; // 4.011 seconds let segmentMax; // 222 let representations; // ['480p', '720p', '1080p']
49. … and initialize these let representation = representations[0]; // '480p' let segmentNumber = 1; let video = document.querySelector('video'); let mediaSource; let sourceBuffer;
50. Initialize video and JS objects mediaSource = new MediaSource(); mediaSource.addEventListener('sourceopen', () => { sourceBuffer = mediaSource.addSourceBuffer(mimeType); }); video.src = URL.createObjectURL(mediaSource); 2⃣
51. Download initialization segment const initUrl = initTmpl .replace('$RepresentationID$', representation); const initResponse = await fetch(initUrl); const initBuffer = await initResponse.arrayBuffer(); sourceBuffer.appendBuffer(initBuffer);
53. Download individual segments const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); 3⃣
54. Download individual segments async function loadNextSegment() { const segmentUrl = segmentTmpl .replace('$RepresentationID$', representation) .replace('$Number$', segmentNumber++); const segmentResponse = await fetch(segmentUrl); const segmentBuffer = await segmentResponse.arrayBuffer(); sourceBuffer.appendBuffer(segmentBuffer); } 3⃣
55. Video buffering strategies // Load everything at once, until final segment while (notYetAtFinalSegment()) { await loadNextSegment(); }
56. Video buffering strategies // Load next segment only when almost running out of data video.addEventListener('timeupdate', async () => { if (almostRunningOutOfBufferedData()) { await loadNextSegment(); } });
57. Video buffering strategies // Load correct segment when seeking the video video.addEventListener('seeking', async () => { segmentNumber = computeSegmentNumberFromTime( video.currentTime ); await loadNextSegment(); });
58. Switch between video qualities <div class="controls"> <button data-quality="480p">480p</button> <button data-quality="720p">720p</button> <button data-quality="1080p">1080p</button> </div> 4⃣
59. Switch between video qualities controls.addEventListener('click', async (event) => { if (event.target.closest('button')) { representation = event.target.dataset.quality; await loadNextSegment(); } }); 4⃣
60. Switch between video qualities 4⃣
61. 🎉 ✨🎉 Demo time!
62. we don’t need to do it all ourselves 🙌 🙌 🙌
63. Use existing services and libraries
64. Link to full code 👇 https://github.com/arnellebalane/video-streaming-app Thanks for watching! Any questions? 🤔