$30 off During Our Annual Pro Sale. View Details »

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

Phil Nash
April 27, 2021
Programming
1
150

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

WebRTC is the technology that powers video calls in browsers. What you might not know are all the features WebRTC unlocks for the web. In this talk we'll take a journey through WebRTC, from the primitives and protocols to the projects that have been built.

You'll learn the basics of getting two peers connected in a video chat and move on to advanced topics like camera control, screen sharing, audio analysis, using the data channel and more. We might even be able to solve the "You're on mute!" problem that plagues every video chat.

When you understand WebRTC and the related APIs you'll be amazed with what you can build.

--

Links:

Choosing cameras in JavaScript: https://www.twilio.com/blog/choosing-cameras-javascript-mediadevices-api-html
MediaDevices API examples: https://github.com/philnash/mediadevices-camera-selection
Screen capture with getDisplayMedia: https://www.twilio.com/blog/screen-sharing-javascript-twilio-programmable-video

RTC Diagnostics library. Test bitrate, audio input, audio output and video input in the browser: https://github.com/twilio/rtc-diagnostics/

Phil Nash

April 27, 2021
Tweet

More Decks by Phil Nash

See All by Phil Nash
Four steps from JavaScript to TypeScript
philnash
3
260
JavaScript Apps Go Intl - GIDS Live 2021
philnash
2
130
Better API DX with a CLI - APIdays Jakarta
philnash
0
240
The trouble with webhooks - at Apidays Live Hong Kong
philnash
2
110
Four steps from JavaScript to TypeScript
philnash
0
300
Fantastic passwords and where to find them - at NoRuKo
philnash
34
2.3k
The trouble with webhooks - at APIdays Singapore
philnash
0
190
What's going on with Project Fugu? at DevTalks Reimagined
philnash
0
250
Fantastic passwords and where to find them @ WFHConf
philnash
0
250

Other Decks in Programming

See All in Programming
​고군분투 LLM 프로덕트 적용기: Blind Prompting 부터 Agent까지
huffon
2
1.1k
飛び出せ！フロントエンド知見共有会~Next,Astro,Sveltekitを使ったエンジニアたちの声~ - NIFTY Tech Day 2023
niftycorp
0
130
Kaggle-Vesuvius-Challenge-Ink-Detection-2nd-Solution
mipypf
0
1.1k
フロントエンドの書くべきだったテスト、書かなくてよかったテスト
takefumiyoshii
33
11k
Microservices: what does good look like? (Nov 2023)
cer
PRO
0
700
技術書を読む技術（JJUG CCC 2023 Fall）
yonetty
1
270
コードを読みやすくしよう！〜秒で身に付くよい命名〜 (LT) - NIFTY Tech Day 2023
niftycorp
1
140
DIGGLEの分析基盤アーキテクチャ
zakky21
0
230
Next.js App Router での MPA フロントエンド刷新
mugi_uno
28
9.4k
.NET 8世代のBlazorについて
tomokusaba
0
160
GoにおけるCall Graphを用いた 大規模コードベースの影響調査
ffjlabo
0
270
Modern Java for the Masses! Is Java Still Relevant?
deepu105
1
460

Featured

See All Featured
Intergalactic Javascript Robots from Outer Space
tanoku
264
26k
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
130
9.8k
What's in a price? How to price your products and services
michaelherold
236
10k
4 Signs Your Business is Dying
shpigford
172
21k
JazzCon 2018 Closing Keynote - Leadership for the Reluctant Leader
reverentgeek
177
10k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
16
1.5k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
239
1.2M
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
52
19k
How To Stay Up To Date on Web Technology
chriscoyier
780
250k
The Brand Is Dead. Long Live the Brand.
mthomps
48
7.5k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
17
1.4k
Into the Great Unknown - MozCon
thekraken
7
660

Transcript

  1. YOU'RE ON MUTE!
    WEBRTC AND OUR LIVES ON SCREEN
    @philnash

    View Slide

  2. Phil Nash
    @philnash
    https://philna.sh
    [email protected]

    View Slide

  3. YOU'RE ON MUTE!
    WEBRTC AND OUR LIVES ON SCREEN
    @philnash

    View Slide

  4. OUR LIVES ON
    SCREEN
    @philnash

    View Slide

  5. WEBRTC
    VS
    @philnash

    View Slide

  6. GREAT FOR WORK,
    FRIENDS OR FAMILY
    @philnash

    View Slide

  7. NOT SO GREAT FOR
    ONE OFFS
    @philnash

    View Slide

  8. NOT SO GREAT FOR
    SPECIALIST APPS
    @philnash

    View Slide

  9. WEBRTC IS FOR ALL
    THE OTHER
    APPLICATIONS
    @philnash

    View Slide

  10. REAL-TIME VIDEO,
    AUDIO AND DATA
    EXPERIENCES FOR
    YOUR USERS
    @philnash

    View Slide

  11. Today
    1. Basics of WebRTC
    2. WebRTC Usage
    3. Controlling the user experience
    4. Tools and takeaways
    @philnash

    View Slide

  12. BASICS OF WEBRTC
    @philnash

    View Slide

  13. @philnash

    View Slide

  14. Signalling
    @philnash

    View Slide

  15. Peer to peer
    @philnash

    View Slide

  16. SFU
    @philnash

    View Slide

  17. ICE
    • ICE — Interactive Connectivity Establishment
    • NAT — Network Address Translation
    • STUN — Session Traversal Utilities for NAT
    • TURN — Traversal Using Relays around NAT
    @philnash

    View Slide

  18. THE APIS
    @philnash

    View Slide

  19. getUserMedia
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    }).then(stream => { // ... });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  20. RTCPeerConnection
    const pc = new RTCPeerConnection({ iceServers });
    pc.addEventListener("icecandidate", (candidate) => {
    // send candidate to signalling
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  21. RTCPeerConnection
    pc.addStream(stream);
    pc.createOffer().then(offer => {
    pc.setLocalDescription(offer);
    // send offer to peer
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  22. RTCPeerConnection
    pc.setRemoteDescription(offer);
    pc.createAnswer().then(answer => {
    pc.setLocalDescription(answer);
    // send answer to peer
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  23. RTCPeerConnection
    pc.setRemoteDescription(answer);
    pc.addEventListener("track", event => {
    // Connected!
    // show remote video in UI
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  24. RTCDataChannel
    const dataChannel = pc.createDataChannel("my-data-channel");
    dataChannel.addEventListener("message", event => {
    console.log(event.data);
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  25. RTCDataChannel
    pc.addEventListener("datachannel", event => {
    event.channel.addEventListener("open", () => {
    event.channel.send("Hi!");
    });
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  26. WEBRTC USAGE
    @philnash

    View Slide

  27. @philnash

    View Slide

  28. @philnash

    View Slide

  29. IMPROVING THE
    USER EXPERIENCE
    @philnash

    View Slide

  30. MEDIA
    @philnash

    View Slide

  31. getUserMedia
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    }).then(stream => { // ... });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  32. getUserMedia - Permissions
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    })
    .then(stream => { // success })
    .catch(error => { // permission denied });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  33. getUserMedia - Media Constraints
    {
    audio: true,
    video: true
    }
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  34. Media Constraints - facingMode
    {
    audio: true,
    video: {
    facingMode: "user" // "environment", "right", "left"
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  35. Media Constraints - deviceId
    {
    audio: true,
    video: {
    deviceId: { exact: device.id }
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  36. enumerateDevices
    navigator.mediaDevices.enumerateDevices()
    .then(devices => {
    devices.forEach(device => {
    console.log(`${device.id}: ${device.label}`);
    });
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  37. enumerateDevices - Permissions
    navigator.mediaDevices.enumerateDevices()
    .then(devices => {
    devices.forEach(device => {
    console.log(device.label); // ???
    });
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  38. Media Constraints - resolution
    {
    audio: true,
    video: {
    width: 1920,
    height: 1080
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    07.
    @philnash

    View Slide

  39. Media Constraints - resolution
    {
    video: {
    width: { min: 320, max: 1920, ideal: 1920 },
    height: { min: 240, max: 1080, ideal: 1080 }
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  40. Media Constraints - resolution
    {
    video: {
    width: { exact: 1920 },
    height: { exact: 1080 }
    }
    }
    // =>
    🚨 OverconstrainedError
    01.
    02.
    03.
    04.
    05.
    06.
    07.
    @philnash

    View Slide

  41. Media Constraints - torch
    🔦😄
    {
    video: {
    advanced: [{ torch: true }],
    }
    }
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  42. Media Constraints - torch
    🔦😄
    {
    video: {
    advanced: [{ torch: false }], // doesn't work
    🤨
    }
    }
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  43. Applying media constraints
    videoTrack.applyConstraints({
    width: 1920,
    height: 1080
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  44. Applying media constraints
    videoTrack.applyConstraints({
    deviceId: { exact: device.id }
    });
    ❌ Don't do this!
    ❌
    01.
    02.
    03.
    @philnash

    View Slide

  45. Changing cameras
    videoTrack.stop();
    navigator.mediaDevices.getUserMedia({
    video: {
    deviceId: { exact: device.id }
    }
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  46. DEMO
    @philnash

    View Slide

  47. RECOMMENDATION:
    ALWAYS PROVIDE A
    WAY TO CHECK AND
    CHANGE INPUTS
    @philnash

    View Slide

  48. RECOMMENDATION:
    UNDERSTAND THE
    AVAILABLE MEDIA
    CONSTRAINTS
    @philnash

    View Slide

  49. SCREEN SHARE
    @philnash

    View Slide

  50. Screen share
    navigator.mediaDevices.getDisplayMedia()
    .then(stream => {
    // add stream track to peer connection
    })
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  51. DEMO
    @philnash

    View Slide

  52. RECOMMENDATION:
    LET PEOPLE SHARE
    SCREENS WITHOUT
    AN EXTENSION
    @philnash

    View Slide

  53. WEB AUDIO API
    @philnash

    View Slide

  54. Web Audio API
    Not part of WebRTC
    Vital part of displaying audio to the user
    @philnash

    View Slide

  55. DEMO
    @philnash

    View Slide

  56. YOU'RE ON MUTE!
    @philnash

    View Slide

  57. DEMO
    @philnash

    View Slide

  58. DATA CHANNEL
    @philnash

    View Slide

  59. RTCDataChannel
    const dataChannel = pc.createDataChannel("my-data-channel");
    dataChannel.addEventListener("message", event => {
    console.log(event.data);
    });
    dataChannel.send("Hi!");
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  60. DEMO
    @philnash

    View Slide

  61. USEFUL TOOLS
    @philnash

    View Slide

  62. OPEN SOURCE
    DIAGNOSTICS
    @philnash

    View Slide

  63. DEMO
    @philnash

    View Slide

  64. RTC Diagnostics
    https://github.com/twilio/rtc-diagnostics/
    @philnash

    View Slide

  65. OUR LIVES ON
    SCREEN
    @philnash

    View Slide

  66. WEBRTC IS
    EVERYWHERE
    @philnash

    View Slide

  67. WEBRTC IS
    POWERFUL BECAUSE
    IT IS ON THE WEB
    @philnash

    View Slide

  68. WE HAVE THE TOOLS
    TO BUILD GREAT
    WEBRTC
    EXPERIENCES
    @philnash

    View Slide

  69. Thanks!
    @philnash
    https://philna.sh
    [email protected]

    View Slide