$30 off During Our Annual Pro Sale. View Details »

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

You're on Mute! WebRTC and our Lives on Screen - GIDS Live 2021

WebRTC is the technology that powers video calls in browsers. What you might not know are all the features WebRTC unlocks for the web. In this talk we'll take a journey through WebRTC, from the primitives and protocols to the projects that have been built.

You'll learn the basics of getting two peers connected in a video chat and move on to advanced topics like camera control, screen sharing, audio analysis, using the data channel and more. We might even be able to solve the "You're on mute!" problem that plagues every video chat.

When you understand WebRTC and the related APIs you'll be amazed with what you can build.

--

Links:

Choosing cameras in JavaScript: https://www.twilio.com/blog/choosing-cameras-javascript-mediadevices-api-html
MediaDevices API examples: https://github.com/philnash/mediadevices-camera-selection
Screen capture with getDisplayMedia: https://www.twilio.com/blog/screen-sharing-javascript-twilio-programmable-video

RTC Diagnostics library. Test bitrate, audio input, audio output and video input in the browser: https://github.com/twilio/rtc-diagnostics/

Phil Nash

April 27, 2021
Tweet

More Decks by Phil Nash

Other Decks in Programming

Transcript

  1. YOU'RE ON MUTE!
    WEBRTC AND OUR LIVES ON SCREEN
    @philnash

    View Slide

  2. Phil Nash
    @philnash
    https://philna.sh
    [email protected]

    View Slide

  3. YOU'RE ON MUTE!
    WEBRTC AND OUR LIVES ON SCREEN
    @philnash

    View Slide

  4. OUR LIVES ON
    SCREEN
    @philnash

    View Slide

  5. WEBRTC
    VS
    @philnash

    View Slide

  6. GREAT FOR WORK,
    FRIENDS OR FAMILY
    @philnash

    View Slide

  7. NOT SO GREAT FOR
    ONE OFFS
    @philnash

    View Slide

  8. NOT SO GREAT FOR
    SPECIALIST APPS
    @philnash

    View Slide

  9. WEBRTC IS FOR ALL
    THE OTHER
    APPLICATIONS
    @philnash

    View Slide

  10. REAL-TIME VIDEO,
    AUDIO AND DATA
    EXPERIENCES FOR
    YOUR USERS
    @philnash

    View Slide

  11. Today
    1. Basics of WebRTC
    2. WebRTC Usage
    3. Controlling the user experience
    4. Tools and takeaways
    @philnash

    View Slide

  12. BASICS OF WEBRTC
    @philnash

    View Slide

  13. @philnash

    View Slide

  14. Signalling
    @philnash

    View Slide

  15. Peer to peer
    @philnash

    View Slide

  16. SFU
    @philnash

    View Slide

  17. ICE
    • ICE — Interactive Connectivity Establishment
    • NAT — Network Address Translation
    • STUN — Session Traversal Utilities for NAT
    • TURN — Traversal Using Relays around NAT
    @philnash

    View Slide

  18. THE APIS
    @philnash

    View Slide

  19. getUserMedia
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    }).then(stream => { // ... });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  20. RTCPeerConnection
    const pc = new RTCPeerConnection({ iceServers });
    pc.addEventListener("icecandidate", (candidate) => {
    // send candidate to signalling
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  21. RTCPeerConnection
    pc.addStream(stream);
    pc.createOffer().then(offer => {
    pc.setLocalDescription(offer);
    // send offer to peer
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  22. RTCPeerConnection
    pc.setRemoteDescription(offer);
    pc.createAnswer().then(answer => {
    pc.setLocalDescription(answer);
    // send answer to peer
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  23. RTCPeerConnection
    pc.setRemoteDescription(answer);
    pc.addEventListener("track", event => {
    // Connected!
    // show remote video in UI
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  24. RTCDataChannel
    const dataChannel = pc.createDataChannel("my-data-channel");
    dataChannel.addEventListener("message", event => {
    console.log(event.data);
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  25. RTCDataChannel
    pc.addEventListener("datachannel", event => {
    event.channel.addEventListener("open", () => {
    event.channel.send("Hi!");
    });
    });
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  26. WEBRTC USAGE
    @philnash

    View Slide

  27. @philnash

    View Slide

  28. @philnash

    View Slide

  29. IMPROVING THE
    USER EXPERIENCE
    @philnash

    View Slide

  30. MEDIA
    @philnash

    View Slide

  31. getUserMedia
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    }).then(stream => { // ... });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  32. getUserMedia - Permissions
    navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
    })
    .then(stream => { // success })
    .catch(error => { // permission denied });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  33. getUserMedia - Media Constraints
    {
    audio: true,
    video: true
    }
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  34. Media Constraints - facingMode
    {
    audio: true,
    video: {
    facingMode: "user" // "environment", "right", "left"
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  35. Media Constraints - deviceId
    {
    audio: true,
    video: {
    deviceId: { exact: device.id }
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  36. enumerateDevices
    navigator.mediaDevices.enumerateDevices()
    .then(devices => {
    devices.forEach(device => {
    console.log(`${device.id}: ${device.label}`);
    });
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  37. enumerateDevices - Permissions
    navigator.mediaDevices.enumerateDevices()
    .then(devices => {
    devices.forEach(device => {
    console.log(device.label); // ???
    });
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  38. Media Constraints - resolution
    {
    audio: true,
    video: {
    width: 1920,
    height: 1080
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    07.
    @philnash

    View Slide

  39. Media Constraints - resolution
    {
    video: {
    width: { min: 320, max: 1920, ideal: 1920 },
    height: { min: 240, max: 1080, ideal: 1080 }
    }
    }
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  40. Media Constraints - resolution
    {
    video: {
    width: { exact: 1920 },
    height: { exact: 1080 }
    }
    }
    // =>
    🚨 OverconstrainedError
    01.
    02.
    03.
    04.
    05.
    06.
    07.
    @philnash

    View Slide

  41. Media Constraints - torch
    🔦😄
    {
    video: {
    advanced: [{ torch: true }],
    }
    }
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  42. Media Constraints - torch
    🔦😄
    {
    video: {
    advanced: [{ torch: false }], // doesn't work
    🤨
    }
    }
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  43. Applying media constraints
    videoTrack.applyConstraints({
    width: 1920,
    height: 1080
    });
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  44. Applying media constraints
    videoTrack.applyConstraints({
    deviceId: { exact: device.id }
    });
    ❌ Don't do this!

    01.
    02.
    03.
    @philnash

    View Slide

  45. Changing cameras
    videoTrack.stop();
    navigator.mediaDevices.getUserMedia({
    video: {
    deviceId: { exact: device.id }
    }
    });
    01.
    02.
    03.
    04.
    05.
    06.
    @philnash

    View Slide

  46. DEMO
    @philnash

    View Slide

  47. RECOMMENDATION:
    ALWAYS PROVIDE A
    WAY TO CHECK AND
    CHANGE INPUTS
    @philnash

    View Slide

  48. RECOMMENDATION:
    UNDERSTAND THE
    AVAILABLE MEDIA
    CONSTRAINTS
    @philnash

    View Slide

  49. SCREEN SHARE
    @philnash

    View Slide

  50. Screen share
    navigator.mediaDevices.getDisplayMedia()
    .then(stream => {
    // add stream track to peer connection
    })
    01.
    02.
    03.
    04.
    @philnash

    View Slide

  51. DEMO
    @philnash

    View Slide

  52. RECOMMENDATION:
    LET PEOPLE SHARE
    SCREENS WITHOUT
    AN EXTENSION
    @philnash

    View Slide

  53. WEB AUDIO API
    @philnash

    View Slide

  54. Web Audio API
    Not part of WebRTC
    Vital part of displaying audio to the user
    @philnash

    View Slide

  55. DEMO
    @philnash

    View Slide

  56. YOU'RE ON MUTE!
    @philnash

    View Slide

  57. DEMO
    @philnash

    View Slide

  58. DATA CHANNEL
    @philnash

    View Slide

  59. RTCDataChannel
    const dataChannel = pc.createDataChannel("my-data-channel");
    dataChannel.addEventListener("message", event => {
    console.log(event.data);
    });
    dataChannel.send("Hi!");
    01.
    02.
    03.
    04.
    05.
    @philnash

    View Slide

  60. DEMO
    @philnash

    View Slide

  61. USEFUL TOOLS
    @philnash

    View Slide

  62. OPEN SOURCE
    DIAGNOSTICS
    @philnash

    View Slide

  63. DEMO
    @philnash

    View Slide

  64. RTC Diagnostics
    https://github.com/twilio/rtc-diagnostics/
    @philnash

    View Slide

  65. OUR LIVES ON
    SCREEN
    @philnash

    View Slide

  66. WEBRTC IS
    EVERYWHERE
    @philnash

    View Slide

  67. WEBRTC IS
    POWERFUL BECAUSE
    IT IS ON THE WEB
    @philnash

    View Slide

  68. WE HAVE THE TOOLS
    TO BUILD GREAT
    WEBRTC
    EXPERIENCES
    @philnash

    View Slide

  69. Thanks!
    @philnash
    https://philna.sh
    [email protected]

    View Slide