Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let's build a video calling app with Web techno...

Let's build a video calling app with Web technologies and Firebase!

Arnelle Balane

October 08, 2022
Tweet

More Decks by Arnelle Balane

Other Decks in Programming

Transcript

  1. Arnelle Balane Software Developer at Newlogic Google Developers Expert for

    Web Technologies I write about Web stuff on my blog, arnellebalane.com @arnellebalane
  2. 1. Get streaming audio and video data 2. Determine network

    information, such as IP addresses and ports to use 3. Determine information about media capabilities, such as resolutions and available media codecs Video calling apps need to:
  3. 4. Facilitate communications to initiate or close connections, and exchange

    network and media information 5. Stream audio and video data Video calling apps need to…
  4. • media input devices, like the user’s camera or microphone

    We can obtain media streams from: navigator.mediaDevices.getUserMedia();
  5. • a display device • can be used to share

    a user’s screen We can obtain media streams from: navigator.mediaDevices.getDisplayMedia();
  6. • an HTML canvas element • can be used to

    add e.g. a whiteboard feature in the video calling app We can obtain media streams from: canvasElement.captureStream();
  7. const peerConnection = new RTCPeerConnection(); local peer remote peer your

    browser your friend’s browser direct peer-to-peer connection
  8. • Returns a peer’s public IP address and port STUN

    (Session Traversal Utilities for NAT) • Relays data through an intermediary server in case direct connection is not available TURN (Traversal Using Relay NAT)
  9. • The process of determining IP addresses and ports that

    can be used to establish a connection “finding candidates” • Each potential option for establishing a peer-to-peer connection ICE candidates
  10. Define ICE options for peer connection const peerConnection = new

    RTCPeerConnection({ iceServers: [{ urls: ['stun:stun.l.google.com:19302'], }], });
  11. v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0

    a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000
  12. v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0

    a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000
  13. (1) So you want to call your friend… you create

    an “offer” // YOU const offer = await peerConnection.createOffer(); await peerConnection.setLocalDescription(offer); // to be continued in the next slide...
  14. (2) Send your “offer” to your friend through signaling //

    YOU signaling.postMessage({ type: 'offer', payload: offer.toJSON(), });
  15. (3) Your friend receives offer through signaling // YOUR FRIEND

    signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'offer') { // to be continued in the next step... } };
  16. (4) Your friend sets your offer as their remote peer

    // YOUR FRIEND const peerConnection = new RTCPeerConnection(); await peerConnection.setRemoteDescription(payload); // payload contains your offer // to be continued in the next step...
  17. (5) Your friend creates an “answer” and sets it as

    their local peer // YOUR FRIEND const answer = await peerConnection.createAnswer(); await peerConnection.setLocalDescription(answer); // to be continued in the next step...
  18. (6) Your friend sends their answer to you through signaling

    // YOUR FRIEND signaling.postMessage({ type: 'answer', payload: answer.toJSON(), });
  19. (7) You receive the “answer” and set it as your

    remote peer // YOU signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'answer') { await peerConnection.setRemoteDescription(payload); // payload is your friend's answer } }
  20. onicecandidate when a new ICE candidate is determined // YOU

    peerConnection.onicecandidate = (event) => { signaling.postMessage({ type: 'candidate', payload: event.candidate.toJSON(), }); };
  21. onicecandidate when a new ICE candidate is determined // YOUR

    FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'candidate') { await peerConnection.addIceCandidate(payload); // payload is your ice candidate } };
  22. ontrack when our friend’s media stream arrives to us //

    YOU peerConnection.ontrack = (event) => { displayMediaStream(event.streams[0]); // event.streams are our friend's media streams };
  23. 1. Allow many calls to happen at the same time

    2. Allow multiple call participants 3. Allow anyone online to join a call Changes we’ll make
  24. 1. Create a Firebase project, enable Firestore 2. Install Firebase

    CLI 3. Initialize Firebase in our project 4. Change signaling mechanism to use Firestore Using Firebase
  25. Import what we need We’ll use Firebase JS SDK v9

    (modular) import { initializeApp } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-app.js'; import { getFirestore, collection, doc, addDoc, getDoc, setDoc, deleteDoc, getDocs, onSnapshot, } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-firestore.js';
  26. Initialize app initializeApp({ apiKey: 'your-api-key', authDomain: 'project-id.firebaseapp.com', projectId: 'project-id', storageBucket:

    'project-id.appspot.com', messagingSenderId: '...', appId: 'your-app-id', }); const db = getFirestore();
  27. Start a call const params = new URLSearchParams(location.search); const callId

    = params.get('call_id'); const participantsRef = collection( db, 'calls', callId, 'participants' ); const participantRef = await addDoc(participantsRef, {});
  28. Establish P2P connections const participants = await getDocs(participantsRef); for (const

    participantDoc of participants.docs) { if (participantDoc.id !== participantRef.id) { const offer = await offerPeerConnection(participantDoc.id); await setDoc( doc(participantDoc.ref, 'peers', participantRef.id), { offer } ); } }
  29. Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async

    (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.offer) { const answer = await answerPeerConnection(change.doc.id, data.offer); await setDoc( doc(participantsRef, change.doc.id, 'peers', participantRef.id), { answer } ); } } } });
  30. Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async

    (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.answer) { await completePeerConnection(change.doc.id, data.answer); } } } });
  31. Share ICE candidates peerConnection.onicecandidate = async ({ candidate }) =>

    { await addDoc( collection(participantsRef, change.doc.id, 'candidates'), { candidate, peerId: participantRef.id, } ); };
  32. Share ICE candidates const candidatesRef = collection(participantRef, 'candidates'); onSnapshot(candidatesRef, async

    (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const { peerId, candidate } = change.doc.data(); await receiveRemoteIceCandidate(peerId, candidate); } } });
  33. Leave a call onSnapshot(participantsRef, async (snapshot) => { for (const

    change of snapshot.docChanges()) { if (change.type === 'removed') { disconnectPeerConnection(change.doc.id); } } });