Share
Embed
Let's build a video calling app with Web technologies and Firebase!
Arnelle Balane
October 08, 2022
Transcript
66 slides
-
Software Developer, Newlogic Arnelle Balane @arnellebalane Let’s build a video
calling app with Web technologies and Firebase -
Arnelle Balane Software Developer at Newlogic Google Developers Expert for
Web Technologies I write about Web stuff on my blog, arnellebalane.com @arnellebalane -
1. Get streaming audio and video data 2. Determine network
information, such as IP addresses and ports to use 3. Determine information about media capabilities, such as resolutions and available media codecs Video calling apps need to: -
4. Facilitate communications to initiate or close connections, and exchange
network and media information 5. Stream audio and video data Video calling apps need to… -
• media input devices, like the user’s camera or microphone
We can obtain media streams from: navigator.mediaDevices.getUserMedia(); -
• a display device • can be used to share
a user’s screen We can obtain media streams from: navigator.mediaDevices.getDisplayMedia(); -
• an HTML canvas element • can be used to
add e.g. a whiteboard feature in the video calling app We can obtain media streams from: canvasElement.captureStream(); -
const peerConnection = new RTCPeerConnection(); local peer remote peer your
browser your friend’s browser direct peer-to-peer connection -
Add media stream to the peer connection mediaStream.getTracks().forEach((track) => {
peerConnection.addTrack(track, mediaStream); }); -
Concepts Interactive Connectivity Establishment (ICE) Negotiate the best way for
the peers to connect -
• Returns a peer’s public IP address and port STUN
(Session Traversal Utilities for NAT) • Relays data through an intermediary server in case direct connection is not available TURN (Traversal Using Relay NAT) -
• The process of determining IP addresses and ports that
can be used to establish a connection “finding candidates” • Each potential option for establishing a peer-to-peer connection ICE candidates -
Define ICE options for peer connection const peerConnection = new
RTCPeerConnection({ iceServers: [{ urls: ['stun:stun.l.google.com:19302'], }], }); -
v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0
a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000 -
v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0
a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000 -
1. Fetch API + Server-Sent Events 2. WebSockets 3. BroadcastChannel
API Signaling options -
(1) So you want to call your friend… you create
an “offer” // YOU const offer = await peerConnection.createOffer(); await peerConnection.setLocalDescription(offer); // to be continued in the next slide... -
(2) Send your “offer” to your friend through signaling //
YOU signaling.postMessage({ type: 'offer', payload: offer.toJSON(), }); -
(3) Your friend receives offer through signaling // YOUR FRIEND
signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'offer') { // to be continued in the next step... } }; -
(4) Your friend sets your offer as their remote peer
// YOUR FRIEND const peerConnection = new RTCPeerConnection(); await peerConnection.setRemoteDescription(payload); // payload contains your offer // to be continued in the next step... -
(5) Your friend creates an “answer” and sets it as
their local peer // YOUR FRIEND const answer = await peerConnection.createAnswer(); await peerConnection.setLocalDescription(answer); // to be continued in the next step... -
(6) Your friend sends their answer to you through signaling
// YOUR FRIEND signaling.postMessage({ type: 'answer', payload: answer.toJSON(), }); -
(7) You receive the “answer” and set it as your
remote peer // YOU signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'answer') { await peerConnection.setRemoteDescription(payload); // payload is your friend's answer } } -
onicecandidate when a new ICE candidate is determined // YOU
peerConnection.onicecandidate = (event) => { signaling.postMessage({ type: 'candidate', payload: event.candidate.toJSON(), }); }; -
onicecandidate when a new ICE candidate is determined // YOUR
FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'candidate') { await peerConnection.addIceCandidate(payload); // payload is your ice candidate } }; -
ontrack when our friend’s media stream arrives to us //
YOU peerConnection.ontrack = (event) => { displayMediaStream(event.streams[0]); // event.streams are our friend's media streams }; -
1. Allow many calls to happen at the same time
2. Allow multiple call participants 3. Allow anyone online to join a call Changes we’ll make -
1. Create a Firebase project, enable Firestore 2. Install Firebase
CLI 3. Initialize Firebase in our project 4. Change signaling mechanism to use Firestore Using Firebase -
Database structure collections and documents calls :call_id participants :participant_id peers
:peer_id candidates :candidate_id -
Import what we need We’ll use Firebase JS SDK v9
(modular) import { initializeApp } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-app.js'; import { getFirestore, collection, doc, addDoc, getDoc, setDoc, deleteDoc, getDocs, onSnapshot, } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-firestore.js'; -
Initialize app initializeApp({ apiKey: 'your-api-key', authDomain: 'project-id.firebaseapp.com', projectId: 'project-id', storageBucket:
'project-id.appspot.com', messagingSenderId: '...', appId: 'your-app-id', }); const db = getFirestore(); -
Start a call const params = new URLSearchParams(location.search); const callId
= params.get('call_id'); const participantsRef = collection( db, 'calls', callId, 'participants' ); const participantRef = await addDoc(participantsRef, {}); -
Establish P2P connections // Firestore :first-participant peers :second-participant - offer
:second-participant peers :first-participant - answer -
Establish P2P connections const participants = await getDocs(participantsRef); for (const
participantDoc of participants.docs) { if (participantDoc.id !== participantRef.id) { const offer = await offerPeerConnection(participantDoc.id); await setDoc( doc(participantDoc.ref, 'peers', participantRef.id), { offer } ); } } -
Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async
(snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.offer) { const answer = await answerPeerConnection(change.doc.id, data.offer); await setDoc( doc(participantsRef, change.doc.id, 'peers', participantRef.id), { answer } ); } } } }); -
Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async
(snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.answer) { await completePeerConnection(change.doc.id, data.answer); } } } }); -
Share ICE candidates // Firestore :first-participant candidates :candidate-id - peer_id
- candidate :candidate-id - peer_id - candidate -
Share ICE candidates peerConnection.onicecandidate = async ({ candidate }) =>
{ await addDoc( collection(participantsRef, change.doc.id, 'candidates'), { candidate, peerId: participantRef.id, } ); }; -
Share ICE candidates const candidatesRef = collection(participantRef, 'candidates'); onSnapshot(candidatesRef, async
(snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const { peerId, candidate } = change.doc.data(); await receiveRemoteIceCandidate(peerId, candidate); } } }); -
Leave a call const $leaveButton = document.querySelector('.leave-call'); $leaveButton.addEventListener('click', async ()
=> { await deleteDoc(participantRef); window.close(); }) -
Leave a call onSnapshot(participantsRef, async (snapshot) => { for (const
change of snapshot.docChanges()) { if (change.type === 'removed') { disconnectPeerConnection(change.doc.id); } } });