Let's build a video calling app with Web technologies and Firebase!

Transcript

1. Software Developer, Newlogic Arnelle Balane @arnellebalane Let’s build a video calling app with Web technologies and Firebase
2. Arnelle Balane Software Developer at Newlogic Google Developers Expert for Web Technologies I write about Web stuff on my blog, arnellebalane.com @arnellebalane
3. WebRTC
4. Part 1 of 2 Concepts
5. 1. Get streaming audio and video data 2. Determine network information, such as IP addresses and ports to use 3. Determine information about media capabilities, such as resolutions and available media codecs Video calling apps need to:
6. 4. Facilitate communications to initiate or close connections, and exchange network and media information 5. Stream audio and video data Video calling apps need to…
7. Represents a stream of audio and/or video content https://developer.mozilla.org/en-US/docs/Web/API/MediaStream Concepts MediaStream API
8. • media input devices, like the user’s camera or microphone We can obtain media streams from: navigator.mediaDevices.getUserMedia();
9. • a display device • can be used to share a user’s screen We can obtain media streams from: navigator.mediaDevices.getDisplayMedia();
10. • an HTML canvas element • can be used to add e.g. a whiteboard feature in the video calling app We can obtain media streams from: canvasElement.captureStream();
11. const mediaStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true, });
12. Concepts RTCPeerConnection API Represents a connection between two peers https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection
13. const peerConnection = new RTCPeerConnection(); local peer remote peer your browser your friend’s browser direct peer-to-peer connection
14. Add media stream to the peer connection mediaStream.getTracks().forEach((track) => { peerConnection.addTrack(track, mediaStream); });
15. Concepts Interactive Connectivity Establishment (ICE) Negotiate the best way for the peers to connect
16. Image from https://web.dev/webrtc-basics/
17. • Returns a peer’s public IP address and port STUN (Session Traversal Utilities for NAT) • Relays data through an intermediary server in case direct connection is not available TURN (Traversal Using Relay NAT)
18. • The process of determining IP addresses and ports that can be used to establish a connection “finding candidates” • Each potential option for establishing a peer-to-peer connection ICE candidates
19. Define ICE options for peer connection const peerConnection = new RTCPeerConnection({ iceServers: [{ urls: ['stun:stun.l.google.com:19302'], }], });
20. Concepts Session Description Protocol (SDP) Describes a peer’s network and media information
21. v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000
22. v=0 o=- 6273550205898152643 4 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 2 a=extmap-allow-mixed a=msid-semantic: WMS i5NpLCJMg7hiYh1kG5YyR6EIvxcfO8QgZ3OA m=video 60159 UDP/TLS/RTP/SAVPF 96 97 102 122 127 121 125 107 108 109 124 120 39 40 45 46 98 99 100 101 123 119 114 115 116 c=IN IP4 49.145.37.142 a=rtcp:9 IN IP4 0.0.0.0 a=candidate:4077567720 1 udp 2122260223 192.168.1.10 65452 typ host generation 0 network-id 1 network-cost 10 a=candidate:85641020 1 udp 1686052607 49.145.37.142 60159 typ srflx raddr 192.168.1.10 rport 65452 generation 0 network-id 1 network-cost 10 a=candidate:3179889176 1 tcp 1518280447 192.168.1.10 9 typ host tcptype active generation 0 network-id 1 network-cost 10 a=ice-ufrag:PUyC a=ice-pwd:eJeVV3MKP5+MDfPYAA6WUcKV a=rtpmap:96 VP8/90000 a=rtpmap:108 H264/90000
23. Concepts Signaling Facilitate communications between peers (to exchange SDP, initiate and end calls, etc.)
24. 1. Fetch API + Server-Sent Events 2. WebSockets 3. BroadcastChannel API Signaling options
25. Initialize the signaling mechanism const signaling = new BroadcastChannel('signaling');
26. (1) So you want to call your friend… you create an “offer” // YOU const offer = await peerConnection.createOffer(); await peerConnection.setLocalDescription(offer); // to be continued in the next slide...
27. (2) Send your “offer” to your friend through signaling // YOU signaling.postMessage({ type: 'offer', payload: offer.toJSON(), });
28. (3) Your friend receives offer through signaling // YOUR FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'offer') { // to be continued in the next step... } };
29. (4) Your friend sets your offer as their remote peer // YOUR FRIEND const peerConnection = new RTCPeerConnection(); await peerConnection.setRemoteDescription(payload); // payload contains your offer // to be continued in the next step...
30. (5) Your friend creates an “answer” and sets it as their local peer // YOUR FRIEND const answer = await peerConnection.createAnswer(); await peerConnection.setLocalDescription(answer); // to be continued in the next step...
31. (6) Your friend sends their answer to you through signaling // YOUR FRIEND signaling.postMessage({ type: 'answer', payload: answer.toJSON(), });
32. (7) You receive the “answer” and set it as your remote peer // YOU signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'answer') { await peerConnection.setRemoteDescription(payload); // payload is your friend's answer } }
33. 🎉 ✨🎉 Connection established!
34. Concepts Events Just a couple more things to handle…
35. onicecandidate when a new ICE candidate is determined // YOU peerConnection.onicecandidate = (event) => { signaling.postMessage({ type: 'candidate', payload: event.candidate.toJSON(), }); };
36. onicecandidate when a new ICE candidate is determined // YOUR FRIEND signaling.onmessage = async (event) => { const { type, payload } = event.data; if (type === 'candidate') { await peerConnection.addIceCandidate(payload); // payload is your ice candidate } };
37. ontrack when our friend’s media stream arrives to us // YOU peerConnection.ontrack = (event) => { displayMediaStream(event.streams[0]); // event.streams are our friend's media streams };
38. Link to full code 👇 https://github.com/arnellebalane/video-conference -app/tree/webrtc 🎉 ✨🎉 Demo time!
39. Part 2 of 2 Using Firebase
40. 1. Allow many calls to happen at the same time 2. Allow multiple call participants 3. Allow anyone online to join a call Changes we’ll make
41. 1. Create a Firebase project, enable Firestore 2. Install Firebase CLI 3. Initialize Firebase in our project 4. Change signaling mechanism to use Firestore Using Firebase
42. Database structure collections and documents calls :call_id participants :participant_id peers :peer_id candidates :candidate_id
43. Import what we need We’ll use Firebase JS SDK v9 (modular) import { initializeApp } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-app.js'; import { getFirestore, collection, doc, addDoc, getDoc, setDoc, deleteDoc, getDocs, onSnapshot, } from 'https://www.gstatic.com/firebasejs/9.10.0/firebase-firestore.js';
44. Initialize app initializeApp({ apiKey: 'your-api-key', authDomain: 'project-id.firebaseapp.com', projectId: 'project-id', storageBucket: 'project-id.appspot.com', messagingSenderId: '...', appId: 'your-app-id', }); const db = getFirestore();
45. Start a call https://call-app.com?call_id=any-call-id // Firestore calls :any-call-id participants :first-participant
46. Start a call const params = new URLSearchParams(location.search); const callId = params.get('call_id'); const participantsRef = collection( db, 'calls', callId, 'participants' ); const participantRef = await addDoc(participantsRef, {});
47. Start a call
48. Join a call https://call-app.com?call_id=any-call-id // Firestore calls :any-call-id participants :first-participant :second-participant
49. Establish P2P connections // Firestore :first-participant peers :second-participant - offer
50. Establish P2P connections // Firestore :first-participant peers :second-participant - offer :second-participant peers :first-participant - answer
51. Establish P2P connections const participants = await getDocs(participantsRef); for (const participantDoc of participants.docs) { if (participantDoc.id !== participantRef.id) { const offer = await offerPeerConnection(participantDoc.id); await setDoc( doc(participantDoc.ref, 'peers', participantRef.id), { offer } ); } }
52. Establish P2P connections
53. Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.offer) { const answer = await answerPeerConnection(change.doc.id, data.offer); await setDoc( doc(participantsRef, change.doc.id, 'peers', participantRef.id), { answer } ); } } } });
54. Establish P2P connections const peersRef = collection(participantRef, 'peers'); onSnapshot(peersRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const data = change.doc.data(); if (data.answer) { await completePeerConnection(change.doc.id, data.answer); } } } });
55. Establish P2P connections
56. Share ICE candidates // Firestore :first-participant candidates :candidate-id - peer_id - candidate :candidate-id - peer_id - candidate
57. Share ICE candidates peerConnection.onicecandidate = async ({ candidate }) => { await addDoc( collection(participantsRef, change.doc.id, 'candidates'), { candidate, peerId: participantRef.id, } ); };
58. Share ICE candidates const candidatesRef = collection(participantRef, 'candidates'); onSnapshot(candidatesRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'added') { const { peerId, candidate } = change.doc.data(); await receiveRemoteIceCandidate(peerId, candidate); } } });
59. Share ICE candidates
60. Leave a call const $leaveButton = document.querySelector('.leave-call'); $leaveButton.addEventListener('click', async () => { await deleteDoc(participantRef); window.close(); })
61. Leave a call onSnapshot(participantsRef, async (snapshot) => { for (const change of snapshot.docChanges()) { if (change.type === 'removed') { disconnectPeerConnection(change.doc.id); } } });
62. Link to full code 👇 https://github.com/arnellebalane/video-conference -app/tree/main 🎉 ✨🎉 Demo time!
63. 1. Use Firebase Local Emulator for Firestore 2. chrome://webrtc-internals Additional tips
64. Shameless plugs 😂 https://dev.to/arnellebalane/lets-build-a-video -calling-app-part-one-2ggl (Part Two is still in progress, maybe next week)
65. Shameless plugs 😂 Repository: https://github.com/arnellebalane /video-conference-app Learn more 👇 https://hacktoberfest.com
66. 🤔🤔🤔 Any questions?