Build WebRTC MCU on Browser

Build MCU for WebRTC on Web browser.
Using WebRTC, Canvas, Web Audio.
Original presentation was made at WebRTC meetup Tokyo #11.
Youtube https://youtu.be/0RYWw1nPxio?t=50m41s

Transcript

1. WebRTC Meetup Tokyo #11 Build WebRTC MCU on Browser INFOCOM CORPORATION Masashi Ganeko @massie_g Aug. 8th, 2016 1
2. About me • Masashi Ganeko / @massie_g – working for a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013, 2016) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2016) – http://www.slideshare.net/mganeko 2
3. Topic of today • MCU : Multipoint Control Unit – Used for Video Conference • Web Browser – WebRTC, Canvas, Web Audio 3
4. P2P vs. MCU 4 Browser A Browser B Browser D Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load Good for Mobile
5. Past trial：WebRTC Meetup Tokyo #4 • Absurd trial with WebRTC ＋α (Oct. 2014) – (3) mix video absurd way – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - ) 5 HDMI Capture unit Mix video with HDMI capture
6. It is possible with only browser at 2016 • Canvas captureStream() – Firefox 43- , Chrome 51- • Handle remote audio with Web Audio API – Firefox (around)40-, Chrome 49- 6
7. Browser MCU DEMO • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=54m4s – 54:04 to 55:22 7
8. Browser MCU DEMO 8 Grid mix
9. Inside MCU Server：Video 9 RTCPeerConnection A MediaStream <video> element RTCPeerConnection D MediaStream <canvas> element drawImage() Keep drawing with requestAnimationFrame() <video> element drawImage()
10. Inside MCU Server：Video 10 RTCPeerConnection A RTCPeerConnection D <canvas> element MediaStream captureStream(fps)
11. Inside MCU Server：Video 11 RTCPeerConnection A MediaStream <video> element RTCPeerConnection D MediaStream <canvas> element MediaStream drawImage() Keep drawing with requestAnimationFrame() <video> element captureStream(fps) drawImage()
12. Process of MCU Server：Video • Get Remote MediaStream through RTCPeerConnection – play in <video> element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimationFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnection.addStream(), and send it to remote peer 12
13. Tricky points of MCU Server：Video • Using window.requestAnimationFrame() – Will not be called when the window/tab is completely hidden – Animation will stop if the window is minimized or other hidden by other tab • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17 • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 13 DEMO
14. Tricky points of MCU Server：Audio • One mixed video is enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC application developers – If it is not familiar to you, please try yourself • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html • Please use headphone, and laud volume • Click [Start] button • It is so hard to speak 14 DEMO
15. Inside MCU Server：Audio • Prepare "minus one" Audio – Mix audio of all members, except the user • Need many combination of audio – 4 combination for 4 members – N combination for N members 15 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
16. Inside MCU Server：Audio 16 RTCPeerConnection A MediaStream create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode
17. Inside MCU Server：Audio 17 RTCPeerConnection D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix
18. Inside MCU Server：Audio 18 RTCPeerConnection A MediaStream RTCPeerConnection D MediaStream create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix
19. Process of MCU Servder：Audio • Get remote MediaStream through RTCPeerConnection • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDestinationNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(destinationNode); • Get MediaStream – Pass to RTCPeerConnection. addStream(), and send to remote peer • * repeat for all members 19
20. Inside of MCU Server：Video & Audio 20 RTCPeerConnection A RTCPeerConnection D MediaStream (Video A+B+C+D) MediaStream (Audio B+C+D) RTCPeerConnection B RTCPeerConnection C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Multi-Stream with Video only MediaStream and Audio only MediaStream
21. Browser MCU DEMO 2 • Canvas is So flexible ! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h35s – 1:00:35 to 1:01:29 21
22. Browser MCU DEMO 2 22 Stripe Variable Zoom
23. Browser MCU DEMO 3 • 3D with Canvas + WebGL • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h1m51s – 1:01:51 to 1:02:21 23
24. Browser MCU DEMO 3 24 3D with WebGL
25. Browser MCU DEMO 3 25 Stripe with 7 members Grid with 17 members Frame was dropped, in case of over 16 members Too heavy load, even for core-i7 CPU
26. Browser MCU DEMO 4 26 • Record mix video/audio with MediaRecorder • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h3m47s – 1:03:47 to 1:04:33
27. Inside of MCU Server: Recording 27 MediaStream (Video A+B+C+D) <canvas> element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C+D) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM
28. MCU vs. SFU 28
29. SFU: Selective Forwarding Unit 29 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load
30. Browser MCU DEMO 5 • It is NOT possible to choose display layout on member side, with MCU • ... NO, it is possible! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h6m49s – 1:06:49 to 1:07:25 30
31. Browser MCU DEMO 5 31 Zooming on User/Member side
32. Inside of zooming on MCU member 32 MediaStream (Video A+B+C+D) hidden <video> element drawImage() drawImage() drawImage() drawImage() It is possible to change layout with Canvas in member side (It is possible to use even multiple canvas)
33. Appendix：painful point of MCU • Adjusting timing is painful for commercial MCU – Adjust timing for video of all members – Adjust timing of video and audio (lip sync) • For adjusting timing, it is need to wait for frame or block – But, impossible to wait forever – So it is necessary to timeout and send alternative frame/block • Sending Black frame, or repeat last frame • Sending silent block • Browser MCU does not care about adjusting timing 33
34. Appendix：MCU Sample page • Start page (not http, but https) – https://mcu.talkin.info/ • Prepare a room as MCU server – Type room name – Click [Prepare a Room as MCU Server] – Wait for connection, or click [Re-Connect] • Join the room – Type room name, then click [Join to Room] – Click [Start video] to use camera/microphone – Click [Connect] 34
35. Conclusion • Potential of WebRTC + Canvas is unlimited – It is very powerful toy – It is possible to send local video file • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37 • But, CPU load of server side is super heavy – It is not scalable • Maybe useful, if user can take care of server side browser • Original Japanese Slide is on SlideShare – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser 35
36. Thank you! 36