Build WebRTC iOS Gateway on Browser

Building WebRTC - iOS Safari Gateway on PC Web Browser, using webaudio, canvas, websocket.
This is a presentation for WebRTC Meetup Tokyo 14 at Mar. 23, 2017

Transcript

1. WebRTC Meetup Tokyo #14 Build WebRTC - iOS Gateway on Browser INFOCOM CORPORATION Masashi Ganeko @massie_g Mar. 23th, 2017 1
2. About me •  Masashi Ganeko / @massie_g –  working for a research team of INFOCOM CORPORATION •  WriRng a series of WebRTC on HTML5Experts.jp (HTML5 tech site) –  h[ps://html5experts.jp/series/webrtc2016/ •  English PresentaRon for WebRTC (2013-2017) –  h[ps://speakerdeck.com/mganeko •  Japanese PresentaRon for WebRTC (2013-2017) –  h[p://www.slideshare.net/mganeko 2
3. Topic of today •  iOS Browser does not support WebRTC yet –  Webkit is on the way, but Safari may take much longer •  It is possible to communicate today, with other features –  Canvas, Web Audio, WebSocket •  Thanks for preceding trials: –  Audio: "Using WebAudio for pseudo WebRTC" @leader22 •  h[p://leader22.github.io/slides/webaudio_tokyo-1/ –  Video: Snapshoto from "Introducing WebRTC SFU Sora" @voluntas •  h[ps://gist.github.com/voluntas/0d6621d15947a24e710b0610093a5d20 3
4. DEMO •  PC Chrome ßà iOS Safari •  Source code –  h[ps://github.com/mganeko/webrtcexpjp/tree/master/gateway •  (There are some problems with handling portrait photo) 4
5. PC Chrome à iOS Safari (1) Video 5 getUserMedia() MediaStream <video> <canvas> Blob JPEB setInterval() drawImage() toBlob() Worker WebSocket (socket.io) Blob JPEB socket.io Server postMessage() Worker WebSocket (socket.io) ArrayBuffer postMessage() ArrayBuffer <img> Blob createObjectURL() PC Chrome iOS Safari
6. PC Chrome à iOS Safari (2) Audio 6 getUserMedia() MediaStream Worker WebSocket (socket.io) socket.io Server Worker a WebSocket (socket.io) PC Chrome iOS Safari WebAudio ScriptProcessor [Float32Array] MediaStreamAudioSourceNode [Uint8Array] compress postMessage() ArrayBuffer (decompress) WebAudio 6 AudioBufferSource ArrayBuffer decoceAudioData() AudioDesRnaRonNode postMessage()
7. PC Chrome ß iOS Safari : Image 7 <img> toDataURL () Worker WebSocket (socket.io) DataURL socket.io Server postMessage() <img> <img> <img> <canvas> drawImage() DataURL toBlob () <img> DataURL img.src Worker WebSocket (socket.io) postMessage() DataURL iOS Safari PC Chrome
8. Reduce data size: Video/Image •  While using TCP, delay will occur on high traffic condiRon –  It is not preferable for real-Rme video/audio communicaRon –  Too keep enough margin, data should be reduced •  Image 240x180 / sec –  RGB RAW Image …　240x180x24bit = 1 Mbit, 129 kb –  PNG … about 760 kbit, 96 kb –  JPEG … about 110 kbit, 14 kb –  → reduce frequency to once per 2 sec. 　… 56 kbit/sec 8
9. Reducing Image data 9 1/8
10. Reduce data size: Audio 10 •  Sampling rate of WebAudio is plaporm dependent. ( NOT browser dependent ) –  44.1kHz (Mac)　←　Use this rate for calculaRon –  48kHz (Windows, Linux, iOS) •  Compression –  WebAudio original data …44.1kHz × 32bit float = 1411 kbps –  16bit liner PCM … 44.1kHz × 16bit Integer = 705 kbps •  CD is 44.1kHz ×16bit × 2ch(Stereo) = 1411 kbps –  Thin out samples to reduce rate to 22 kHz … 350 kbps –  μ-law compression: 22kHz, 8bit Integer … 175 kbps •  G.711 (PCMU) : 8kHz, 8bit Integer … 64kbps –  mp3 compression … use 36 kbps bitrate
11. 11 1/40 Reducing Audio data
12. μ-law algorithm •  Irreversible audio compress algorithm •  Reduce bits of each sample •  Used in G.711 for VoIP –  G.711: Sampling rate 8kHz → bitrate 64kbps •  Wikipedia –  h[ps://en.wikipedia.org/wiki/G.711 –  PCMU … Use μ-law in North America and Japan –  PCMA … Use A-law in Europe •  22kHz μ-law is enough for WiFi, but not enough for mobile •  Ex) Trying primiRve audio compression with WebAudio (Japanese lang.) –  h[p://qiita.com/massie_g/items/49183a03b015b9ea27eb 12
13. Logarithm representaRon in μ-law 13 Linear 8 steps (3bit PCM) Logarithm 8 steps: fine for small, rough for big
14. μ-law algorithm 14 μ = 255（8ビット）
15. PCMU/PCMA in WebRTC 15 [Firefox SDP] a=rtpmap:109 opus/48000/2 a=rtpmap:9 G722/8000/1 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 [Chrome SDP] a=rtpmap:111 opus/48000/2 a=rtcp-v:111 transport-cc a=fmtp:111 minpRme=10;useinbandfec=1 a=rtpmap:103 ISAC/16000 a=rtpmap:104 ISAC/32000 a=rtpmap:9 G722/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:106 CN/32000 a=rtpmap:105 CN/16000 a=rtpmap:13 CN/8000 a=rtpmap:126 telephone-event/8000
16. G.722 •  Sampling rate:16kHz, Bitrate:48, 56, 64kbps •  ADPCM (adapRve differenRal pulse code modulaRon) •  Compression algorithm –  Calculate diff of previous sample •  Should be small value (than sample) –  Use logarithm for difference •  Fine for small value •  Rough for big value 16 Linear PCM ADPCM ケータイWatch ケータイ用語の基礎知識より h[p://k-tai.watch.impress.co.jp/cda/arRcle/keyword/2936.html
17. Use lamejs for MP3 Compression •  To reduce audio size, use MP3 compression •  lamejs –  h[ps://github.com/zhuker/lamejs •  Not streaming, short length MP3 files •  Input: Array of 16bit integer –  Specify channels(Mono/Stereo), Sampling rate for input data •  Output: Array of 8bit integer –  Specify bitrate (kbps) for output –  Min limit：　bitrate >= 32kbps(22kHz input), bitrate >= about 36kbps (24kHz Input) 17
18. Sample from GitHub var channels = 1, sampleRate = 44100, kbps = 128; // mono, 44.1kHz, 128kbps var mp3encoder = new lamejs.Mp3Encoder(channels, sampleRate, kbps); var sampleBlockSize = 1152; // be[er to use 　576 * n var samples = new Int16Array(44100); // 1sec sample var mp3Data = []; for (var i = 0; i < samples.length; i += sampleBlockSize) { sampleChunk = samples.subarray(i, i + sampleBlockSize); var mp3buf = mp3encoder.encodeBuffer(sampleChunk); if (mp3buf.length > 0) { mp3Data.push(mp3buf); } } var mp3buf = mp3encoder.flush(); //finish wriRng mp3 if (mp3buf.length > 0) { mp3Data.push(new Int8Array(mp3buf)); } 18
19. Tips to use lamejs •  Reduce overhead of MP3 data –  Header + Tag = 132byte –  Longer block of audio to convert –  Max limit of block for WebAudio ScriptProcessor : 16384 sample for each event •  à 370 ms (44.1kHz), 340 ms (48kHz) 19 Header Compressed data MP3 tag 4byte 128byte 1500 - 2000 byte •  Reduce sample before compress：　44.1kHz→22.05kHz, 48kHz→24kHzに •  Consider Min limit for output bitreate •  bitrate >= 32kbps(22kHz input), bitrate >= about 36kbps (24kHz Input) •  Use WebWorker for compress … compression Rme is 50ms - 100ms / block
20. A|er reducing data → stable transfer •  Image … 56kbps •  Audio … 36 kbps 20 92kbps Total 1/40 1/8
21. Combine Browser MCU and iOS Gateway 21 Browser A Browser B Browser C MCU iOS Safari Image/2sec, MP3 Audio Image WebRTC Meetup Tokyo #11 Build WebRTC MCU on browser h[ps://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser h[ps://speakerdeck.com/mganeko/build-webrtc-mcu-on-browser Browser
22. DEMO •  PC Chrome ßà Browser MCU ßà iOS Safari 22
23. Inside of MCU Server：Video 23 RTCPeerConnecRon A MediaStream <video> element <canvas> element drawImage() <img> element drawImage() Worker WebSocket (socket.io) DataURL Blob JPEB toBlob() Blob JPEB from iOS to iOS setInterval() Keep drawing with requestAnimaRonFrame()
24. Inside of MCU Server：Audio 24 RTCPeerConnecRon A MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode ScriptProcessor mix Worker WebSocket (socket.io) MP3 [Uint8Array] [Uint16Array] [Flaoat32Array] lamejs to iOS create with AudioContext . createMediaStreamSource()
25. Inside of MCU Server: Recording 25 MediaStream (Video A+B+C+D) <canvas> element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM
26. Conclusion •  iOS Browser does not support WebRTC yet… So What? –  The GOAL is not WebRTC, but "CommunicaRon" with iOS user •  Try everything what you have now! –  Most advantage or "Web"RTC is flexibility to combine with other Web features, such as: –  Canvas, WebAudio, WebSocket •  It was interesRng to follow history of audio codecs 26
27. Thank you! 27