Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WebRTC - ultra low latency delivery of sub second

dd.telecaster
September 11, 2020

WebRTC - ultra low latency delivery of sub second

Introduction
Expectations for Sub-Second
Chunked Streaming
Zoom vs WebRTC
Browser Support

WebRTC Technology
Why realtime?
Framework
Connection Method
Various Protocol Used

Performance of WebRTC
How low latency
Quality - Encoding Challenges
TEST - Quality, Resolution and Bit-rate
Stability
Smartphone viewing
Number of concurrent viewers
Interactivity
Troubleshoot
Latest Case

TIPS
Delivery using CDN
Encoder
Verification

dd.telecaster

September 11, 2020
Tweet

Other Decks in Technology

Transcript

  1. 1 WebRTC - ultra low latency delivery of sub second

    Solutions Engineer – Daisuke Doi Limelight Networks, Inc. 24 September 2020
  2. 2 Daisuke Doi (Limelight Networks, Inc.) Engaged in consulting and

    project management as an engineer who proposes solution construction for over 10 years in the broadcasting, post-production and entertainment industries at domestic / global companies. Video editing environment and media asset management, industrial VR, Engaged in introducing video solutions for entertainment facilities, etc. both in Japan and overseas. Live interactive streaming is the future of video expression Believing in the development of communication, I joined Limelight Networks Japan Co., Ltd. in early 2019. Hobbies are video editing of my band. We are considering implementing live streaming for business verification. Self-Introduction solutions engineer who translates customer business requirements into concrete solutions.
  3. 3 Introduction • Expectations for Sub-Second • Chunked Streaming •

    Zoom vs WebRTC • Browser Support WebRTC Technology • Why realtime? • Framework • Connection Method • Various Protocol Used Performance of WebRTC • How low latency • Quality - Encoding Challenges • TEST - Quality, Resolution and Bit-rate • Stability • Smartphone viewing • Number of concurrent viewers • Interactivity • Troubleshoot • Latest Case TIPS • Delivery using CDN • Encoder • Verification Agenda What is a sub-second real-time video delivery service using WebRTC?
  4. 4 Expectations for Sub-Second Expectations for a live distribution business

    created by ultra-low latency or interactive communication Live Sports & Event • No gap against match progress • Watching experience from multiple viewpoints News • Contribution to promptness • Election bulletins and disaster reports • News for regions that cannot be covered by broadcasting Interactive Streaming • Live events such as games and e-sports • Comment online, chat, vote, poll, and more • Second screen • Live experience from multiple viewpoints with no gap between real and streaming Live & Event Venue
  5. 5 Various live streaming solutions 1 Sec 60 Sec 10

    Sec 5 Sec Normal HTTP Live Latency Ultra-Low-Latency 30 Sec Traditional HTTP Live Streaming Low Latency CMAF WebRTC Ultra-Low Latency Delivery Small Chunk Streaming Latency of Cable TV Low Latency Chunk streaming is now mainstream-live and real-time delivery are different
  6. 6 zoom • Specialized in Video conference • 100 -

    1000 viewer (until 10,000 by webinar mode) • The delay increases as the number of viewers increases • Video receive is difficult in the 1.2 Mbps or lower band zoom and WebRTC • Suitable for web conferences • Realtime • Challenges for large-scale distribution • Already stable technology WebRTC • Distribution for viewers and customization to individual UI • Constant ultra-low latency • Support ABR • No plugin required • Can be delivered only by browser and camera zoom vs WebRTC Video conference system zoom and WebRTC
  7. 7 WebRTC Conference Service Encoder CDN RTMP Upload (H.264) WebRTC

    Viewer Service Mainly used and customized for two services WebRTC Workflow Web Browser Web Browser Web Browser Web Browser H.264 or VP8/Opus WebRTC (H.264/Opus) Web Browser
  8. 8 8 Required Codec: VP8 and H.264 (IETF - RFC7742)

    Opus (Audio) 4K support?: VP9 -> AV1 H.265 has a lots of patents…. Safari: • Supports VP8 last year (2019) • VP9 is not supported MS Edge supports by Chromium version IE Doesn’t support Browser Support Supported by major browsers, no plugin or app installation required
  9. 10 Why realtime? Communication method and transmission method 3sec 6sec

    9sec Chunk streaming creates a delimiter with the set number of seconds and then sends it out sequentially. UDP Communication WebRTC keeps sending anyway HLS / DASH WebRTC
  10. 11 Connection • UDP • ICE • STUN • TURN

    • SDP • Signaling Original Framework • Media Channel • Data Channel • Congestion Control • Retransmission control Encrypted communication • DTLS • SRTP • SRTCP • SCTP Framework of WebRTC Ultra-low latency and communication basics
  11. 12 STUN • Required for NAT TURN • Relay Server

    (not required) WebRTC connection establishment method STUN 、TURN、and ICE SDP SDP ICE • Confirmation of routes and connection candidates Connectivity Check  takes 5 second
  12. 13 WebRTC Protocols: a collection of different protocols Various protocols

    are used to enable communication on UDP + security Complicated....
  13. 14 WebRTC Protocols:DTLS - SRTP and SRTCP SRTCP • Used

    for signaling • Control and track individual packet delivery • Multiplexed on the same port as SRTP Data encryption and signaling on UDP + security communication SRTP • Encrypt video and audio data body • Exchange keys for SRTP using DTLS • He doesn't care if packet loss happens.. DTLS • Ensures security similar to that offered by TLS DTLS SRTP SRTCP UDP
  14. 15 WebRTC data channel runs over SCTP • Data channel

    is used when calling javascript API required for WebRTC. • Tunnel over DTLS • Provides flow and congestion control mechanisms not found in UDP, but not as good as TCP Table: bloggeek.me WebRTC Protcols:SCTP Hybrid protocol of UDP and TCP
  15. 16 WebRTC Protocol expected in the future:QUIC Research and development

    by Google as an alternative to the current protocol QUIC • Positioned between UDP and TCP, UDP provides similar functionality to TCP • Better congestion control • The purpose of improving Web performance, not just WebRTC • Responsible for the transport layer of HTTP / 3 X
  16. 17 Current status of WebRTC Performance? What is the performance

    of WebRTC? -Verification using Limelight Networks distribution equipment
  17. 18 •How low latency •Quality - Encoding Challenges •TEST -

    Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case Current status of WebRTC What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment Test Equipments Encoder : OBS Studio Hardware : DELL Latitude 7390 (2019) Windows 10 64bit, 16GB Memory, Core i5 7300U 2.6Ghz Delivery System : Limelight Networks Camera : Logicool webcam C920
  18. 19 Current status of WebRTC 250ms-300ms always ultra low latency

    •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  19. 20 Current status of WebRTC Video clarity • High Bit-Rate

    • The right resolution for devices • Camera performance and settings Unsuitable video (Camera and encoder challenges) • Video of the entire camera moving • An image of a person moving quickly at a short distance • Live video with lots of lighting •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  20. 21 Current status of WebRTC FHD / HD / SD,

    3Mbps - 9Mbps 15fps - 60fps, 64 - 128Kbps audio 1. Demand for desktop distribution is increasing 2. There is no way to avoid dropping video frames due to WebRTC spec 3. Since there is almost no lack of voice, there is no discomfort 4. 1080p was heavy processing. Encoder limit at 4.5Mbps 60fps 5. Encoder limit at 720p 9Mbps 60fps 6. Quality that can withstand most services with 3 Mbps. Distribute Media playback from desktop using WebRTC TEST •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  21. 22 Current status of WebRTC The video is affected by

    the network condition and loss occurs, but it is unavoidable due to the specifications. However, the sound does not stop, so there is little discomfort There is no discomfort in viewing because the audio does not almost stop due to FEC implemented in Opus. Audio Video •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  22. 23 Current status of WebRTC 4G / iPhone 8 &

    iPhone 11s HD (720p) 1Mbps - 6Mbps 30fps or 60fps Smartphone viewing looks more beautiful than a PC. There is little discomfort in dropping frames. •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  23. 24 Current status of WebRTC Challenge in expanding the number

    of concurrent connections •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  24. 25 Current status of WebRTC Chat with Websocket Betting function

    Voice communication Online Drone Racing (e-Sports) Deliver video from drone to supporters (passengers) Operate the actual drone in several regions, Distribution of e-Sports with game elements added in AR / VR Passengers help pilots •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  25. 26 Current status of WebRTC Most of the troubles are

    caused by the sending / receiving network or encoder trouble. Network for ingest Encoder backup recommended. •How low latency •Quality - Encoding Challenges •TEST - Quality, Resolution and Bit-rate •Stability •Smartphone viewing •Number of concurrent viewers •Interactivity •Troubleshoot •Latest Case What is the performance of WebRTC? -Verification using Limelight Networks distribution equipment
  26. 27 Use Case: Post Production Realtime streaming creates a high

    quality remote editing system After the COVID-19, many business and service got to adopt an online/remote solution. One of the solution which replace traditional Post Production Studio service to all online. Editors, Directors and Sponsors can join in editing place from all remote site. Challenge Solution Benefit • To ensure interactivity, a real-time delivery platform without delay • Encoder settings • Player implementation • Secure distribution • Subsecond streaming • 6 - 7 sec latency HLS • Ondemand • Director can direct a video editing with qualified video quality. Realtime streaming provides interactivity for remote editing on a post production work. Editing Studio Directors, Producers and Sponsors Solution: for a remote Editing
  27. 28 WebRTC WebRTC WebRTC Edge Global Edge Network Video Source

    Encoder RTMP Up load WebRTC Edge WebRTC Edge Streaming Distribution Streaming Distribution Streaming Distribution WebRTC Primary Backup ex):WebRTC distribution Workflow using CDN Overview of real-time streaming and CDN distribution
  28. 29 Verified encoder • AWS - Elemental Live • Haivison

    - Kurabyte ◦ Commercial encoder • OBS Project - OBS Studio ◦ Free Software (Single Ingest) • Telestream - Wirecast ◦ Paid Software (Multi Ingest ready) • CEREVO - LiveShell.x ◦ Small Box (Multi Ingest ready) Encoder: RTMP From free to appliance servers, depending on the application Encoder settings • RTMP • SD - FHD • 15fps - 30fps • Video : 500kbps 〜 3Mbps • Audio : 64kbps 〜 128kbps • CBR : 1 (WebRTC recommended) • Keyframe : 1 (WebRTC recommended) • Profile : Baseline (WebRTC recommended)
  29. 30 Encoder Configuration OBS Studio, the most used encoder for

    personals and businesses 1. Delivery URL & Stream Key 2. Output Settings 3. Video Settings 4. Audio Settings ① ② ③ ④
  30. 31 WebRTC Viewing environment verification Prepare a delivery environment, delivery

    contents, and viewing environment that match the actual service case Trade-off between low latency and continuous playback The point of service provision is the use case and the provision of benefits due to ultra-low latency