WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

SkyWay UG Tokyo #2
2018-02-05

1. Transcoding video streams
 from WebRTC MICHII Shunsuke pixiv Inc. SkyWay UG Tokyo #2
 2018-02-05
2. MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦௕
 ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦ໾һ ImageFluxͰ͸GoΛॻ͍ͨΓΠϯϑϥपΓΛ΍ͬͨΓͱ͔͍Ζ͍Ζ
3. ࠓ೔͸ͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜ͸ͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
4. ࠓ೔͸ͳ͞ͳ͍͜ͱ • WebRTCͱ͸ͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕΍ͬͯ͘ΕΔ͜ͱ
 ઀ଓཱ֬ɺηογϣϯͷ؅ཧɺSRTPͷ҉߸Խղআ…… • SkyWayͰͲ͏΍ͬͯ΍Δͷ͔……ʢRTPΛऔΓग़ͤΕ͹େৎ෉ʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
5. ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS഑৴͢ΔαʔϏεΛ։ൃதʂ
 https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํ͸ੋඇ͓੠͕͚͍ͩ͘͞ʂ
 ΤϯδχΞ΋ืू͍ͯ͠·͢ʂ
6. ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack
 stream TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
7. H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Msgpack
 stream TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
8. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ
9. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ
10. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯ৘ใ ಈըετϦʔϜ
11. WebRTCͰૹΒΕͯ͘ΔಈըΛ
 τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
12. WebRTCͰૹΒΕͯ͘ΔಈըΛ
 τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ 1, 2͸WebRTC SFU Sora͕΍ͬͯ͘Ε·͢
13. RTP (Realtime Transfer Protocol) RFC3550
 RTP: A Transport Protocol for Real-Time Applications • Ի੠ɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮૷͞Ε͍ͯΔ • TCPΛ࢖͍ͬͯͳ͍ͷͰɺॱং੍ޚ΍᫔᫓੍ޚΛRTPϨΠϠͰ࣮૷Ͱ͖Δ
 →ݴ͍׵͑Ε͹ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋Δ • SRTP͸ʢͬ͘͟Γݴ͑͹ʣRTPΛ҉߸Խͨ͠΋ͷ
14. ࣮ମ͸UDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸Խํࣜ͸ϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload
 ….
15. ϖΠϩʔυͷ൑ผํ๏ • ϖΠϩʔυͷϑΥʔϚοτ͸payload typeͰ൑ผ͢Δ • payload type͸7bit͔͠ͳ͘ɺ͙͢ʹ଍Γͳ͘ͳͬͨͷͰɺݱࡏͰ͸
 96-127ΛಈతʹׂΓ౰ͯͯ࢖͏ʢRFC3551ʣ • payload typeͱϑΥʔϚοτͷϚοϐϯά͸SDPͷattributeͱͯ͠
 ΍ΓͱΓ͞ΕΔ • طʹొ࿥͞Ε͍ͯΔͷ͸IANAͰ؅ཧ͞Ε͍ͯΔ
 https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
16. ॱংอূ • RTP͸UDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹ͸γʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛ࢖ͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕
 ͋·Γ଴ͭͱ஗ԆͷݪҼʹͳΔͷͰέʔεόΠέʔε • ͍·ͷ࣮૷Ͱ͸ύέοτ͕ඈΜͩ৔߹͸100ݸ͘Β͍଴ͭΑ͏ʹ͍ͯ͠Δ
17. ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜ͸ҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184
 RTP Payload Format for H.264 Video RFC7587
 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01
 RTP Payload Format for VP9 Video
18. RTP Payload Format for H.264 Video • H.264 NALU (NAL Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
19. H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰ͸NALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ • NAL͸ಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰ఻ૹ͠΍͍͢Α͏ʹந৅Խ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺ͹ΕΔͻͱ͔ͨ ·Γʹ੔ܗ͍ͯ͠Δ
20. H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμ͸H.264/AVC Bytestream Format (Annex B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺ͹ΕΔ • جຊతʹ͸NALUΛ00 00 00 01Ͱܨ͛Ε͹Α͍
 ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͹͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
21. ෼ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζ͸Ұൠతʹ͸1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ෼ׂͯ͠ෳ਺ͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺ෼ׂύέοτ͸NAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
22. ෼ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰ͸STAP-AͱFU-A͕࢖ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢Ε͹Α͍ • STAP-A: ෳ਺ͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳ਺ͷNALUΛ෼ׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨ΋ͷ
 جຊతʹ͸start bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
23. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ 4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
24. SDP (Session Description Protocol) RFC4566
 SDP: Session Description Protocol • ηογϣϯ৘ใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͸͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ޿͘࢖ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
25. ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
26. ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
27. Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>: ϝσΟΞͷछྨʢaudio, video,…)
 <port>: ૹ৴ݩϙʔτ
 <proto>: ϓϩτίϧ
 <fmt>: ϑΥʔϚοτ৘ใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ
 ͋·Γ࢖͑Δ৘ใ͸ͳ͍
28. Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ
 <value>: ͦͷ஋ • SDPʹ஋Λ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ • WebRTCͰ֦ு͞Ε͍ͯΔ஋͸͍͍ͩͨa=ʹೖ͍ͬͯΔ
29. SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2
 a=rtpmap:120 H264/90000 • payload typeͷmapping͸rtpmapͰࢦఆ͞ΕΔ • ্ͷྫͩͱɺH.264ͷpayload type͸120ɺOpus͸109 • 2൪໨ͷ஋͸ϑΥʔϚοτ/प೾਺/νϟϯωϧ਺
 H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕΍ͬͯ͘Δ͜ͱ͕Θ͔Δ
30. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ 4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
31. ·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹ͸ɺ
 RTPύέοτΛσίʔυͰ͖Ε͹Α͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹ͸WebRTC SFU Sora͕ศར • ImageFlux Live։ൃதͰ͢ʂ