Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
WebRTC動画をトランスコードする / Transcoding video streams ...
Search
Harukasan
PRO
February 05, 2018
Technology
5
1.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
SkyWay UG Tokyo #2
2018-02-05
Harukasan
PRO
February 05, 2018
Tweet
Share
More Decks by Harukasan
See All by Harukasan
Successor to PicoRabbit: Ruby Programming Envorinment / RubyKaigi 2025 follow up
harukasan
PRO
1
190
Write your own mrbgem, Create your own device
harukasan
PRO
1
120
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
500
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
530
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
840
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.7k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.7k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.9k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
3k
Other Decks in Technology
See All in Technology
KMP の Swift export
kokihirokawa
0
330
Goにおける 生成AIによるコード生成の ベンチマーク評価入門
daisuketakeda
2
100
AIAgentの限界を超え、 現場を動かすWorkflowAgentの設計と実践
miyatakoji
0
130
生成AIとM5Stack / M5 Japan Tour 2025 Autumn 東京
you
PRO
0
210
Modern_Data_Stack最新動向クイズ_買収_AI_激動の2025年_.pdf
sagara
0
200
AI Agentと MCP Serverで実現する iOSアプリの 自動テスト作成の効率化
spiderplus_cb
0
490
Large Vision Language Modelを用いた 文書画像データ化作業自動化の検証、運用 / shibuya_AI
sansan_randd
0
100
E2Eテスト設計_自動化のリアル___Playwrightでの実践とMCPの試み__AIによるテスト観点作成_.pdf
findy_eventslides
0
110
定期的な価値提供だけじゃない、スクラムが導くチームの共創化 / 20251004 Naoki Takahashi
shift_evolve
PRO
3
300
バイブコーディングと継続的デプロイメント
nwiizo
2
420
PLaMoの事後学習を支える技術 / PFN LLMセミナー
pfn
PRO
9
3.8k
How to achieve interoperable digital identity across Asian countries
fujie
0
110
Featured
See All Featured
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
31
9.7k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
9
580
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.7k
Unsuck your backbone
ammeep
671
58k
How GitHub (no longer) Works
holman
315
140k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
Code Review Best Practice
trishagee
72
19k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Building Better People: How to give real-time feedback that sticks.
wjessup
368
20k
Producing Creativity
orderedlist
PRO
347
40k
Docker and Python
trallard
46
3.6k
Transcript
Transcoding video streams from WebRTC MICHII Shunsuke pixiv Inc. SkyWay
UG Tokyo #2 2018-02-05
MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһ ImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ • WebRTCͱͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……
• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ
ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack stream
TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Msgpack
stream TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯใ ಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol) RFC3550 RTP: A Transport Protocol for
Real-Time Applications • ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮͞Ε͍ͯΔ • TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ • SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload ….
ϖΠϩʔυͷผํ๏ • ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ • payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ • payload
typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ • طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ • RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε
• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184 RTP Payload Format for H.264 Video
RFC7587 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video • H.264 NALU (NAL
Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •
NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex
B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ • جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit
Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢ΕΑ͍ • STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ
جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol) RFC4566 SDP: Session Description Protocol •
ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ͘ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:
ϝσΟΞͷछྨʢaudio, video,…) <port>: ૹ৴ݩϙʔτ <proto>: ϓϩτίϧ <fmt>: ϑΥʔϚοτใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ <value>: ͦͷ • SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •
WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000 • payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ
• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109 • 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར • ImageFlux
Live։ൃதͰ͢ʂ