Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
Search
FromAtom
September 05, 2019
Technology
7
13k
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
iOSDC 2019
Day 0 17:50~
Track A
FromAtom
September 05, 2019
Tweet
Share
More Decks by FromAtom
See All by FromAtom
「UIは英語なのにアプリ内リンクは日本語だ!」を防ぐコツ / pixiv App Night 2024-10-24
fromatom
0
75
『SwiftUIならiOS, macOSの両方で動くエディタアプリが簡単に作れる』 と思ったら大間違いだよ! / pixiv App Night 2024-01-25
fromatom
1
480
君だけのGFMエディタを作ろう! / iOSDC Japan 2023
fromatom
0
2k
サポートiOSバージョンを定期的にあげる仕組みづくり / iOSDC Japan 2022
fromatom
1
3.6k
僕たちが 『Appのプライバシーに関する質問への回答』 そして『ATT』に対応するまでの物語 / iOSDC Japan 2021
fromatom
1
3.8k
やってみよう! iOSDCデザインスポンサー! / iOSDC Japan 2021 LT
fromatom
2
1.3k
デバッグメニューのメンテナンスが大変だったので、専用アプリを作りました。 / iOSDC Japan 2020
fromatom
9
6.3k
🎉 esa 生誕5周年記念パーティー(\( ⁰⊖⁰)/) 🎉 / esa the 5th anniversary
fromatom
1
4.3k
複数のライブ映像を同期再生するのが大変だったので知見をお伝えします / iOSDC Japan 2018
fromatom
8
7.8k
Other Decks in Technology
See All in Technology
Devinで模索する AIファースト開発〜ゼロベースから始めるDevOpsの進化〜
potix2
PRO
8
3.5k
ビジネスとデザインとエンジニアリングを繋ぐために 一人のエンジニアは何ができるか / What can a single engineer do to connect business, design, and engineering?
kaminashi
2
350
OpenLane-V2ベンチマークと代表的な手法
kzykmyzw
0
110
テストって楽しい!開発を加速させるテストの魅力 / Testing is Fun! The Fascinating of Testing to Accelerate Development
aiandrox
0
100
Mastraに入門してみた ~AWS CDKを添えて~
tsukuboshi
0
320
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
250
От ручной разметки к LLM: как мы создавали облако тегов в Lamoda. Анастасия Ангелова, Data Scientist, Lamoda Tech
lamodatech
0
790
品質文化を支える小さいクロスファンクショナルなチーム / Cross-functional teams fostering quality culture
toma_sm
0
150
読んで学ぶ Amplify Gen2 / Amplify と CDK の関係を紐解く #jawsug_tokyo
tacck
PRO
1
250
生成AIのユースケースをとにかく集めてまるっと学ぶ!/ all about generative ai usecases
gakumura
2
230
Linuxのパッケージ管理とアップデート基礎知識
go_nishimoto
0
470
Writing Ruby Scripts with TypeProf
mame
0
350
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
38
1.7k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Faster Mobile Websites
deanohume
306
31k
Agile that works and the tools we love
rasmusluckow
328
21k
It's Worth the Effort
3n
184
28k
Product Roadmaps are Hard
iamctodd
PRO
52
11k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
The Cost Of JavaScript in 2023
addyosmani
49
7.7k
The Language of Interfaces
destraynor
157
25k
RailsConf 2023
tenderlove
30
1.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.2k
Building Flexible Design Systems
yeseniaperezcruz
329
38k
Transcript
εΫϦʔϯ৴ػೳͷ ࣮͕େมͩͬͨͷͰ ݟΛ͓͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A
pixiv.inc FromAtom
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /
pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱ εϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ • ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ • ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔ εΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ • ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ৴
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ ‣ ৴Ͱ͖ͳ͍ • iOS 10+
‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡Έ ReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
None
None
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ ΧϝϥPush௨ͱಉ͡ • iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨ͠
ͪΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖Δ iOS 9+
ը։࢝
ըऴྃ
iOS 10+ ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁ iOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮ • Broadcast Upload ExtensionΛ࣮͢Δ • ৄ͘͜͠ͷޙ iOS11+
ͷͱҰॹʹ͠·͢
iOS 11+ Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ • ϚΠΫԻΞϓϦԻऔಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ • iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ ‣ ௨ऩ͞ΕΔͷͰҙ ίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+ ͍͜͠ͷͰҰ୴ཧ
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast
Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ ReplayKitͱεΫϦʔϯ৴
iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌ྃ
iOS 11+ Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ • Լهͷ2ͭʹ֘ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ ‣ iOS11+
- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖Δ Broadcast Upload Extension
࣮खॱ
࣮खॱ
࣮खॱ
࣮खॱ
iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍ SetupUI Extensionͱ
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • TARGETS͔ΒSetupUI ExtensionΛબ • GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ • InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢ •
"NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ • ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮ • ৴༻ͷॲཧBroadcast Upload
ExtensionଆͰ࣮ • SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏ ‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+ Ұ୴ཧ
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer ੈք
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠ ੈք
iOS 11+ Broadcast Upload Extension
• ॳظঢ়ଶͰϑΝΠϧ2ͭ • "SampleHandler.swift" ʹ࣮Λॻ͍͍ͯ͘ ࣮खॱ
iOS 11+ SampleHandler.swift ʹ ࣮Λॻ͍͍ͯ͘
None
• ৴։࢝࣌ʹݺΕΔ • ϩάΠϯॲཧ • ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
None
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճWebRTCͷ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBufferΛૹΔ
࣮खॱ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ • ͑Δͱແ༻ͰKill͞ΕΔ •
Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:) SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+ Ի
• ϚΠΫԻͱΞϓϦԻ • ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖Δ Ի
iOS 11+ Sora iOS SDKlibwebrtcͷ্༷ Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳ Sora iOS SDK with libwebrtc
To Be Continued
iOS 11+ ͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ2छྨ ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏ - ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δ ը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ
• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δ ը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ • ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏ • ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏
·ͱΊ