Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
Search
FromAtom
September 05, 2019
Technology
7
13k
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
iOSDC 2019
Day 0 17:50~
Track A
FromAtom
September 05, 2019
Tweet
Share
More Decks by FromAtom
See All by FromAtom
『SwiftUIならiOS, macOSの両方で動くエディタアプリが簡単に作れる』 と思ったら大間違いだよ! / pixiv App Night 2024-01-25
fromatom
1
330
君だけのGFMエディタを作ろう! / iOSDC Japan 2023
fromatom
0
1.8k
サポートiOSバージョンを定期的にあげる仕組みづくり / iOSDC Japan 2022
fromatom
1
3.2k
僕たちが 『Appのプライバシーに関する質問への回答』 そして『ATT』に対応するまでの物語 / iOSDC Japan 2021
fromatom
1
3.4k
やってみよう! iOSDCデザインスポンサー! / iOSDC Japan 2021 LT
fromatom
2
1.3k
デバッグメニューのメンテナンスが大変だったので、専用アプリを作りました。 / iOSDC Japan 2020
fromatom
9
5.9k
🎉 esa 生誕5周年記念パーティー(\( ⁰⊖⁰)/) 🎉 / esa the 5th anniversary
fromatom
1
4.1k
複数のライブ映像を同期再生するのが大変だったので知見をお伝えします / iOSDC Japan 2018
fromatom
8
7.3k
【いまさら聞けない】SwiftのOptionalってなに?
fromatom
0
3.4k
Other Decks in Technology
See All in Technology
OSTという文化を組織に根付かせてみた
sansantech
PRO
2
380
『GRANBLUE FANTASY Relink』キャラクターの魅力を支えるリグ・シミュレーション制作事例
cygames
0
120
『GRANBLUE FANTASY: Relink』最高の「没入感」を実現するカットシーン制作手法とそれを支える技術
cygames
1
140
PDF Viewer作成の今までとこれから
hunachi
0
470
自作Cコンパイラ 8時間の奮闘
soukouki
0
840
なにもしてないのにNew Relicのデータ転送量が増えていたときに確認したこと
tk3fftk
2
230
チームビルディングは"感性"で向き合おう / Team Building with Awareness
kohzas
0
250
突撃! 隣のAmazon Bedrockユーザー 〜YouはどうしてAWSで?〜
minorun365
PRO
3
390
watsonx.ai Dojo 環境準備について
oniak3ibm
PRO
0
300
開発者の定量・定性データを組み合わせて開発者体験を把握するための取り組み
ham0215
1
130
eBPFのこれまでとこれから
yutarohayakawa
10
3.2k
再考 アクターモデル/ reconsider actor model
ytake
0
360
Featured
See All Featured
Music & Morning Musume
bryan
46
6k
Git: the NoSQL Database
bkeepers
PRO
425
64k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.6k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
Ruby is Unlike a Banana
tanoku
96
11k
Designing for Performance
lara
604
68k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2k
Bash Introduction
62gerente
608
210k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
24
610
Clear Off the Table
cherdarchuk
91
320k
Transcript
εΫϦʔϯ৴ػೳͷ ࣮͕େมͩͬͨͷͰ ݟΛ͓͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A
pixiv.inc FromAtom
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /
pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱ εϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ • ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ • ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔ εΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ • ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ৴
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ ‣ ৴Ͱ͖ͳ͍ • iOS 10+
‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡Έ ReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
None
None
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ ΧϝϥPush௨ͱಉ͡ • iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨ͠
ͪΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖Δ iOS 9+
ը։࢝
ըऴྃ
iOS 10+ ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁ iOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮ • Broadcast Upload ExtensionΛ࣮͢Δ • ৄ͘͜͠ͷޙ iOS11+
ͷͱҰॹʹ͠·͢
iOS 11+ Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ • ϚΠΫԻΞϓϦԻऔಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ • iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ ‣ ௨ऩ͞ΕΔͷͰҙ ίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+ ͍͜͠ͷͰҰ୴ཧ
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast
Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ ReplayKitͱεΫϦʔϯ৴
iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌ྃ
iOS 11+ Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ • Լهͷ2ͭʹ֘ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ ‣ iOS11+
- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖Δ Broadcast Upload Extension
࣮खॱ
࣮खॱ
࣮खॱ
࣮खॱ
iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍ SetupUI Extensionͱ
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • TARGETS͔ΒSetupUI ExtensionΛબ • GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ • InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢ •
"NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ • ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮ • ৴༻ͷॲཧBroadcast Upload
ExtensionଆͰ࣮ • SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏ ‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+ Ұ୴ཧ
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer ੈք
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠ ੈք
iOS 11+ Broadcast Upload Extension
• ॳظঢ়ଶͰϑΝΠϧ2ͭ • "SampleHandler.swift" ʹ࣮Λॻ͍͍ͯ͘ ࣮खॱ
iOS 11+ SampleHandler.swift ʹ ࣮Λॻ͍͍ͯ͘
None
• ৴։࢝࣌ʹݺΕΔ • ϩάΠϯॲཧ • ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
None
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճWebRTCͷ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBufferΛૹΔ
࣮खॱ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ • ͑Δͱແ༻ͰKill͞ΕΔ •
Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:) SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+ Ի
• ϚΠΫԻͱΞϓϦԻ • ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖Δ Ի
iOS 11+ Sora iOS SDKlibwebrtcͷ্༷ Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳ Sora iOS SDK with libwebrtc
To Be Continued
iOS 11+ ͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ2छྨ ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏ - ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δ ը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ
• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δ ը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ • ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏ • ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏
·ͱΊ