PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

1J1ΛԠ༻ͨ͠഑৴ίϝϯτόʔػೳͷ ։ൃൿ࿩ͱٕज़ͷৄղ גࣜձࣾϛϥςΟϒɹઍ٢ྑ੒ل @OBSV@KQO OBSVKQO aϐΫνϟɾΠϯɾϐΫνϟ

͜ͷτʔΫͰ͓࿩͢͠Δ͜ͱ ɾ1J1جຊͷৼΔ෣͍ͱ࣮૷ ɾϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ ɾ3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ ɾʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ɾඳըίετͱͷઓ͍

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ ৼΔ෣͍

ΞϓϦ֎Ͱͷಈըίϯςϯπͷ࠶ੜҠಈ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

ը໘ͷ୺ʹӅ͢ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

ϐϯνɾΠϯϐϯνɾΞ΢τͰαΠζͷมߋ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

όοΫάϥ΢ϯυҠߦͰࣗಈදࣔ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

ઃఆˠϐΫνϟΠϯϐΫνϟͰʮࣗಈతʹ։࢝ʯͷมߋ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ "71JDUVSF*O1JDUVSF$POUSPMMFS

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS "71JDUVSF*O1JDUVSF$POUSPMMFS ɾ1J1ͷ࣮૷ͷͨΊʹ࢖͏ ɾ"71JDUVSF*O1JDUVSF$POUSPMMFS͸J04͔Βଘࡏ͍ͯ͠Δ ɹɾJ04࣌఺Ͱ͸Ұ෦ͷJ1BEͷΈରԠ ɹɾJ04͔ΒJ1IPOFͰ΋1J1͕࢖͑ΔΑ͏ʹ ɾ$POUFOU4PVSDF͔Βੜ੒͢Δ˞
ɹɾinit(contentSource: AVPictureInPictureController.ContentSource) ˞ଞͷΠχγϟϥΠβ΋͋Δ͕DPOWFOJFODFͳͷͰ͜͜Ͱ͸ແࢹ͢Δ

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS "71JDUVSF*O1JDUVSF$POUSPMMFSͷ֓ཁ ɾ$POUFOU4PVSDFͷछྨ ɹɾछྨͷϨΠϠʔ͔Βੜ੒Ͱ͖Δ˞ ɹɾ"71MBZFS-BZFS͔"74BNQMF#VGGFS%JTQMBZ-BZFS ˞"71JDUVSF*O1JDUVSF7JEFP$BMM7JFX$POUSPMMFS͔Β΋ੜ੒Ͱ͖Δ͕ಛघͳػೳͰ"QQMFͷڐՄ͕ͳ͍ͱ։ൃͰ͖ͳ͍ͷͰແࢹ init( sampleBufferDisplayLayer:
AVSampleBufferDisplayLayer, playbackDelegate: AVPictureInPictureSampleBufferPlaybackDelegate ) init(playerLayer: AVPlayerLayer) J04 J04

"71MBZFS-BZFSΛ࢖࣮ͬͨ૷

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS ४උ $BQBCJMJUJFT#BDLHSPVOE.PEFT1JDUVSFJO1JDUVSFΛ༗ޮʹ͢Δ "7"VEJP4FTTJPOͷΧςΰϦΛQMBZCBDL͔QMBZ"OE3FDPSEʹ͢Δ USZ"7"VEJP4FTTJPOTIBSFE*OTUBODF TFU$BUFHPSZ QMBZCBDL "7'PVOEBUJPO1J11MBZFS1JDUVSFJO1JDUVSF1MBZCBDLXJUI"7,JU

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS ߏ੒ "71MBZFS-BZFS 6*7JFX$POUSPMMFS 6*#VUUPO "71JDUVSF*O1JDUVSF$POUSPMMFS

"71JDUVSF*O1JDUVSF$POUSPMMFSͷॳظԽ // AVPictureInPictureController の生成 let controller = AVPictureInPictureController(playerLayer: playerLayer) //
バックグラウンド移行時に自動でピクチャ・イン・ピクチャを起動するか controller.canStartPictureInPictureAutomaticallyFromInline = true QMBZFS-BZFS 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS

Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ /// ピクチャ・イン・ピクチャの開始 @IBAction private func didTapStartButton(_ sender: UIButton) {
pictureInPictureController.startPictureInPicture() } /// ピクチャ・イン・ピクチャの停止 @IBAction private func didTapStopButton(_ sender: UIButton) { pictureInPictureController.stopPictureInPicture() } 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS

OBSVKQOQJQFYFSDJTF FYBNQMFTQJQCBTJD "71MBZFS-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷

"74BNQMF#VGGFS%JTQMBZ-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷ FYBNQMFTQJQDVTUPN FYBNQMFTQJQDBNFSB

ϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ϛϥςΟϒͰͷ഑৴தͷίϝϯτͱϓογϡ௨஌ ɾϛϥςΟϒ͸഑৴ΞϓϦͳͷͰࢹௌऀ͕͍Δ ɹɾࢹௌऀ͸ςΩετͰίϝϯτΛ͢Δ ɹɾ഑৴ऀ͸جຊతʹϛϥςΟϒΞϓϦΛ։͍͍ͯͳ͍ ɹɹɾήʔϜͳͲଞͷΞϓϦΛ։͍ͯ഑৴͍ͯ͠Δ ɹɾ഑৴ऀʹࢹௌऀ͔ΒͷίϝϯτΛಧ͚Δඞཁ͕͋Δ ɾJ04Ͱ͸ैདྷ͔Βϓογϡ௨஌ͰϢʔβʔʹίϝϯτΛಧ͚͍ͯΔ

ϓογϡ௨஌ʹΑΔ഑৴ͷίϝϯτදࣔʢ഑৴ίϝϯτόʔະ࢖༻ʣ

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ഑৴ίϝϯτόʔ ɾ഑৴ίϝϯτόʔ͸ϓογϡ௨஌ʹมΘͬͯίϝϯτΛಧ͚Δ ɹɾ1J1Λ׆༻ͯ͠ಠࣗͷ6*ͰίϝϯτΛಧ͚Δ ɹɾৼΔ෣͍͸ී௨ͷ1J1ͱಉ͡ ɹɾ1J1͕දࣔ͞Ε͍ͯΔؒɺϓογϡ௨஌͸දࣔ͞Εͳ͍

഑৴ίϝϯτόʔʹΑΔ഑৴ͷίϝϯτදࣔ

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ίϝϯτ΍Ϊϑτͷ಺༰ ࠷େߦ ഑৴ͷܦա࣌ؒ ࢹௌऀ਺ ଃΒΕͨΪϑτͷ਺ ഑৴Ի੠ͷPOPGG ഑৴ऀͷΤϞϞ Ξόλʔ ഑৴ऀͷΤϞϞͷഎܠը૾
഑৴ίϝϯτόʔͷཁૉ

3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ ໌ࣔతʹ3%ͱͯ͠։ൃΛਐΊͨ ɾੈͷதʹ·ͩͳ͍࣮૷͕ͩͬͨԿਓ͔͸ՄೳੑΛײ͍ͯͨ͡ ɾԿ͕Ͳ͜·Ͱ࣮૷Ͱ͖Δ͔෼͔Βͳ͔ͬͨ ɾΤϯδχΞ͕ٕज़ͷݕূΛ͠ͳ͕Βओମతʹ։ൃΛਐΊ͍ͯ ͘ඞཁ͕͋ͬͨ ɾࣦഊͯ͠΋ؾʹ͠ͳ͍

3%։ൃ͔Βੜ·Εͨʮ഑৴ίϝϯτόʔʯ ഑৴ίϝϯτόʔػೳͷϦϦʔε·Ͱ ɹJ04ͷσϕϩούʔϕʔλ൛ϦϦʔε ɹࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ ɹJ04ϦϦʔε ɹ3%։࢝ ɹJ04ϦϦʔε ɹ഑৴ίϝϯτόʔػೳϦϦʔε

ࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ ɾ"71JDUVSF*O1JDUVSF$POUSPMMFSͱ"74BNQMF#VGGFS%JTQMBZ-BZFS Λ૊Έ߹ΘͤͨϓϩτλΠϓ͕ͻͬͦΓͱ࡞ΒΕͨ ɾςΩετΛදࣔ͢Δ͚ͩͷγϯϓϧͳ಺༰͕ͩɺՄೳੑΛײ͡Δ ʹ͸े෼ͳ࣮૷ͩͬͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

3%։࢝ ɾ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖Δͷ͔ ɹɾϝϯςφϯεੑͷ͋Δ࣮૷͕Ͱ͖Δ͔ ɾ3FQMBZ,JU΍8FC35$Λ࢖༻͍ͯ͠ΔϛϥςΟϒʹ૊ΈࠐΊΔ͔ ɾٕज़తʹԿ͕Ͱ͖ΔͰ͖ͳ͍Λ1.΍σβΠφʔʹ఻͑Δ ɾࢼ࡞඼Λ࡞ٕͬͯज़తͳݶքΛ୳͍ͬͯͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

ϏοτϚοϓͷՃ޻'14 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ 6*,JUͷ࠶ݱ ࢼ࡞඼Λ࡞Γͳ͕Βٕज़తͳ໰୊Λ୳͍ͬͯͨ

J04ϦϦʔε ɾ഑৴தʹϓογϡ௨஌͕දࣔ͞Εͳ͘ͳΔͱ͍͏ࣄ͕݅ى͖ͨ ɾ഑৴தͷϓογϡ௨஌Λ੍ݶ͢Δػೳ͕௥Ճ͞Ε͍ͯͨ ɹɾઃఆΛมߋ͠ͳ͍ͱࢹௌऀͷίϝϯτ͕ಧ͔ͳ͘ͳͬͯ͠·͏
3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ σϑΥϧτ͸Φϑ

J04ϦϦʔε ɾ1J1ͷίϝϯτදࣔػೳͳΒઃఆؔ܎ͳ͘࢖͑Δ ɾ͜Ε͕௥͍෩ʹͳΓਖ਼ࣜʹεέδϡʔϧ͕੾ΒΕͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
σϑΥϧτ͸Φϑ

3%Ͱ੔͑ΒΕͨϦϦʔε·ͰͷಓͷΓ ɾॳظͷϓϩτλΠϓ͕ଘࡏͨ͠ ɾՄೳੑΛ୳ΔͨΊͷ3%Λձ͕ࣾਪਐͨ͠ ɹɾ1.΍σβΠφʔ΋͔ͳΓڠྗతͩͬͨ ɾJ04ͷࣄނ͕ޙԡ͠ʹͳͬͨ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 1J1ͱಈըͷؔ܎ 1J1͸ΞϓϦʹ഑ஔ͞ΕͨϨΠϠʔ্Ͱ࠶ੜ͞Ε͍ͯΔಈը ΛΞϓϦͷྖҬ֎Ͱ࠶ੜ͢Δ࢓૊Έ ຊདྷಈը͕࠶ੜ͞ Ε͍ͯΔϨΠϠʔ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ಈը͕࠶ੜ͞Εͯ ͍ΔྖҬ ͭ·Γ1J1͕࠶ੜ͞Ε͍ͯΔظؒ͸ɺΞϓϦͷͲ͔͜Ͱಈը ͕࠶ੜ͞Ε͍ͯΔ 1J1ͱಈըͷؔ܎

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ Ͳ͔͜Βͱ΋ͳ͘഑৴ίϝϯτόʔ͕χϣΩοͱग़͖͍ͯͯ ΔΑ͏ʹݟ͑Δ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ εϩʔө૾

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ ͜ͷਧ͖ग़͠ͷը૾ͷཪଆʹΊͪΌͪ͘Όখ͍͞ "74BNQMF#VGGFS%JTQMBZ-BZFSΛஔ͍ͯಈըΛ࠶ੜ͍ͯ͠Δ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ "74BNQMF#VGGFS%JTQMBZ-BZFS্ͰಈըΛ࠶ੜ͢Δ ɾ"74BNQMF#VGGFS%JTQMBZ-BZFS͸$.4BNQMF#VGGFSΛड͚औͬ ͯ಺༰Λඳը͢Δ ɹ ɾ഑৴ίϝϯτόʔͰ͸ɺ$.4BNQMF#VGGFS͸ಈըͷ֤ϑϨʔϜ ʹରԠ͢Δը૾σʔλΛอ͍࣋ͯ͠Δ˞ ˞$.4BNQMF#VGGFS͸ಈըͷϑϨʔϜ͚ͩͰͳ͘ɺྫ͑͹ෳ਺νϟωϧͷԻ੠σʔλͳͲ΋දݱͰ͖Δ func enqueue(_
sampleBuffer: CMSampleBuffer)

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 1J1ͷඳըʹ༻͍Δ$.4BNQMF#VGGFSͷߏ଄ $.4BNQMF#VGGFS ɾ$.4BNQMF#VGGFS͸಺෦ʹ$71JYFM#VGGFSΛอ࣋͢Δ ɹɾ$71JYFM#VGGFS͸Ұຕͷը૾ʹ૬౰͢ΔσʔλΛ࣋ͭ ɾ͜ΕΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉ͯ͠දࣔ͢Δ $71JYFM#VGGFS Y ˞࣮ࡍʹ͸6*4DSFFONBJOTDBMFഒͷେ͖͞ ˞
"3(#

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸6*,JUͷੈքΛ࠶ݱ͍ͨ͠ ɾຊ౰͸ಈը͕ͩ6*,JUͰݟͨ໨Λ࡞͍ͬͯΔΑ͏ʹݟ͍ͤͨ ɾ৬ਓٕ͗ͯ͢୭΋ϝϯςφϯεͰ͖ͳ͍࢓૊ΈͩͱࠔΔ ɾ"VUP-BZPVU΋׆༻͍ͨ͠ ˠ"VUP-BZPVUͰ഑ஔ͞Εͨ6*7JFXͷඳը಺༰Λ$71JYFM#VGGFS ʹॻ͖ࠐΈ͍ͨ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 6*7JFXͷඳը಺༰Λ$71JYFM#VGGFSʹॻ͖ࠐΉ let renderer = UIGraphicsImageRenderer(size: size) // UIView →
UIImage の変換 let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) } // UIImage → CGImage → CIImage → CVPixelBuffer の変換 if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) CIContext(options: nil).render(ciImage, to: pixelBuffer) } ྫ

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ඳը಺༰ͷߋ৽ස౓ ɾ഑৴࣌ؒ͸))NNTTܗࣜͳͷͰඵʹճߋ৽͢Ε͹े෼ ɾίϝϯτ͸͋·Γߴස౓Ͱߋ৽ͯ͠΋ಡΊͳ͍ ˠඳը಺༰ͷߋ৽ස౓ '14 ͸ͱͨ͠

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ඞཁͳॲཧΛ·ͱΊΔͱ ҰఆִؒͰඳը༻ϧʔϓΛ૸ΒͤΔ ಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ $.4BNQMF#VGGFSΛϨϯμϥʔʹ౉͢ ඳըॲཧΛߦ͏
Ճ޻ͨ͠$.4BNQMF#VGGFSΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉͢

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ "74BNQMF#VGGFS%JTQMBZ-BZFS $POUSPMMFS -PPQFS 4BNQMF#VGGFS'BDUPSZ 3FOEFSFS ϧʔϓॲཧ '14 ͷ੍ޚ $.4BNQMF#VGGFSͷੜ੒
$71JYFM#VGGFSͷՃ޻ 1J1ͷ੍ޚ ։࢝ऴྃ FYBNQMFTQJQDVTUPN͸͜ͷߏ੒Ͱ࣮૷͞Ε͍ͯ·͢

ඳըίετͱͷઓ͍

ඳըίετͱͷઓ͍ ඳըίετ͕େ͖͗͢Δ໰୊ ɾࢼ࡞ஈ֊͔Βඳըίετʹ໰୊͕͋Δ͜ͱ͸෼͔͍ͬͯͨ ɾϛϥςΟϒ͸6OJUZͰΤϞϞΛඳը͍ͯ͠Δ ɾϝΠϯεϨου΁ͷେ͖͍ෛՙ͸6OJUZͷඳըʹӨڹΛ༩͑Δ ɾը໘ͷେ͖͍୺຤͸6OJUZͷඳըίετ΋େ͖͘ɺϝΠϯεϨο υ΁ͷෛՙͷӨڹΛड͚΍͍͢

ඳըίετͱͷઓ͍ 1J1ͷෛՙͰΤϞϞͷඳըʹࢧো͕ग़͍ͯΔ༷ࢠ

$*$POUFYUͷੜ੒͸Ұ౓͚ͩ $71JYFM#VGGFS1PPMͷར༻ ޮ཰ͷΑ͍ඳըํ๏ ඳըͷճ਺ΛݮΒ͢ $71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ ඳըίετͱͷઓ͍ ඳըίετͷվળʹ༗ޮͩͬͨରࡦ

FYBNQMFTQJQQFSGPSNBODF 1J1ʹؔ࿈͢Δॲཧͷฏۉॲཧ͕࣌ؒܭଌͰ͖·͢

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ let renderer = UIGraphicsImageRenderer(size: size) let uiImage =
renderer.image { context in view.layer.render(in: context.cgContext) } if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) CIContext(options: nil).render(ciImage, to: pixelBuffer) }

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ ɾ$*$POUFYUͷੜ੒͸஗͍ ɾJ1IPOFNJOJͰ͓Αͦ<NT>͔͔Δ ɾ$PSF*NBHF1SPHSBNNJOH(VJEFʹ΋ॻ͍ͯ͋Δ ɾ࡞੒ͨ͠ΠϯελϯεΛอ͓࣋ͯ͜͠͏ Core Image Programming Guide
- Getting the Best Performance

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ lazy var context = CIContext(options: nil) // …
let renderer = UIGraphicsImageRenderer(size: size) let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) } if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) context.render(ciImage, to: pixelBuffer) }

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ ࡍʹࡐྉͱͳΔ$71JYFM#VGGFS $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(#

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾ$71JYFM#VGGFSͷੜ੒΋·͋·͋஗͍ ɾd<NT>͘Β͍͔͔Δ CVPixelBufferCreate( kCFAllocatorDefault, Int(size.width), Int(size.height), kCVPixelFormatType_32ARBG, attributes,
&pixelBufferOut )

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾ$71JYFM#VGGFSͷ࠶ར༻ͷͨΊͷ$71JYFM#VGGFS1PPM ɹɾ6*$PMMFDUJPO7JFXͱ$FMMͷؔ܎ੑ ɾ$71JYFM#VGGFSͷऔಘ͕<NT>ఔ౓ͰࡁΉ CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attributes, &pixelBufferPool) CVPixelBufferPoolCreatePixelBuffer(
kCFAllocatorDefault, pixelBufferPool, &pixelBufferOut )

FYBNQMFTQJQDVTUPN 4BNQMF#VGGFS'BDUPSZTXJGUʹ$71JYFM#VGGFS1PPMΛར༻࣮ͨ͠૷͕͋Γ·͢

ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏ ޮ཰ͷΑ͍ඳըํ๏ ɾ6*7JFX͔Β6**NBHFͷੜ੒Λ͢Δํ๏͸͍͔ͭ͋͘Δ ɾESBX)JFSBSDIZํࣜͱMBZFSSFOEFSํࣜͰൺֱͯ͠ΈΔ

ESBX)JFSBSDIZํࣜ ɾESBX)JFSBSDIZΛ࢖ͬͯ6**NBHFΛੜ੒ ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd<NT>͔͔Δ UIGraphicsBeginImageContextWithOptions(renderingSize, false, 0.0) // 描画 view.drawHierarchy(in: frame,
afterScreenUpdates: true) let image = UIGraphicsGetImageFromCurrentImageContext() UIGraphicsEndImageContext() if let cgImage = image?.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

MBZFSSFOEFSํࣜ ɾMBZFSSFOEFSΛ࢖ͬͯ6**NBHFΛੜ੒ ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd<NT>͔͔Δ let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let image
= imageRenderer.image { context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏ ޮ཰ͷΑ͍ඳըํ๏ ɾESBX)JFSBSDIZํࣜΑΓ΋MBZFSSFOEFSํࣜͷํ͕<NT> ΄Ͳૣ͍ ɾESBX)JFSBSDIZ͸6*7JFXͷΞχϝʔγϣϯʹ௥ैͯ͠ඳըͰ ͖ΔͳͲͷಛੑ͕͋Δ͕ɺ஗͍ ɹɾΞχϝʔγϣϯ͸ͤ͞ͳ͍ͷͰMBZFSSFOEFSΛ࠾༻

ඳըίετͱͷઓ͍ඳըͷճ਺ ·ͩ·ͩඳըͷίετ͕ߴ͍ ɾϏϡʔͷඳըॲཧ͸·ͩக໋తʹίετ͕ߴ͍ ɾ<NT>ۙ͘ॲཧ͕͔͔࣌ؒͬͯ͠·͏ // 以下のコードで配信コメントバーの全てのビューを // 再起的に描画しようとすると 30[ms] 近くかかってしまう
let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) }

ඳըίετͱͷઓ͍ඳըͷճ਺ Ͳ͜ʹίετ͕͔͔͍ͬͯΔ͔ ɾը૾ 6**NBHF7JFX Λඳըͨ͠ͱ͖ͷίετ͕ߴ͍ ɹɾը૾ʹෆಁ໌ϚεΫ΋͔͚͍ͯͨΓ΋͢Δ ɾը૾Λ͋Β͔͡Ίॖখͯ͠࢖༻ͯ͠΋े෼ͳޮՌ͸ͳ͍ let uiImage =
renderer.image { context in // view の階層が深くなればなるほど時間がかかる view.layer.render(in: context.cgContext) }

ඳըίετͱͷઓ͍ඳըͷճ਺ SFOEFS JODPOUFYU ΛݺͿճ਺ΛݮΒͯ͠Έͨ ɾ͍ΘΏΔυϩʔίʔϧΛݮΒ͢ ɾߋ৽ස౓͕௿͍ཁૉɺߴ͍ཁૉΛ6*7JFXͷUBHͰ෼ྨ͢Δ ɾߋ৽ස౓͕௿͍ཁૉ͸·ͱΊͯҰͭͷը૾ʹ͓ͯ͘͠ ߋ৽ස౓͕௿͍ཁૉ සൟʹߋ৽͞ΕΔཁૉ

ߋ৽ස౓͕௿͍ཁૉͷΈ͔Βੜ੒ͨ͠ը૾͸ ඞཁͳ͚࣌ͩߋ৽ͯ͠ɺΩϟογϡ͓ͯ͘͠ ɾΞόλʔ͕ߋ৽͞Εͨͱ͖ ɾഎܠը૾͕ߋ৽͞Εͨͱ͖ ɾԻ੠ͷ0/0''ͷ੾Γସ͕͑͞Εͨͱ͖ ߋ৽ස౓͕ߴ͍ϏϡʔͷBMQIBΛʹͯ͠6*7JFX͔Β6**NBHFΛੜ੒͢Δ ඳըίετͱͷઓ͍ඳըͷճ਺

ߋ৽ස౓͕ߴ͍ཁૉ͸Ωϟογϡͨ͠ը૾ͷ্ʹ ελϯϓܗࣜʢΠϝʔδʣͰඳը͍ͯ͘͠ ඳըίετͱͷઓ͍ඳըͷճ਺ func renderRedrawnContents(on view: UIView, in context: CGContext)
{ if view.tag == DrawingPolicy.redrawn.tag { view.layer.draw(in: context) } for subview in view.subviews { let origin = subview.frame.origin context.translateBy(x: origin.x, y: origin.y) renderRedrawnContents(on: subview, in: context) context.translateBy(x: -origin.x, y: -origin.y) } } ݸผʹඳը

Ұຕֆͷߋ৽ͱίϝϯτόʔͷඳըͷྲྀΕͷ֓೦ਤ // エモモや背景などが更新された時だけ一枚絵を再生成する if needsToUpdateStableContents { updateStableContentsImage() } let uiImage
= renderer.image { context in // 更新頻度の低い要素をまとめた画像の描画 renderStableContents(in: context.cgContext) // 更新頻度の高い要素は個別に描画する renderRedrawnContents(on: view, in: context.cgContext) } ඳըίετͱͷઓ͍ඳըͷճ਺

ඳըͷճ਺ΛݮΒ͢ޮՌ ɾඳըʹ͔͔Δ͕࣌ؒd<NT>͔Βd<NT>ʹվળ͞Εͨ ɾ͜ͷվળʹΑͬͯը໘ͷνϥ͖ͭ͸શ୺຤Ͱͳ͘ͳͬͨ ɾ࣮͸͜ͷ࣮૷͸2"ͷ࠷ऴ೔Ͱ׬੒ͨ͠ʢʂʣ .JSSBUJW5FDI#MPH ഑৴ίϝϯτόʔʙ1J1ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ ඳըίετͱͷઓ͍ඳըͷճ਺

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ ͞ΒʹඳըॲཧͷશମΛݟ௚ͯ͠ΈΔ ɾ7JFXͷ಺༰͕$71JYFM#VGGFSʹඳը͞ΕΔ·Ͱͷܦ࿏ ɾͲ͏͍ͬͨதؒ෺͕ੜ੒͞Ε͍ͯΔ͔ let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let
image = imageRenderer.image { context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) }

let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let image = imageRenderer.image {
context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ 6*7JFXˠ$($POUFYUˠ6**NBHFˠ$**NBHFˠ$71JYFM#VGGFS

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU 6**NBHF $**NBHF ඳը
ඳը

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $71JYFM#VGGFSͱ$($POUFYUͷؔ܎ ɾ$71JYFM#VGGFS͸"3(#ܗࣜͷσʔλΛอ͍࣋ͯ͠Δ ɾ$($POUFYU͸"3(#ܗࣜͷσʔλΛࢀরͯ͠ੜ੒͢Δ͜ͱ͕ Ͱ͖Δ

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) // CVPixelBuffer が保持しているデータのアドレスを取得する let data = CVPixelBufferGetBaseAddress(pixelBuffer)
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly) // CVPixelBuffer が保持しているデータを参照した CGContext を生成する guard let context = CGContext( data: data, width: width, height: height, bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue // ARGB ) else { return }

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# $($POUFYU

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU ඳը

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU ඳը 6*7JFX $($POUFYU
6**NBHF $**NBHF ඳը ඳը ϝϞϦͷ֬อ΍్தͷॲཧΛݮΒ͢͜ͱͰɺd<NT>ͷ ඳըίετ͕࡟ݮͰ͖Δ

ඳըίετͱͷઓ͍ ඳըίετͱͷઓ͍ͷ݁Ռ ɾ࠷ѱͷঢ়ଶͰ͸ϑϨʔϜ͋ͨΓ<NT>ఔ౓ͷඳըίετ͕ ͔͔͕ͬͨɺd<NT>ఔ౓ʹվળͨ͠ ɾը໘ͷେ͖͍୺຤Ͱ΋6OJUZͷඳըͳͲʹࢧো͸ͳ͘ɺগͳ͍ ෛՙͰ഑৴ίϝϯτόʔͷػೳ͕࣮ݱͰ͖ͨ

·ͱΊ ɾ഑৴ίϝϯτόʔ͸3%ʹΑͬͯੜ·Εֵͨ৽తͳػೳ ɹɾ1J1্ʹ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖ͨ ɹɹɾ"VUP-BZPVUͰ഑ஔ͞ΕɺϝϯςφϯεੑΛ୲อͨ͠ ɹɾෛՙ΋ඞཁ࠷௿ݶʹ཈͑ΒΕͨ ɾ3%Λͨ͠Ձ஋͕͋ͬͨ ɹɾϓϨογϟʔ΋େ͖͔͕ͬͨɺಘΒΕΔܦݧ΋ଟ͔ͬͨ ɹɾ৽͍͠ػೳ͕ੜ·Εɺٕज़ൃ৴ʹ΋ܨ͕ͬͨ

͋Γ͕ͱ͏͍͟͝·ͨ͠

PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

More Decks by naru-jpn

Other Decks in Programming

Featured

Transcript