Upgrade to Pro — share decks privately, control downloads, hide ads and more …

動画だけじゃない!iOS 15のピクチャ・イン・ピクチャを使って好きなUIを表示させよう!

動画だけじゃない!iOS 15のピクチャ・イン・ピクチャを使って好きなUIを表示させよう!

iOS 14まではピクチャ・イン・ピクチャ(以下PiP)を表示させるには動画コンテンツが必要でした。
しかし、新しくiOS 15でPiPのAPIが追加されたことにより動画コンテンツが無いただのUIViewもPiPとして表示させることが可能になりました!

これまでPiPを利用したアプリを3つリリースしてきた経験から、PiPを利用したアプリの開発からリリースするまでについて話したいと思います。

・PiPに好きなUIを表示させる仕組みと実装
・より良いPiP体験の提供
・PiPでできないこと
・Appleの審査を通過する

PiPを使うことでユーザーにより良い体験を与えることができるアプリはたくさんあると自分は感じています。
ぜひこのセッションで得た情報をもとにPiPを使った良いアプリが増えれば良いと願っています!

この資料はiOSDC2022 の補足としてお使いください

Ryo Tsuzukihashi

September 12, 2022
Tweet

More Decks by Ryo Tsuzukihashi

Other Decks in Technology

Transcript

  1. i O S 1 5 ͷ 
 ϐΫ ν ϟ

    ɾ Π ϯ ɾ ϐΫ ν ϟ 
 Λ ࢖ ͬ ͯ 
 ޷ ͖ ͳ U I Λ ද ࣔ ͞ ͤ Α ͏ ʂ ᠃ ڮ ɹ ྋ ಈը͚ͩ͡Όͳ͍ʂ iOSDC Japan 2022 1
  2. @ t s u z u k i 8 1

    7 🐸ࣗݾ঺հ🐸 Yahoo JAPAN! PayPayϑϦϚ iOS։ൃ • Q.UIKitͱSwiftUIͲ͕ͬͪ޷͖ʁ 
 RYO TSUZUKIHASHI ϛχΤϐιʔυ 2019೥৽ଔͷ࣌ʹॳΊͯiOSDCʹࢀՃ ͍͔ͭࣗ෼΋ొஃ͍ͨ͠ͱಌΕΛ๊͘Α ͏ʹͳΓɺҎ߱ຖ೥ࢀՃ ͦͯ͠4೥໨ͷ2022೥ʹ೦ئͷొஃ✌
  3. @ t s u z u k i 8 1

    7 🐸ࣗݾ঺հ🐸 Yahoo JAPAN! PayPayϑϦϚ iOS։ൃ • Q.UIKitͱSwiftUIͲ͕ͬͪ޷͖ʁ 
 A. ͲͪΒ΋޷͖ • Q. झຯ͸? 
 RYO TSUZUKIHASHI ϛχΤϐιʔυ 2019೥৽ଔͷ࣌ʹॳΊͯiOSDCʹࢀՃ ͍͔ͭࣗ෼΋ొஃ͍ͨ͠ͱಌΕΛ๊͘Α ͏ʹͳΓɺҎ߱ຖ೥ࢀՃ ͦͯ͠4೥໨ͷ2022೥ʹ೦ئͷొஃ✌
  4. @ t s u z u k i 8 1

    7 🐸ࣗݾ঺հ🐸 Yahoo JAPAN! PayPayϑϦϚ iOS։ൃ • Q.UIKitͱSwiftUIͲ͕ͬͪ޷͖ʁ 
 A. ͲͪΒ΋޷͖ • Q. झຯ͸? 
 A. ݸਓΞϓϦ։ൃ • Q. ޷͖ͳۦಈ։ൃ͸ʁ 
 RYO TSUZUKIHASHI ϛχΤϐιʔυ 2019೥৽ଔͷ࣌ʹॳΊͯiOSDCʹࢀՃ ͍͔ͭࣗ෼΋ొஃ͍ͨ͠ͱಌΕΛ๊͘Α ͏ʹͳΓɺҎ߱ຖ೥ࢀՃ ͦͯ͠4೥໨ͷ2022೥ʹ೦ئͷొஃ✌
  5. @ t s u z u k i 8 1

    7 🐸ࣗݾ঺հ🐸 Yahoo JAPAN! PayPayϑϦϚ iOS։ൃ • Q.UIKitͱSwiftUIͲ͕ͬͪ޷͖ʁ 
 A. ͲͪΒ΋޷͖ • Q. झຯ͸? 
 A. ݸਓΞϓϦ։ൃ • Q. ޷͖ͳۦಈ։ൃ͸ʁ 
 A. ςετۦಈ։ൃ RYO TSUZUKIHASHI ϛχΤϐιʔυ 2019೥৽ଔͷ࣌ʹॳΊͯiOSDCʹࢀՃ ͍͔ͭࣗ෼΋ొஃ͍ͨ͠ͱಌΕΛ๊͘Α ͏ʹͳΓɺҎ߱ຖ೥ࢀՃ ͦͯ͠4೥໨ͷ2022೥ʹ೦ئͷొஃ✌
  6. PiPͱ͸ • ଞͷΞϓϦΛ࢖༻͠ͳ͕ΒFaceTimeΛ࢖ͬͨΓ ϏσΦΛࢹௌͨ͠ΓͰ͖Δػೳͷ͜ͱ • ΢Οϯυ΢ΛϐϯνΦʔϓϯɾϐϯνΫϩʔζ͢ Δ͜ͱͰαΠζΛมߋ͢Δ͜ͱ͕Ͱ͖Δ • ଞʹ΋Ҡಈɾඇදࣔɾ࡟আ •

    ಈըͷ࠶ੜɾఀࢭɾ15sec ͷૣૹΓͱר͖໭͠ Ҿ༻J1IPOFͷϐΫνϟɾΠϯɾϐΫνϟΛ࢖༻ͨ͠ϚϧνλεΫ IUUQTTVQQPSUBQQMFDPNKBKQHVJEFJQIPOFJQIDDCEJPT
  7. iOS 15͔ΒͷPicture In Picture • ~ iOS 14·Ͱ 
 PiP͢Δ͜ͱ͕Ͱ͖Δͷ͸جຊతʹಈըͷΈ

    • iOS 15͔Β • AVPictureInPictureController.ContentSourceͷ௥Ճ 
 →ɹAVSampleBufferDisplayLayerͷίϯςϯπΛPicture In Pictureαϙʔτ • ͦͷଞPiP༻ͷϝιουͷ௥Ճ
  8. AVSampleBufferDisplayLayer • ѹॖɾඇѹॖͷVideoFrameΛදࣔ͢ΔΦϒδΣΫτ (iOS 8 ~) • CMSampleBufferΛ༩͑Δ͜ͱʹΑͬͯ ಈըΛ࠶ੜ͢Δ͜ͱ͕Ͱ͖Δ CMSampleBuffer

    • ө૾ɺԻ੠ɺ͋Δ͍͸ͦͷ྆ํ౳ɺϝσΟΞσʔλΛ࣋ͪӡͿͨΊͷΦϒδΣΫτ • UIView͔Β࡞ΕΔ ↓ ޷͖ͳUIΛPicture In Pictureͤ͞Δ͜ͱ͕Ͱ͖Δʂ
  9. ࣮ફPiP Ͳ ͷ Α ͏ ʹ P i P Λ

    ࣮ ݱ ͞ ͤ Δ ͷ ͔
  10. αϯϓϧϓϩδΣΫτ • ࣮ػϏϧυ͕Ͱ͖Ε͹ࢼͤΔ https://github.com/tsuzukihashi/sample-pip • PiPManager 
 PiPΛ؅ཧ͢ΔΫϥεɺγϯάϧτϯ • UIViewExtension

    
 UIViewΛCMSampleBufferʹม׵͢ΔExtension • ContentView 
 ϝΠϯͷView ϘλϯͳͲΛ഑ஔ • ContentViewModel 
 PiPManagerΛอ࣋͠Πϕϯτͷड͚౉͠Λ͢Δ • PiPContainerView 
 SwiftUIͰAVSampleBufferDisplayLayerΛඳը͢ΔͨΊͷView
  11. ࣮૷ͷલʹඞཁͳઃఆ • AudioSessionΛ։࢝͢Δ 
 <category: .playAndRecord, mode: .moviePlayback> 
 PiPΛىಈͰ͖ɺ࠶ੜதͷԻָΛࢭΊͣʹࡁΉ૊Έ߹Θͤ

    override init() { let session = AVAudioSession.sharedInstance() do { try session.setCategory(.playAndRecord, mode: .moviePlayback) try session.setActive(true) } catch { print("Failed to set AVAudioSession: \(error)") } } PiPManagerͷॳظԽͰߦ͍ͬͯΔ
  12. PiP͍ͤͨ͞View private let dateLabel: UILabel = { let label =

    UILabel() label.frame = .init( origin: .zero, size: .init( width: UIScreen.main.bounds.width, height: 120 ) ) label.font = .monospacedSystemFont(ofSize: 24, weight: .medium) label.textAlignment = .center label.textColor = .black label.backgroundColor = .white return label }() PiPManager಺Ͱఆٛ
  13. CMSampleBufferΛ࡞੒ • UIViewΛCMSampleBufferʹม׵͢ΔExtension https://gist.github.com/tsuzukihashi/97e379a42e32cc0647aa7a4770d2d9a6 do { return try CMSampleBuffer( imageBuffer:

    pixelBuffer, formatDescription: formatDescription, sampleTiming: getCMSampleTimingInfo() ) } catch { assertionFailure("Failed to create CMSampleBuffer: \(error)") return nil }
  14. UIView →CVPixelBuffer var pixelBuffer: CVPixelBuffer? let createPixelBufferResult: OSStatus = CVPixelBufferCreate(

    kCFAllocatorDefault, Int(size.width), Int(size.height), kCVPixelFormatType_32ARGB, [ kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue!, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue!, kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary, ] as CFDictionary, &pixelBuffer )
  15. CVPixelBuffer →CGContext • CGContextͱ͸ 
 Quartz 2DͷඳըઌΛද͢ 
 ඳըύϥϝʔλͱϖʔδ্ͷϖΠϯτΛѼઌʹϨϯμϦϯά͢ΔͨΊʹඞཁͳ͢΂ͯͷ σόΠεݻ༗৘ใؚ͕·Ε͍ͯΔ

    • Quartz 2Dͱ͸ 
 iOS, tvOS, macOSͷΞϓϦ։ൃͰར༻Ͱ͖Δ2࣍ݩඳըΤϯδϯ 
 ௿ϨϕϧͰܰྔͳ2DϨϯμϦϯάػೳΛఏڙ https://developer.apple.com/documentation/coregraphics/cgcontext
  16. CVPixelBuffer →CGContext guard let context: CGContext = .init( data: CVPixelBufferGetBaseAddress(pixelBuffer),

    width: Int(size.width), height: Int(size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue ) else { return nil }
  17. CVPixelBuffer →CMFormatDescription • CMFormatDescriptionͱ͸ 
 αϯϓϧόοϑΝ಺ͷαϯϓϧΛهड़͢ΔϝσΟΞϑΥʔϚοτهड़ࢠ 
 ΦʔσΟΦɺϏσΦɺ͓Αͼ Muxed ϝσΟΞσʔλͳͲɺϝσΟΞλΠϓʹͱΒΘΕ

    ͳ͍΋ͷͱϝσΟΞʹಛԽͨ͠΋ͷ͕͋Δ • CMVideoFormatDescriptionCreateForImageBuffer 
 ΠϝʔδόοϑΝΛ࢖༻ͯ͠ϏσΦϝσΟΞετϦʔϜ༻ͷFormatDesciptionΛ࡞੒͢ Δ https://developer.apple.com/documentation/coremedia/cmformatdescription-u8g
  18. CMSampleTimingInfo • CMSampleTimingInfoͱ͸ 
 CMSampleBuffer಺ͷ͢΂ͯͷαϯϓϧͷλΠϛϯά৘ใΛूΊͨ΋ͷ • init(duration:presentationTimeStamp:decodeTimeStamp:) • duration 


    αϯϓϧͷظؒ • presentationTimeStamp 
 αϯϓϧ͕දࣔ͞ΕΔ࣌ؒ • decodeTimeStamp 
 αϯϓϧ͕σίʔυ͞ΕΔ࣌ؒ https://developer.apple.com/documentation/coremedia/cmsampletiminginfo
  19. CMSampleTimingInfo let currentTime: CMTime = .init( seconds: CACurrentMediaTime(), preferredTimescale: 60

    ) let timingInfo: CMSampleTimingInfo = .init( duration: .init(seconds: 1, preferredTimescale: 60),ɹ presentationTimeStamp: currentTime, decodeTimeStamp: currentTime )
  20. CMSampleBufferΛ࡞੒ • UIViewΛCMSampleBufferʹม׵͢ΔExtension https://gist.github.com/tsuzukihashi/97e379a42e32cc0647aa7a4770d2d9a6 do { return try CMSampleBuffer( imageBuffer:

    pixelBuffer, formatDescription: formatDescription, sampleTiming: getCMSampleTimingInfo() ) } catch { assertionFailure("Failed to create CMSampleBuffer: \(error)") return nil }
  21. AVSampleBufferDisplayLayerΛSwiftUI͔Β࢖͑ΔΑ͏ʹ͢Δ struct PiPContainerView: UIViewRepresentable { let bufferDisplayLayer: AVSampleBufferDisplayLayer let frame:

    CGRect func makeUIView(context: Context) -> UIView { let view = UIView() view.frame = frame bufferDisplayLayer.frame = view.bounds bufferDisplayLayer.videoGravity = .resizeAspect view.layer.addSublayer(bufferDisplayLayer) return view } func updateUIView(_ uiView: UIView, context: Context) {}ɹ }
  22. AVPictureInPictureController • PiPΛ؅ཧ͢Δίϯτϩʔϥ • AVPictureInPictureController.ContentSourceʹAVSampleBufferDisplayLayerͱdelegate Λ౉͢ pipController = AVPictureInPictureController( contentSource:

    AVPictureInPictureController.ContentSource(ɹɹ sampleBufferDisplayLayer: bufferDisplayLayer, playbackDelegate: self ) ) pipController ? . delegate = self
  23. AVPictureInPictureController • PiPΛ؅ཧ͢Δίϯτϩʔϥ • AVPictureInPictureController.ContentSourceʹAVSampleBufferDisplayLayerͱdelegate Λ౉͢ pipController = AVPictureInPictureController( contentSource:

    AVPictureInPictureController.ContentSource(ɹɹ sampleBufferDisplayLayer: bufferDisplayLayer, playbackDelegate: self ) ) pipController ? . delegate = self PiPManagerΛAVPictureInPictureControllerDelegateͱ AVPictureInPictureSampleBufferPlaybackDelegateʹ४ڌͤ͞Δ
  24. func prepare() { let timerBlock: ((Timer) -> Void) = {

    [weak self] timer in guard let buffer: CMSampleBuffer = self ? . nextBuffer() else { return } self ? . bufferDisplayLayer.enqueue(buffer) } let timer = Timer(timeInterval: 1, repeats: true, block: timerBlock) self.timer = timer RunLoop.main.add(timer, forMode: .default) pipController = … pipController ? . delegate = self } PiPManager
  25. private func nextBuffer() - > CMSampleBuffer? { if bufferDisplayLayer.status =

    = .failed { bufferDisplayLayer.flush() } dateLabel.text = Date().formatted(date: .numeric, time: .complete) return dateLabel.toCMSampleBuffer() } PiPManager • flush() 
 iPhoneΛεϦʔϓঢ়ଶ͔Β෮ؼͨ͠ͱ͖ͳͲ 
 AVQueuedSampleBufferRenderingStatus͕.failedʹͳΓΤϥʔʹͳΔ 
 ͜ͷϝιουΛݺͼอཹதͷαϯϓϧόοϑΝΛഁغ͢Δ Ignoring enqueueSampleBuffer: because status is “failed"
  26. PiPͷΠϕϯτΛݕ஌͢ΔͨΊͷϓϩτίϧ • pictureInPictureControllerWillStartPictureInPicture(_:) 
 Picture in Pictureͷ։࢝͞ΕΔ͜ͱΛ௨஌ • pictureInPictureControllerDidStartPictureInPicture(_:) 


    Picture in Picture͕։࢝͞Εͨ͜ͱ௨஌ • pictureInPictureController(_:failedToStartPictureInPictureWithEr ror:) 
 Picture in Pictureͷىಈʹࣦഊͨ͜͠ͱΛ௨஌ • pictureInPictureControllerWillStopPictureInPicture(_:) 
 Picture in Picture͕ఀࢭ͢Δ͜ͱΛ௨஌ʢ໌ࣔత, Ϣʔβʔ, γεςϜ໰Θͣʣ • pictureInPictureControllerDidStopPictureInPicture(_:) 
 Picture in Picture͕ఀࢭͨ͜͠ͱΛ௨஌
  27. AVPictureInPicture SampleBuffer PlaybackDelegate AV S a m p l e

    B u f f e r D i s p l a y L a y e r ͔ Β ੍ ޚ ͢ Δ
  28. pictureInPictureController(_:skipByInterval:completion:) • Ϣʔβ͕ࢦఆ͞Εִ͚ͨ࣌ؒؒͩલํ·ͨ͸ޙํʹҠಈ͢Δ͜ͱ఻͑Δ • skipInterval͸15 or -15Ͱฦ٫͞ΕΔ func pictureInPictureController( _

    pictureInPictureController: AVPictureInPictureController, skipByInterval skipInterval: CMTime, completion completionHandler: @escaping () -> Void ) { completionHandler() }
  29. pictureInPictureController(_:didTransitionToRenderSize:) • PiPͷαΠζ͕มߋ͞Εͨ͜ͱͱͦͷαΠζΛ௨஌ func pictureInPictureController( _ pictureInPictureController: AVPictureInPictureController, didTransitionToRenderSize newRenderSize:

    CMVideoDimensions ) { dateLabel.text = "w: \(newRenderSize.width) h: \(newRenderSize.height)" if let sampleBuffer = dateLabel.toCMSampleBuffer() { bufferDisplayLayer.enqueue(sampleBuffer) } }
  30. AVPictureInPictureController • startPictureInPicture() 
 ՄೳͰ͋Ε͹PiPΛ։࢝͢Δ • stopPictureInPicture() 
 PiP͕ΞΫςΟϒঢ়ଶͳΒఀࢭ͢Δ •

    isPictureInPictureActive 
 ݱࡏɺPicture in Picture͕༗ޮͰ͋Δ͔Ͳ͏͔ func swapPictureInPicture() { if pipController ?. isPictureInPictureActive == true { pipController ?. stopPictureInPicture() } else { pipController ?. startPictureInPicture() } }
  31. ContentViewModel final class ContentViewModel: ObservableObject { @Published var isReady: Bool

    = false let pipManager: PiPManager = .shared func didTapMainButton() { if isReady { pipManager.reset() } else { pipManager.prepare() } isReady.toggle() } func didTapPiPSwap() { pipManager.swapPictureInPicture() } }
  32. ·ͱΊ • iOS 15͔Β޷͖ͳUIΛPicture In PictureͰදࣔͤ͞Δ͜ͱ͕Ͱ͖ ΔΑ͏ʹͳͬͨ • CMSampleDisplayBuffer͕ॏཁ •

    Picture In Picture͸Ϣʔβʔ΋·ͩෆ׳ΕͳͨΊɺखް͘αϙʔτ ͢Δͱྑ͍ • جຊతͳ࢖͍ํ • ࣗಈΦϑઃఆͷ௥ՃͳͲ
  33. ·ͱΊ • iOS 15͔Β޷͖ͳUIΛPicture In PictureͰදࣔͤ͞Δ͜ͱ͕Ͱ͖ ΔΑ͏ʹͳͬͨ • CMSampleDisplayBuffer͕ॏཁ •

    Picture In Picture͸Ϣʔβʔ΋·ͩෆ׳ΕͳͨΊɺखް͘αϙʔτ ͢Δͱྑ͍ • جຊతͳ࢖͍ํ • ࣗಈΦϑઃఆͷ௥ՃͳͲ ·ͩݟ͵PiPͷΞϓϦΛ࡞ͬͯΈ͍ͯͩ͘͞🙏
  34. ࢀߟURL AppleͷυΩϡϝϯτ https://developer.apple.com/documentation/avkit/accessing_the_camera_while_multitasking ഑৴ίϝϯτόʔ ʙ iOS15 Ͱ࣮ݱ͢Δ৽͍͠ PiP ମݧ https://tech.mirrativ.stream/entry/2021/11/26/114002

    iOS Ͱ೚ҙͷ UIView ΛϐΫνϟʔΠϯϐΫνϟʔ͢Δ https://zenn.dev/uakihir0/articles/211128-uipip UIViewͷදࣔ಺༰ΛCMSampleBu ff erʹ͢Δ https://soranoba.net/programming/uiview-to-cmsamplebu ff er