Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AVFoundation, iOS 11 & Lumina

David Okun
November 07, 2017

AVFoundation, iOS 11 & Lumina

Come learn about what’s new in iOS 11 with AVFoundation, as David Okun walks us through new ways to capture still images, live photos, and depth maps via his open source framework, Lumina.

David Okun

November 07, 2017
Tweet

More Decks by David Okun

Other Decks in Programming

Transcript

  1. WEIRD SWIFT ATX •Meets every Thursday •6-10 pm •Been at

    BennU coffee so far but… •…next week is at Galvanize!!! @dokun24
  2. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 @dokun24
  3. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 @dokun24
  4. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data @dokun24
  5. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina @dokun24
  6. What We’ll Cover • A history of AVFoundation’s input and

    output • What changed in iOS 10 • New methods in iOS 11 • Capturing Depth Data • Lumina • Synchronizing Asynchronously Streaming Data @dokun24
  7. What We Won’t Cover • High Efficiency Image File Format

    (HEIF) • High Efficiency Video Coding (HEVC) @dokun24
  8. What We Won’t Cover • High Efficiency Image File Format

    (HEIF) • High Efficiency Video Coding (HEVC) • AVCaptureDepthDataOutput @dokun24
  9. Reference Material • WWDC 2016, Session 501 • WWDC 2017,

    Session 507 • https://github.com/dokun1/lumina @dokun24
  10. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode @dokun24
  11. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool @dokun24
  12. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool @dokun24
  13. Since iOS 10 AVCapturePhotoSettings var uniqueID: Int64 // generated on

    init var flashMode: AVCaptureDevice.FlashMode var isAutoStillImageStabilizationEnabled: Bool var isHighResolutionPhotoEnabled: Bool var isAutoDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
  14. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:

    AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  15. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings:

    AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  16. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:

    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) @dokun24
  17. Since iOS 10 AVCapturePhotoOutputDelegate func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photoSampleBuffer:

    CMSampleBuffer?, previewPhoto previewPhotoSampleBuffer: CMSampleBuffer?, resolvedSettings: AVCaptureResolvedPhotoSettings, bracketSettings: AVCaptureBracketedStillImageSettings?, error: Error?) func photoOutput(_ output: AVCapturePhotoOutput, willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, willCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didCapturePhotoFor resolvedSettings: AVCaptureResolvedPhotoSettings) func photoOutput(_ output: AVCapturePhotoOutput, didFinishCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings, error: Error?) @dokun24
  18. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool @dokun24
  19. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool @dokun24
  20. Since iOS 10 AVCaptureResolvedPhotoSettings var uniqueID: Int64 var photoDimensions: CMVideoDimensions

    var livePhotoMovieDimensions: CMVideoDimensions var isFlashEnabled: Bool var isStillImageStabilizationEnabled: Bool var isDualCameraFusionEnabled: Bool // iOS 10.2+ @dokun24
  21. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings @dokun24
  22. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int @dokun24
  23. New in iOS 11 AVCapturePhoto var pixelBuffer: CVPixelBuffer? var previewPixelBuffer:

    CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData? @dokun24
  24. New in iOS 11 AVCapturePhoto func cgImageRepresentation() -> Unmanaged<CGImage>? func

    takeRetainedValue() -> Instance func takeUnretainedValue() -> Instance @dokun24
  25. Back to iOS 10 AVCapturePhotoOutputDelegate -> CMSampleBuffer -> UIImage guard

    let imageData = AVCapturePhotoOutput.jpegPhotoDataRepresentation(forJPEGSampleBuffer: sampleBuffer, previewPhotoSampleBuffer: nil) else { return nil } guard let dataProvider = CGDataProvider(data: imageData as CFData) else { return nil } guard let cgImage = CGImage(jpegDataProviderSource: dataProvider, decode: nil, shouldInterpolate: true, intent: CGColorRenderingIntent.defaultIntent) else { return nil } return UIImage(cgImage: cgImage) @dokun24
  26. New in iOS 11 AVCapturePhoto -> UIImage guard let cgImage

    = photo.cgImageRepresentation() else { return nil } return UIImage(cgImage: cgImage.takeUnretainedValue()) @dokun24
  27. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var

    previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
  28. New in iOS 11 AVCapturePhoto @dokun24 var pixelBuffer: CVPixelBuffer? var

    previewPixelBuffer: CVPixelBuffer? var metadata: [String : Any] var resolvedSettings: AVCaptureResolvedPhotoSettings var photoCount: Int var depthData: AVDepthData?
  29. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? @dokun24
  30. Capturing Depth Data AVDepthData var depthDataAccuracy: AVDepthData.Accuracy var isDepthDataFiltered: Bool

    var depthDataQuality: AVDepthData.Quality var cameraCalibrationData: AVCameraCalibrationData? var depthDataMap: CVPixelBuffer @dokun24
  31. Capturing Depth Data // in your capture session configuration if

    self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } @dokun24
  32. Capturing Depth Data // in your capture session configuration if

    self.photoOutput.isDepthDataDeliverySupported { self.photoOutput.isDepthDataDeliveryEnabled = true } // in your preparation for capturing a still image with AVCapturePhotoSettings if self.photoOutput.isDepthDataDeliverySupported { settings.isDepthDataDeliveryEnabled = true } @dokun24
  33. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… @dokun24
  34. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
  35. Lumina • Gratuitous plugging! • A full service camera framework

    • QR codes, bar codes, live photos, the list goes on… • …and the easiest CoreML integration you’ll see @dokun24
  36. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces @dokun24
  37. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos @dokun24
  38. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps @dokun24
  39. Synchronizing Asynchronously Streaming Data • AVCaptureVideoDataOutput -> video frames •

    AVCaptureMetadataOutput -> QR codes, barcodes, faces • AVCapturePhotoOutput -> still photos • AVCaptureDepthDataOutput -> depth data maps • You see the problem here, right? @dokun24
  40. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:

    [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate @dokun24
  41. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizer // iOS 11.0 + init(dataOutputs:

    [AVCaptureOutput]) var delegate: AVCaptureDataOutputSynchronizerDelegate var delegateCallbackQueue: DispatchQueue? @dokun24
  42. Synchronizing Asynchronously Streaming Data AVCaptureDataOutputSynchronizerDelegate func dataOutputSynchronizer(_ synchronizer: AVCaptureDataOutputSynchronizer, didOutput

    synchronizedDataCollection: AVCaptureSynchronizedDataCollection) { let collection = synchronizedDataCollection } @dokun24
  43. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func

    synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24
  44. Synchronizing Asynchronously Streaming Data AVCaptureSynchronizedDataCollection // iOS 11.0 + func

    synchronizedData(for captureOutput: AVCaptureOutput) -> AVCaptureSynchronizedData? @dokun24 var count: Int