Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let's make an Immersive Video with APMP

Let's make an Immersive Video with APMP

This document summarizes a brief explanation of APMP, which was announced at WWDC25.

Shingo Tamaki

August 10, 2025

Transcript

25 slides
  1. ISOBMFF (ISO Base Media File Format) ߏ଄ ISOBMFF File ftyp

    (File Type Box) ϑΝΠϧλΠϓɺϒϥϯυɺޓ׵ੑ৘ใ moov (Movie Box) - ϝλσʔλίϯςφ mvhd (Movie Header) શମͷ࣌ؒ৘ใ trak (Track Box) τϥοΫ৘ใ mvex (Movie Extends) ϑϥάϝϯτ༻֦ு tkhd ϔομʔ mdia ϝσΟΞ mdhd hdlr minf mdat (Media Data Box) ࣮ࡍͷϝσΟΞσʔλʢԻ੠ɾө૾ɾࣈນͳͲʣ ௨ৗϑΝΠϧͷେ෦෼Λ઎ΊΔ free/skip (Free Space) ະ࢖༻ྖҬ udta (User Data) Ϣʔβʔσʔλ ͦͷଞͷBox ֦ுɾΧελϜBox ஫ɿBox ͸֊૚ߏ଄Λ࣋ͪɺ֤Box ͸ size (4bytes) + type (4bytes) + data Ͱߏ੒͞ΕΔ ISOBMFFɺQTFFͱ͸ • ISOBMFF͸ϏσΦ΍ΦʔσΟΦͳͲ ͷϚϧνϝσΟΞσʔλΛؚΉϑΝΠ ϧͷҰൠతͳߏ଄͕ఆٛ͞Εͨࠃࡍ ج४ͷϑΥʔϚοτ • QTFF͕ϕʔε • MP4͸ISOBMFFʹج͍࣮ͮͨࡍͷ ϑΝΠϧܗࣜ
  2. 2. ࣗಈม׵ 1. ·ͣInsta360͔Go ProΛߪೖ͠·͠ΐ͏ 2. Insta360 StudioͰMP4΁ग़ྗ͠·͢ 3. AirDropͰAVPʢvisionOS

    26 betaʣ΁ૹΔ 4. visionOSͰ౰֘ϑΝΠϧΛ։͘ࡍʹม׵͢Δ͔Ͳ͏͔μΠΞ ϩά͕ग़ͯ͘ΔͷͰબͿ
  3. ύοΩϯά if let viewPackingKind = projectedMediaMetadata.viewPackingKind { isFramePacked = true

    if viewPackingKind.caseInsensitiveCompare("SideBySide") == .orderedSame { horizontalScale = 2.0 isSideBySide = true } else if viewPackingKind.caseInsensitiveCompare("OverUnder") == .orderedSame { verticalScale = 2.0 } } let eyeFrameSize = CGSize(width: sourceVideoFrameSize.width / horizontalScale, height: sourceVideoFrameSize.height / verticalScale) ... let cropRectDict = [ kCVImageBufferCleanApertureHorizontalOffsetKey: apertureHorizontalOffset, kCVImageBufferCleanApertureVerticalOffsetKey: apertureVerticalOffset, kCVImageBufferCleanApertureWidthKey: eyeFrameSize.width, kCVImageBufferCleanApertureHeightKey: eyeFrameSize.height ]
  4. ѹॖϓϩύςΟͷઃఆ let MVHEVCVideoLayerIDs = [0, 1] let MVHEVCViewIDs = [0,

    1] let MVHEVCLeftAndRightViewIDs = [0, 1] ... let stereoCompressionProperties: [CFString: Any] = [ kVTCompressionPropertyKey_MVHEVCVideoLayerIDs: MVHEVCVideoLayerIDs, kVTCompressionPropertyKey_MVHEVCViewIDs: MVHEVCViewIDs, kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs: MVHEVCLeftAndRightViewIDs, kVTCompressionPropertyKey_HasLeftStereoEyeView: true, kVTCompressionPropertyKey_HasRightStereoEyeView: true ]
  5. ѹॖϓϩύςΟͷઃఆʢ2ʣ let projectionKind = projectedMediaMetadata.projectionKind ... compressionProperties[kVTCompressionPropertyKey_ProjectionKind] = kCMFormatDescriptionProjectionKind_HalfEquirectangular ...

    let baselineInMicrometers = UInt32(1000.0 * baselineInMillimeters) compressionProperties[kVTCompressionPropertyKey_StereoCameraBaseline] = baselineInMicrometers ... let encodedHorizontalFOV = UInt32(1000.0 * horizontalFOV) compressionProperties[kVTCompressionPropertyKey_HorizontalFieldOfView] = encodedHorizontalFOV
  6. Τϯίʔμʔͷઃఆ let outputSettings: [String: Any] = [ AVVideoCodecKey: AVVideoCodecType.hevc, AVVideoWidthKey:

    eyeFrameSize.width, AVVideoHeightKey: eyeFrameSize.height, AVVideoCompressionPropertiesKey: compressionProperties ]
  7. ॻ͖ग़͠ for (layerID, eye) in zip(MVHEVCVideoLayerIDs, eyes) { let pixelBuffer

    = try pixelBufferPool.makeMutablePixelBuffer() ... ੾ΓऔΓൣғࢉग़ॲཧʢলུʣ ... CVBufferSetAttachment(imageBuffer, kCVImageBufferCleanApertureKey, cropRectDict as CFDictionary, CVAttachmentMode.shouldPropagate) VTSessionSetProperty(session, key: kVTPixelTransferPropertyKey_ScalingMode, value: kVTScalingMode_CropSourceToCleanAperture) pixelBuffer.withUnsafeBuffer { cvPixelBuffer in guard VTPixelTransferSessionTransferImage(session, from: imageBuffer, to: cvPixelBuffer) == noErr else { fatalError("Error during pixel transfer session for layer \(layerID)") } } // Create and append a tagged buffer for this eye. let tags: [CMTag] = [.videoLayerID(Int64(layerID)), .stereoView(eye)] taggedBuffers.append(.init(tags: tags, content: .pixelBuffer(.init(pixelBuffer)))) }
  8. ॻ͖ग़͠ // Create and append a tagged buffer for this

    eye. let tags: [CMTag] = [.videoLayerID(Int64(layerID)), .stereoView(eye)] taggedBuffers.append(.init(tags: tags, content: .pixelBuffer(.init(pixelBuffer))))
  9. !

More Decks by Shingo Tamaki

Other Decks in Technology