Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AR without SceneKit

naru-jpn
February 07, 2018

AR without SceneKit

Create AR application using Metal without SceneKit

naru-jpn

February 07, 2018
Tweet

More Decks by naru-jpn

Other Decks in Programming

Transcript

  1. AR without SceneKit
    #mobilearmeetup @Timers 2018/2/7

    View full-size slide

  2. Relation of ARKit and SceneKit

    View full-size slide

  3. ARKit
    Camera
    SceneKit
    Pixel Buffer
    3D Info Rendering
    ARSCNView

    View full-size slide

  4. ARKit
    Camera
    SceneKit
    Pixel Buffer
    3D Info Rendering
    ARSCNView
    Light Estimation
    Plane Detection
    Manage 3D Objects

    View full-size slide

  5. Remove SceneKit and Use Metal

    View full-size slide

  6. ARKit
    Camera
    SceneKit
    Pixel Buffer
    3D Info Rendering
    ARSCNView
    Light Estimation
    Plane Detection
    Manage 3D Objects

    View full-size slide

  7. ARKit
    Camera
    Pixel Buffer
    3D Info
    Light Estimation
    Plane Detection

    View full-size slide

  8. Metal
    ARKit
    Camera
    Pixel Buffer
    3D Info
    Light Estimation
    Plane Detection
    Rendering

    View full-size slide

  9. Metal
    ARKit
    Camera
    Pixel Buffer
    3D Info Rendering
    Light Estimation
    Plane Detection
    Take Photo

    View full-size slide

  10. MetalAR
    https://github.com/naru-jpn/MetalAR

    View full-size slide

  11. Metal
    ARKit
    Camera
    Pixel Buffer
    3D Info
    Light Estimation
    Plane Detection
    Rendering
    Take Photo

    View full-size slide

  12. // Create ARSession
    session = ARSession()
    // ARSessionDelegate
    session.delegate = self

    View full-size slide

  13. Metal
    ARKit
    Camera
    Pixel Buffer
    3D Info
    Light Estimation
    Plane Detection
    Rendering
    Take Photo

    View full-size slide

  14. // Defined in ARSessionDelegate
    // Called when ARFrame is updated
    func session(_ session: ARSession,
    didUpdate frame: ARFrame) {
    // …
    }

    View full-size slide

  15. A running AR session continuously captures video frames
    from the device camera. For each frame, ARKit analyzes
    the image together with data from the device's motion
    sensing hardware to estimate the device's real-world
    position.
    ARFrame
    https://developer.apple.com/documentation/arkit/arframe

    View full-size slide

  16. Metal
    ARKit
    Camera
    Pixel Buffer
    3D Info
    Light Estimation
    Plane Detection
    Rendering
    Take Photo

    View full-size slide

  17. Pipeline of off-screen rendering
    Draw
    video frame
    Filter
    Draw
    Instances
    ARFrame MTLTexture

    View full-size slide

  18. Pipeline of off-screen rendering
    Filter
    Draw
    Instances
    ARFrame MTLTexture
    Draw
    video frame
    Commands sent to GPU

    View full-size slide

  19. Pipeline of off-screen rendering
    Filter
    Draw
    Instances
    ARFrame MTLTexture
    MPSImageSobel declared in Metal Performance Shaders
    Draw
    video frame
    Commands sent to GPU

    View full-size slide

  20. Performance
    on iPhone7

    View full-size slide

  21. Filter
    Draw
    Instances
    ARFrame MTLTexture
    Draw
    video frame
    3.3 ms
    Rendering

    View full-size slide

  22. Making Image to save
    drawHierarchy(in:afterScreenUpdates:)
    MTLTexture → CGImage
    7.0 ~ 10.0 ms
    available on background thread
    30.0 ~ 40.0 ms
    required to execute on main thread

    View full-size slide

  23. Drawing original 3D objects
    User Interaction (available hitTest(_:types:) for ARFrame)
    Physics engine
    Skeletal Animation

    View full-size slide

  24. Documents
    Metal Best Practices Guide
    https://developer.apple.com/library/content/documentation/
    3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#//
    apple_ref/doc/uid/TP40016642-CH27-SW1
    Metal Shading Language Specification
    https://developer.apple.com/metal/Metal-Shading-Language-
    Specification.pdf

    View full-size slide

  25. AR Template Project

    View full-size slide