AR without SceneKit

Create AR application using Metal without SceneKit

Transcript

1. AR without SceneKit #mobilearmeetup @Timers 2018/2/7
2. Relation of ARKit and SceneKit
3. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView
4. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light Estimation Plane Detection Manage 3D Objects
5. Remove SceneKit and Use Metal
6. ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light Estimation Plane Detection Manage 3D Objects
7. ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection
8. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection Rendering
9. Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation Plane Detection Take Photo
10. MetalAR https://github.com/naru-jpn/MetalAR
12. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection Rendering Take Photo
13. // Create ARSession session = ARSession() // ARSessionDelegate session.delegate = self
14. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection Rendering Take Photo
15. // Defined in ARSessionDelegate // Called when ARFrame is updated func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
16. A running AR session continuously captures video frames from the device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
17. Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection Rendering Take Photo
18. Pipeline of off-screen rendering Draw video frame Filter Draw Instances ARFrame MTLTexture
19. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw video frame Commands sent to GPU
20. Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel declared in Metal Performance Shaders Draw video frame Commands sent to GPU
21. Performance on iPhone7
22. Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms Rendering
23. Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~ 10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
24. Challenges
25. Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame) Physics engine Skeletal Animation …
26. References
27. Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
28. AR Template Project