Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AR without SceneKit
Search
naru-jpn
February 07, 2018
Programming
0
390
AR without SceneKit
Create AR application using Metal without SceneKit
naru-jpn
February 07, 2018
Tweet
Share
More Decks by naru-jpn
See All by naru-jpn
配信アプリのためのリアルタイムプッシュ通知ぼかしの夢
narujpn
3
710
PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar
narujpn
3
3.4k
Updating an App to Use Swift Concurrency 解説
narujpn
2
230
PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar
narujpn
0
930
App Extension のスタックトレース情報からクラッシュを解析/集計する / Analyzing app extension's stack trace
narujpn
3
1.2k
ミラティブとWebRTC - WebRTC framework の中身を覗いてみよう / WebRTC framework AudioUnit Processing
narujpn
1
2k
CoreML3のオンデバイストレーニングでつくる母音推定
narujpn
0
380
AltConfと周辺の歩き方
narujpn
0
1.9k
エンジニア経験を活かしたスクラムマスターとして 開発チームとプロダクトを成長させる
narujpn
1
360
Other Decks in Programming
See All in Programming
ECMAScript、Web標準の型はどう管理されているか / How ECMAScript and Web standards types are maintained
petamoriken
3
370
最高の構文木の設計 2024年版
yui_knk
7
2.4k
1人で挑むSwiftコンパイラ 〜型システム入門編〜
s_shimotori
0
310
LangChainでWebサイトの内容取得やGitHubソースコード取得
shukob
0
120
【TID2024】模擬講義:プログラマと一緒にゲームをデザインしてみよう!
akatsukigames_tech
0
380
Swiftコードバトル必勝法
toshi0383
0
140
null or undefined
susisu
20
4k
Meet BrowserEngineKit
swiftty
0
190
実践 Advanced CallKit 〜快適な通話の実現に向けて〜
mot_techtalk
3
110
めざせ!WKWebViewマスター! / WKWebView Master
marcy731
3
480
iOSDC 2024 SMBファイル共有をSwiftで実装する
kishikawakatsumi
1
110
Playwrightから始めるVisual Regression Testingのススメ by とっと
totto2727
2
1.8k
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
28
2.2k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Fashionably flexible responsive web design (full day workshop)
malarkey
400
65k
Build your cross-platform service in a week with App Engine
jlugia
228
18k
Speed Design
sergeychernyshev
18
400
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
122
18k
GraphQLとの向き合い方2022年版
quramy
43
13k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
109
6.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
22
1.7k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Infographics Made Easy
chrislema
239
18k
Transcript
AR without SceneKit #mobilearmeetup @Timers 2018/2/7
Relation of ARKit and SceneKit
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
Remove SceneKit and Use Metal
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering
Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation
Plane Detection Take Photo
MetalAR https://github.com/naru-jpn/MetalAR
None
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Create ARSession session = ARSession() // ARSessionDelegate session.delegate =
self
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Defined in ARSessionDelegate // Called when ARFrame is updated
func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
A running AR session continuously captures video frames from the
device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
Pipeline of off-screen rendering Draw video frame Filter Draw Instances
ARFrame MTLTexture
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw
video frame Commands sent to GPU
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel
declared in Metal Performance Shaders Draw video frame Commands sent to GPU
Performance on iPhone7
Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms
Rendering
Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~
10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
Challenges
Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)
Physics engine Skeletal Animation …
References
Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading
Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
AR Template Project