Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AR without SceneKit
Search
naru-jpn
February 07, 2018
Programming
0
420
AR without SceneKit
Create AR application using Metal without SceneKit
naru-jpn
February 07, 2018
Tweet
Share
More Decks by naru-jpn
See All by naru-jpn
配信アプリのためのリアルタイムプッシュ通知ぼかしの夢
narujpn
3
950
PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar
narujpn
3
4.4k
Updating an App to Use Swift Concurrency 解説
narujpn
2
350
PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar
narujpn
0
1.2k
App Extension のスタックトレース情報からクラッシュを解析/集計する / Analyzing app extension's stack trace
narujpn
3
1.6k
ミラティブとWebRTC - WebRTC framework の中身を覗いてみよう / WebRTC framework AudioUnit Processing
narujpn
1
2.2k
CoreML3のオンデバイストレーニングでつくる母音推定
narujpn
0
450
AltConfと周辺の歩き方
narujpn
0
2k
エンジニア経験を活かしたスクラムマスターとして 開発チームとプロダクトを成長させる
narujpn
1
420
Other Decks in Programming
See All in Programming
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
ioquatix
4
2.2k
kiroとCodexで最高のSpec駆動開発を!!数時間で web3ネイティブなミニゲームを作ってみたよ!
mashharuki
0
680
コードとあなたと私の距離 / The Distance Between Code, You, and I
hiro_y
0
170
Cursorハンズオン実践!
eltociear
2
1.1k
Introduce Hono CLI
yusukebe
6
2.8k
私はどうやって技術力を上げたのか
yusukebe
44
19k
Go言語の特性を活かした公式MCP SDKの設計
hond0413
1
240
その面倒な作業、「Dart」にやらせませんか? Flutter開発者のための業務効率化
yordgenome03
1
130
フロントエンド開発のためのブラウザ組み込みAI入門
masashi
5
1.8k
Goで実践するドメイン駆動開発 AIと歩み始めた新規プロダクト開発の現在地
imkaoru
4
850
Claude CodeによるAI駆動開発の実践 〜そこから見えてきたこれからのプログラミング〜
iriikeita
0
290
CSC305 Lecture 04
javiergs
PRO
0
270
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
10
600
Raft: Consensus for Rubyists
vanstee
140
7.1k
How GitHub (no longer) Works
holman
315
140k
Scaling GitHub
holman
463
140k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
Side Projects
sachag
455
43k
What's in a price? How to price your products and services
michaelherold
246
12k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Transcript
AR without SceneKit #mobilearmeetup @Timers 2018/2/7
Relation of ARKit and SceneKit
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
Remove SceneKit and Use Metal
ARKit Camera SceneKit Pixel Buffer 3D Info Rendering ARSCNView Light
Estimation Plane Detection Manage 3D Objects
ARKit Camera Pixel Buffer 3D Info Light Estimation Plane Detection
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering
Metal ARKit Camera Pixel Buffer 3D Info Rendering Light Estimation
Plane Detection Take Photo
MetalAR https://github.com/naru-jpn/MetalAR
None
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Create ARSession session = ARSession() // ARSessionDelegate session.delegate =
self
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
// Defined in ARSessionDelegate // Called when ARFrame is updated
func session(_ session: ARSession, didUpdate frame: ARFrame) { // … }
A running AR session continuously captures video frames from the
device camera. For each frame, ARKit analyzes the image together with data from the device's motion sensing hardware to estimate the device's real-world position. ARFrame https://developer.apple.com/documentation/arkit/arframe
Metal ARKit Camera Pixel Buffer 3D Info Light Estimation Plane
Detection Rendering Take Photo
Pipeline of off-screen rendering Draw video frame Filter Draw Instances
ARFrame MTLTexture
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture Draw
video frame Commands sent to GPU
Pipeline of off-screen rendering Filter Draw Instances ARFrame MTLTexture MPSImageSobel
declared in Metal Performance Shaders Draw video frame Commands sent to GPU
Performance on iPhone7
Filter Draw Instances ARFrame MTLTexture Draw video frame 3.3 ms
Rendering
Making Image to save drawHierarchy(in:afterScreenUpdates:) MTLTexture → CGImage 7.0 ~
10.0 ms available on background thread 30.0 ~ 40.0 ms required to execute on main thread
Challenges
Drawing original 3D objects User Interaction (available hitTest(_:types:) for ARFrame)
Physics engine Skeletal Animation …
References
Documents Metal Best Practices Guide https://developer.apple.com/library/content/documentation/ 3DDrawing/Conceptual/MTLBestPracticesGuide/index.html#// apple_ref/doc/uid/TP40016642-CH27-SW1 Metal Shading
Language Specification https://developer.apple.com/metal/Metal-Shading-Language- Specification.pdf
AR Template Project