Metal for Mortals

Presentation on Apple's Metal 3D Graphics API for the try! Swift New York Chapter Meetup.

1. ME TAL f or M ORTAL S Douglass Turner douglass.turner@gmail.com @dugla 781 775 3708
2. So, what’s so special about 3D …
3. Aim the camera
4. Elastic Image Software LLC Copyright Steve Jubinville Select and pose the model
5. Elastic Image Software LLC Copyright © 2006 Florian Wild Design the appearance. Light the scene.
6. How do we get there …?
7. Easy. SceneKit.
8. SceneKit is a high-level API that makes it easy to build 3D graphic apps.
12. Apple provides an entire game demo in SceneKit Fox: Building a SceneKit Game with the Xcode Scene Editor
14. … but we like things hard. We like Metal.
15. Apple provides a set of 3D frameworks iOS/MacOS
16. Metal/MetalKit SceneKit GLKit Model I/O
17. Metal is our focus GLKit Matrix & Vector Library Model I/O Meshes. Model API SceneKit Model design and layout
18. Metal GLKit Model I/O SceneKit Elastic Image API I blend the essential bits of GLKit, Model I/O and SceneKit with Metal
19. Metal is a low level API. It is all about feeding the GPU.
20. Elastic Image Software LLC Think of the GPU as a dragster
21. Elastic Image Software LLC It does one thing insanely well
22. The GPU is a SIMD computing beast
23. SnapChat and Instagram are GPU apps
24. The GPU is the workhorse of modern Artificial Intelligence.
25. HelloMetal is a handful of demos each highlighting one aspect of Metal. https://github.com/turner/HelloMetal
26. A single-view app with an attached 3D manipulator called an ArcBall that responds to pan gestures. The View delegates to a Renderer where the magic happens. The Demos
27. The Renderer draws a 3D object on the screen. Lets start simple with a quadrilateral. The Demos
28. 3D shapes are defined by vertices. Each is a point in 3-space.
29. We attach a set of attributes to each vertex. • position: x, y, z • surface normal: nx, ny, nz, • texture coordinate: s, t • color: r, g, b
30. • x, y, z • vertex location in 3-space • nx, ny, nz • surface normal • s, t • texture coordinate • r, g, b • vertex color
31. EIQuad
32. surface normals
33. s,t define a coordinate system on the surface of a 3D object s t
34. x, y, z s, t A simple quad
35. x, y, z s, t Something more interesting
36. All HelloMetal demos use textured objects. For even simple shapes, textures are an easy way to create visual detail.
37. HELLO
38. HELLO CAMERAPLANE
39. HELLO RENDERPASS
40. HELLO MODEL I/O
41. HELLO SCENEKIT
42. HELLO LIGHT
43. HELLO
44. EIView
45. Renderer
46. Renderer
47. Renderer
48. Renderer
49. Renderer
50. A shaders defines the look of a shape Shaders come in pairs • vertex shader • fragment shader
51. textureVertexShader textureFragmentShader textureVertexShader textureVertexShader
52. texture.metal
53. Respond to View size changes Change viewport aspect-ratio in response to view size (aspect-ratio) change Rebuild all transformation matrices in response to ArcBall manipulation Renderer
54. EIArcball is my implementation of the classic Ken Shoemake Arcball paper. Arcball uses Quaternions (4-dimensional vectors) to enable an intuitive mapping between pan gestures and 3D rotations. (See: SceneKit modeler for an example) EIArcball
55. Renderer
56. Our fragment shader named textureFragmentShader pulls “samples” from the Metal texture to use as the color source. Create a Metal texture from an image named kids_grid_3x3 in our asset catalog
57. demo
58. HELLO CAMERAPLANE
59. Position a texture on the far plane of the camera to act as a backdrop. Hello CameraPlane
60. Use a translucent texture Hello CameraPlane
61. demo
62. HELLO RENDERPASS
63. The idea: Rather then render directly to the screen in one go, render separate passes and combine them in a final pass. The Render Pass Hello RenderPass
64. The Render Pass Break a large problem into smaller more tractable, decoupled, pieces. Turn a 3D task into a 2D task. Easier to tweak. “Bake” computationally tasks into inexpensive texture lookups. Render like the pros. Hello RenderPass
65. Scene from Avatar
66. first pass: Render using a Metal texture we create as the destination. Render results are cached in the texture. final pass: Use the texture from the first pass to texture- map a quad aligned to the camera. Hello RenderPass The Game Plan
67. Render Pass Renderer Create a render pass descriptor to configure a render pass object.
68. Render Pass Renderer Create color texture and depth texture to render into.
69. Render Pass Renderer Create final pass quad and shader for final pass render
70. Render Pass Renderer Draw the first pass
71. Render Pass Renderer We now have the results of the first pass render stored in the texture of the render pass descriptor we created
72. Render Pass Renderer Draw the final pass
73. finalpass.metal
74. finalpass.metal
75. finalpass.metal
76. This is just the tip of the iceberg… The render pass approach is the key to insane visual f/x and hyper real rendering.
80. HELLO MODEL I/O
81. Model I/O is a Apple framework with a rich set of features for handling and processing 3D models. HelloMetal uses it only for its shape library: • PLANE • SPHERE • CUBE See WWDC 2015 Video: Managing 3D Assets with Model I/O
83. demo
84. HELLO SCENEKIT
85. The SceneKit framework enables you to build highly sophisticated 3D apps and games. I provided support for access to the SceneKit modeler and importing them via the SceneKit - Model I/O bridging APIs.
86. SceneKit modeler demo
87. With HelloSceneKit and HelloLight you can go grab a 3D model from the Internet - I like 3DOcean - and play with it.
88. demo
89. With SceneKit model import we have far more sophisticated models to play with. Lets take a closer look at how texture maps and and the s-t texture coordinates interact.
90. Visualization of s-t coordinates
91. HELLO LIGHT
92. So, what has been missing in these little demos? Lights! Specifically, shading that simulates the interaction between a light source and a 3D surface.
93. Recall the surface normals attached to each vertex. They are the fundamental surface attribute responsible for creating a shaded (lit) look.
94. Lets take a closer look at surface normals.
95. Visualization of surface normals
96. Notice the visual detail that immediately becomes apparent just by visualizing the surface normals.
97. One equation to rule them all. The fundamental governing equation for basic illumination. Cosine is everywhere in computer graphics.
98. Lets look at the what happens when we rotate the surface normals.
99. Visualization eye space transformed surface normals
100. demo
101. T HANK Y OU Douglass Turner douglass.turner@gmail.com @dugla 781 775 3708