Introduce technical issue and solution of various genre games for Apple Vision Pro

Transcript

1. Introduce technical issue and solution of various genre games for Apple Vision Pro visionOS TC 2025
2. Self introduction Cova from Graf fi ty inc.
 Job: Unity Engineer Author and Reviewer 2 @8bitdots https://qiita.com/Cova8bitdot @cova8bit.bsky.social
3. Graffity makes the entertainment with AR technology 3 Overview Gra ffi ty has consistently delivered spatial entertainment. • Released two smartphone AR titles and two spatial games for Apple Vision Pro. • Most recently, we released the Apple Vision Pro spatial game “Shuriken Survivor.” • We operate Gra ffi ty AR Studio, which supports new B2C businesses utilizing AR, with numerous achievements collaborating with major clients. • Planned and developed the smartphone AR game “LOST ANIMAL PLANET” with NTT QONOQ. • We also run Gra ff i ty Spatial Lab, which supports planning and development of Apple Vision Pro applications. • Planned and developed the spatial video experience “Ploom Dive” for Ploom Shop, operated under the leadership of JT. • In addition, we have developed over 10 spatial applications to date. Spatial Game 「Gra ff i ty AR Studio」 「Gra ff i ty Spatial Lab」
4. Gra ffi ty has already published serveral game apps for Apple Vision Pro 4 Craftrium Ninja Gaze Typing Shuriken Survivor
5. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
6. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
7. Ninja Gaze Typing was developed for the U.S. launch of Apple Vision Pro in February 2024. 7 Case: Ninja Gaze Typing
8. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
9. Ninja Gaze Typing was developed for the U.S. launch of Apple Vision Pro in February 2024. Since we considered gaze to be essential, the prototype was built using the Meta Quest Pro, which supports eye- tracking. 9 Considerations for Developing Apps on Unreleased Devices 2023 Jun Apple Vision Pro announced at WWDC ’23 2023 Sep • Created a prototype using Meta Quest Pro • Verified the UX of gaze interaction 2023 Oct Began development for AVP, continuing to use Unity 2024 Feb • AVP released in the U.S. • Released Ninja Gaze Typing (Unity version)
10. Developing for a new device is extremely challenging in many ways. Issues encountered during development • No physical device available / device-speci fi c errors / works only in Simulation Mode • Try every possible way to obtain a device • Contact both Unity and Apple • Di ff i cult to determine whether something is a bug or an intended speci fi cation • Contact both Unity and Apple • No prior examples to reference • Keep pushing forward • Don’t give up • Research in English (There is little Japanese information) 10 Considerations for Developing Apps on Unreleased Devices
11. 11 It was featured in the Apple Vision Pro keynote at WWDC 2024 as one of the released apps. Here！ Considerations for Developing Apps on Unreleased Devices
12. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
13. Shuriken Survivor which is released same time on Japan launch of Apple Vision Pro is Rogue-like tower defence shooter. 13 Case: Shuriken Survivor
14. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
15. There are some differences of definition between XRHand and visionOS 15 Best Practices for Integrating Hand Tracking into Games XRHand visionOS • XRHand has palm bone, but visionOS doesn’t have it • XRHand does NOT have de fi nition of neither 25 .handForearmWrist nor 26 .handForearmArm • There is extended de fi tinio on VisionOSHand.JointID from PolySpatial SDK. • There is no palm bone. • Developers have to estimate normal vector of palm by information of other bones
16. 16 Patterns that are easy to recognize Speeds that are easy to recognize Patterns that are di ff i cult to recognize • pose where the tracked points are not occluded from the sensors as much as possible • For a shuriken-throwing gesture: around 60 cm/s • Poses where many tracked points are occluded from the camera’s perspective • Even the same pose may fail depending on the angle • Unusual fi nger movements such as twisting • Extremely slow or extremely fast motions • A typical hand-waving gesture is actually very fast When implementing hand tracking and hand gestures, various considerations are required to maintain high estimation accuracy. Best Practices for Integrating Hand Tracking into Games
17. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
18. 18 Original Mesh • Each captured mesh is color-coded • Mesh information can be obtained for each segmented region • Materials with strong re fl ectivity may take longer to detect Apple Vision Pro can capture the real-world mesh at high precision and divide it into multiple objects. Real-Time Integration of Spatial Mesh and Pathfinding Systems • WxHxD = 3.5m x 2.5m x 4.8m
19. In game development (Unity), Navigation Meshes are commonly used for pathfinding. • A lightweight NavMesh for path fi nding can be baked from the actual mesh • As shown on the right, baking generates mesh data that represents the walkable area (light blue region) • Agents then move within this walkable area using shortest-path algorithms 19 Feature Source : https://docs.unity3d.com/ja/2019.4/Manual/nav- NavigationSystem.html Real-Time Integration of Spatial Mesh and Pathfinding Systems
20. By rebaking the NavMesh at regular intervals and using plane detection, it is possible to build a system where agents move while recognizing the real-world environment as shown below. 20 Collision with Spatial Mesh Obstacle detection and avoidance using a combination of plane Source : https://www.gizmodo.jp/2024/07/shuriken-survivor.html Real-Time Integration of Spatial Mesh and Pathfinding Systems
21. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
22. Development with Unity： use Unity Profiler Actual performance on Apple Vision Pro：use RealityKit Trace 22 Unity Pro fi ler RealityKit Trace • Performance check CPU, RAM, Rendering cost such as UI • Actual performance on the device. • Check CPU/GPU time • Number of rendered polygons • Number of hangs Performance Profiling and Optimization Techniques
23. Developers need to investigate five major areas. 1. RealityKit Frames 2. RealityKit Metrics 3. Run Loops 4. Time Pro fi ler 5. Hangs 23 https://qiita.com/Cova8bitdot/items/eb7ad7ab6706bb4e8029 Performance Profiling and Optimization Techniques
24. • The process of calculating how a mesh should deform according to bone movement • The way it moves is defined by weight painting 24 What’s skinning? Apple Vision Pro performs poorly with skinning operations. Because of this, using a large amount of skeletal animation makes it difficult to maintain 90 FPS. Performance Profiling and Optimization Techniques
25. 1. Reduce number of targets which requires skinning. 2. Reduce number of polygons. 3. Consider to use another expression such as Vertex Animation Texture. 25 Solution Performance Profiling and Optimization Techniques Apple Vision Pro performs poorly with skinning operations. Because of this, using a large amount of skeletal animation makes it difficult to maintain 90 FPS.
26. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
27. Craftrium is an aquarium simulator game developed with Shochiku. 27 Case: Craftrium
28. 28 Craftrium is an aquarium simulator game developed with Shochiku. Case: Craftrium
29. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
30. If we place objects in space or let fi sh swim freely without any constraints, visual issues tend to occur. 30 Guidelines for Placing Objects in 3D Space Bad pattern① Bad pattern② • Objects and fi sh may intersect or clip into each other • They may move outside the drawable area • Fish may appear partially cut o ff • Objects may be clipped visually
31. Spatial grid can solve servaral problems. 31 Guidelines for Placing Objects in 3D Space Place on spatial grid East to control and implement of the Hand gesture! • Prevents objects from being placed outside the drawable area • Snaps to a grid, reducing positional shifts caused by small hand jitters • The light blue dots in the fi gure on the right represent the grid points Move Rotate Zoom
32. Agenda 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
33. 33 Path fi nding algorithm: A* • Concepts commonly introduced in graph theory textbooks • Assumes planar graphs for simplicity of explanation NavMesh requires the Mesh, but We cannot place full of meshes on volume space. So we use a path fi nding algorithum with the Spatial-Grid. Path Movement Algorithms for Allowing Fish to Swim Freely in Space Source :https://www.nc.ii.konan-u.ac.jp/projects/algo/page/shortestpath/
34. 34 Path fi nding algorithm: A* Extention for 3D • Concepts commonly introduced in graph theory textbooks • Assumes planar graphs for simplicity of explanation A version with more visual complexity and a larger number of connected nodes • The number of adjacent nodes increases significantly ◦ Simple up / down / left / right / front / back → 6 nodes ◦ Including diagonals → 9 + 8 + 9 = 26 nodes • Exploration cost increases dramatically Fundamentally the same as graphs on a plane NavMesh requires the Mesh, but We cannot place full of meshes on volume space. So we use a path fi nding algorithm with the Spatial-Grid. Path Movement Algorithms for Allowing Fish to Swim Freely in Space Source :https://www.nc.ii.konan-u.ac.jp/projects/algo/page/shortestpath/
35. • Became able to explicitly de fi ne passable grids • Light blue point →OK • Red point→NG • With completely random selection, the fi sh often showed strange behavior such as 180° turns or sudden vertical movements. • By having the fi sh reference only the grids within its forward hemisphere, these unnatural turns were signi fi cantly reduced. 35 Path Movement Algorithms for Allowing Fish to Swim Freely in Space NavMesh requires the Mesh, but We cannot place full of meshes on volume space. So we use a path fi nding algorithum with the Spatial-Grid.
36. Out R&D team enabled boid simulation with Spatial Grid and A* algorithum 36 Path Movement Algorithms for Allowing Fish to Swim Freely in Space
37. Conculusion 1. Case: Ninja Gaze Typing A. Considerations for Developing Apps on Unreleased Devices 2. Case: Shuriken Survivor A. Best Practices for Integrating Hand Tracking into Games B. Real-Time Integration of Spatial Mesh and Path fi nding Systems C. Performance Pro fi ling and Optimization Techniques 3. Case: Craftrium A. Guidelines for Placing Objects in 3D Space B. Path Movement Algorithms for Allowing Fish to Swim Freely in Space
38. Thank you for listening