Mobile Dev Japan #3 - Safie Viewer Android & Sceneview

Talk about how we implemented Sceneview in Safie Viewer at Mobile Dev Japan in Tokyo

Transcript

1. July 26, 2024 CHA　Jerome Safie Inc. Development Division 2 - Mobile Team Safie Viewer Android & Sceneview
2. © Safie Inc.｜ Self-Introduction - Jerome CHA, French - Joined Safie in November 2023 as Android Engineer - Previously worked at Kubell (ex Chatwork) and Enigmo (BUYMA) - Hobbies : Workout (ex-JBBF Bodybuilder) / Making my own robot - Twitter @yujiro45 2
3. © Safie Inc.｜ Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 3
4. © Safie Inc.｜ Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 4
5. © Safie Inc.｜ Sceneview Recap - https://sceneview.github.io/ - Library that uses Google’s ARCore and Filament to display 3D / AR contents - Included 2 views : Sceneview & ARSceneview - My previous presentation @ Mobile Dev Japan #1 https://speakerdeck.com/jeromecha/ar-with-sceneview-on-android 5 ARSceneview Sceneview
6. © Safie Inc.｜ Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 6
7. © Safie Inc.｜ Safie Viewer - Cloud service for security cameras - Clear images, easy to setup, easy to use - Videos are in HD 30fps 7
8. © Safie Inc.｜ Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 8
9. © Safie Inc.｜ Safie’s new camera - Safie GO 360 - Launched in February 1st 2024 - Compact and lightweight, easy to transport - Built-in LTE for cloud recording anytime, anywhere - Horizontal and vertical viewing angle of 182° 9 https://safie.co.jp/news/3033/
10. © Safie Inc.｜ Web Demo 10 speed x3
11. © Safie Inc.｜ 3D model - 2 installations 11 Wall Ceiling
12. © Safie Inc.｜ 3D model on Android 12 - 3D Model created via Blender - iOS : .dae format → convert to .scn format with Xcode - Android : .glb format
13. © Safie Inc.｜ 3D model on Android 13
14. © Safie Inc.｜ Texture Mapping 14 How can we map the camera’s stream on the texture of the 3D model
15. © Safie Inc.｜ Texture Mapping 15 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt
16. © Safie Inc.｜ Texture Mapping 16 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt It is a PlaneNode, so we cannot use it
17. © Safie Inc.｜ Texture Mapping 17 https://github.com/SceneView/sceneview-android/blob/1e303adf92f1f7b67eb9aa4 f9fc731d75213ac97/samples/ar-augmented-image/src/main/java/io/github/scenevi ew/sample/araugmentedimage/video/ExoPlayerNode.kt We need VideoMaterial & ExoPlayer
18. © Safie Inc.｜ Texture Mapping 18 We need to make our own VideoMaterial with ExoPlayer!
19. © Safie Inc.｜ Texture Mapping 19 VideoMaterial ExoPlayer
20. © Safie Inc.｜ Texture Mapping 20 ExoPlayerVideoMaterial!
21. © Safie Inc.｜ Texture Mapping 21 Engine : Google Filament’s engine Exoplayer : for the video’s stream materialLoader : for the texture’s creation
22. © Safie Inc.｜ Texture Mapping 22 Create a SurfaceTexture
23. © Safie Inc.｜ Texture Mapping 23 Create a Stream from Filament based on the surfaceTexture
24. © Safie Inc.｜ Texture Mapping 24 Create a Sceneview Texture for our modelNode based on Filament’s Stream
25. © Safie Inc.｜ Texture Mapping 25 Create a VideoInstance for our modelNode later
26. © Safie Inc.｜ Texture Mapping 26 Setup our ExoPlayer
27. © Safie Inc.｜ Texture Mapping 27 Use our own CustomVideoMaterial
28. © Safie Inc.｜ Texture Mapping 28
29. © Safie Inc.｜ User’s interaction 29 How can we limit the 3D model rotation?
30. © Safie Inc.｜ User’s interaction 30 To limit the 3D model rotation : - Remove the default’s user interaction - Develop your own behavior from scratch
31. © Safie Inc.｜ User’s interaction 31 To remove the default behavior : - Create a subclass of SceneView - Setup cameraManipulator to null
32. © Safie Inc.｜ User’s interaction - Drag (Wall position) 32
33. © Safie Inc.｜ User’s interaction - Drag (Ceiling position) 33
34. © Safie Inc.｜ User’s interaction - Zoom 34
35. © Safie Inc.｜ User’s interaction - Zoom 35 Zoom in a specific point on the 3D model - Pinch in/out : Take the middle point between the two fingers and zoom to that position - Double Tap : Double tap to zoom in/out - If tap outside the 3D model, then zoom in the middle
36. © Safie Inc.｜ User’s interaction - Zoom 36
37. © Safie Inc.｜ User’s interaction - Zoom 37 Screen
38. © Safie Inc.｜ User’s interaction - Zoom 38 Screen
39. © Safie Inc.｜ User’s interaction - Zoom 39 Screen
40. © Safie Inc.｜ User’s interaction - Zoom 40 Screen This is called a HitResult
41. © Safie Inc.｜ User’s interaction - Zoom 41
42. © Safie Inc.｜ Agenda 1. Sceneview Recap 2. Safie Viewer 3. Our implementation of Sceneview 4. Problems 42
43. © Safie Inc.｜ Problems Problem 1 3D Model hit box 43
44. © Safie Inc.｜ Problem 1 : 3D Model hit box 44 Tapping outside is considered as the edge of the 3D model (but only once)
45. © Safie Inc.｜ Problem 1 : 3D Model hit box 45 Swiping outside also triggered the 3D Model to move
46. © Safie Inc.｜ Problem 1 : 3D Model hit box 46 https://github.com/SceneView/sceneview-android/issues/522
47. © Safie Inc.｜ Problem 1 : 3D Model hit box 47 Resolved by putting a PlaneNode on the top of the 3D model to intercept the hit : Diameter of hemisphere *assuming that the center of the hemisphere is at (0,0) *
48. © Safie Inc.｜ Problem 1 : 3D Model hit box 48
49. © Safie Inc.｜ Problems 2 Problem 2 ExoPlayer on multiple surfaces 49
50. © Safie Inc.｜ Problems 2 : ExoPlayer on multiple surfaces 50 Switching Fisheye ⟷ 360° view with a button
51. © Safie Inc.｜ Problems 2 : ExoPlayer on multiple surfaces 51 https://github.com/google/ExoPlayer/issues/6939
52. © Safie Inc.｜ Problems 2 : ExoPlayer on multiple surfaces 52 The only solution was to reinit ExoPlayer each time you press the button
53. © Safie Inc.｜ We are hiring! More info : https://open.talentio.com/r/1/c/safie/pages/84134 53 We are looking for Android / iOS engineers!