google/mediapipe で始めるARアプリ開発/iOSDC2020

1. google/mediapipe Ͱ࢝ ΊΔARΞϓϦ։ൃ ϨΪϡϥʔτʔΫʢ20෼ʣ noppe 1
2. ର৅ • ARΞϓϦʹڵຯ͕͋Δ • τϥοΩϯάʹڵຯ͕͋Δ • ػցֶश෼͔Βͳͯ͘΋໰୊͋Γ·ͤΜɻ 2
3. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 3
6. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 6
7. mediapipe ͱ͸ʁ 7
8. mediapipe github.com/google/mediapipe • ॲཧ(Culculator)Λܨ͛ͯύΠϓϥΠϯʢGraphʣΛߏங͢Δ πʔϧ • ओʹը૾ॲཧͷύΠϓϥΠϯߏஙʹ࢖ΘΕΔ • MLॲཧΛࠞͥࠐΉࣄ͕ग़དྷΔ 8
11. mediapipe Culculator͸ɺ࣮૷ࡁΈͷ෺΋࢖͑Δ • LuminanceCalculator RGBΛड͚औΓɺً౓ը૾Λग़ྗ͢Δ • SobelEdgesCalculator SobelϑΟϧλΛ͔͚ͨը૾Λग़ྗ͢Δ ...etc 11
13. ࣮૷ࡁΈͷGraph ͜ΕΒͷ࣮૷͸͙͢ʹ࢖͑Δios ࠓ೔͸͜ΕΛ࢖͏ྲྀΕΛղઆ ios Ұ෦iOSͰ͸࣮૷͕ͳ͍΋ͷ΋͋Δ 13
14. ΞδΣϯμ • άϥϑͷϏϧυ • ARKitͱͷ࿈ܞ • Ͱ͖ΔࣄɾͰ͖ͳ͍ࣄ 14
15. άϥϑͷϏϧυ 15
16. github.com/noppefoxwolf/HandTracker ࠓճ࢖͏ιʔείʔυ 16
17. Graph • ߏஙͨ͠Graph͸όΠφϦͱͯ͠Ϗϧυ ͢Δ͜ͱ͕Ͱ͖Δ • όΠφϦΛObjC͔Β࢖͏࣮૷͕͋Δͷ ͰɺͦΕΒΛ࢖ͬͯϑϨʔϜϫʔΫΛ࡞ Δࣄ͕Ͱ͖Δɻ • ࠓ೔͸σϞͰ঺հͨ͠ϋϯυτϥοΧʔ ΛϑϨʔϜϫʔΫͱͯ͠Ϗϧυ 17
19. άϥϑͷϏϧυ ϏϧυπʔϧΛ࢖ͬͯϏϧυΛߦ͏ • bazelbuild/bazel (ϕΠθϧ) • Kubernetes΍TensorFlowͰ࠾༻ Track E - Ryo Aoyama BazelΛར༻ͨ͠Micro Modular Architecture 19
20. άϥϑͷϏϧυ bazelͷΠϯετʔϧ͸εΫϦϓτམͱ͖࣮ͯͯ͠ߦ͢Δ͚ͩ curl -LO "https://.../bazel-3.2.0-installer-darwin-x86_64.sh" chmod +x "bazel-3.2.0-installer-darwin-x86_64.sh" ./bazel-3.2.0-installer-darwin-x86_64.sh 20
21. άϥϑͷϏϧυ XcodeͰ΋Ϗϧυ͕Ͱ͖Δtulsi - bazelbuild/tulsi tulsi BuildPhaseScriptͰbazelΛୟ͚ͩ͘ͳͷͰɺඞਢͰ͸ͳ͍ 21
22. άϥϑͷϏϧυ tulsi͸ϏϧυεΫϦϓτୟ͚ͩ͘ͰΠϯετʔϧͰ͖Δʢཁbazelʣ git clone git@github.com:bazelbuild/tulsi.git ./build_and_run.sh 22
23. ͦͷ΄͔ඞཁͳ΋ͷ • Xcode • ࣌ؒ • ωοτϫʔΫ 23
24. ϏϧυͷྲྀΕ 1. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 2. BUILDϑΝΠϧʹ੒Ռ෺ͷ৘ใ΍ґଘੑͳͲΛهड़͢Δ 3. bazelͰbuild͢Δ 4. framework͕ग़དྷ্͕Δ 24
25. ϑϨʔϜϫʔΫͷίʔυ Λॻ͘ 25
26. ϑϨʔϜϫʔΫͷίʔυΛॻ͘ 1. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞Δ 2. MPPGraphΛ։࢝͢Δ 3. MPPGraphʹը૾ΛૹΔ 4. delegateͰ݁ՌΛड͚औΔ 26
28. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL ...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28
29. MPPGraphΛ։࢝͢Δ Graph಺Ͱར༻͢Δ֤छίϯϙʔωϯτͷॳظԽ͕ߦΘΕΔ - (void)start { [graph startWithError: nil]; } 29
30. MPPGraphʹը૾ΛૹΔ - (void)sendPixelBuffer:(CVPixelBufferRef)pixelBuffer { [self.mediapipeGraph sendPixelBuffer:pixelBuffer intoStream:@"input_video" packetType:MPPPacketTypePixelBuffer]; } 30
31. delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto& timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31
32. BUILDϑΝΠϧॻ͘ 32
33. BUILDϑΝΠϧͱ͸ bazelͰϏϧυ͢ΔϑΝΠϧ΍ґଘؔ܎Λهड़ͨ͠ϑΝΠϧ 33
34. objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"], data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34
36. bazelͰϏϧυ͢Δ $ bazel build mediapipe/iosdc:HandTracker 36
37. framework͕ग़དྷ্͕Δ githubʹஔ͍͓͖ͯ·ͨ͠ https://github.com/noppefoxwolf/HandTracker 37
38. ARKitͱͷ࿈ܞ 38
39. ARKitͱͷ࿈ܞ ARKitͷ࢓্༷ग़དྷͳ͍ࣄΛmediapipeͰิ͏͜ͱͰɺදݱྗΛղ ์Ͱ͖Δ • AR্ۭؒͷΦϒδΣΫτΛखͰૢ࡞͢Δ • ݕग़ͨ͠ΦϒδΣΫτͷԣʹΩϟϥΫλʔΛ഑ஔ͢Δ • إͷࠎ֨Λิਖ਼͠ͳ͕Β൅৭Λม͑Δhair hair Hair Segmentation͸ݱঢ়iOSඇରԠͰ͕͢ 39
40. ARKitͱͷ࿈ܞΞϓϦ։ൃ ͜Μͳ΋ͷΛ࡞Γ·͢ 40
42. ARKit࿈ܞͷྲྀΕ 1. ARFrame͔ΒcaptureImageΛऔΓग़͢ 2. PixelFormatΛม׵ 3. Tracker΁ૹΔ 4. DelegateͰ݁ՌΛड͚औΔ 5. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ 42
43. ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... // ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43
44. PixelFormatΛม׵ ը૾σʔλ͕ͲͷΑ͏ʹ֨ೲ͞Ε͍ͯΔ͔Λද͍ͯ͠Δ • ARKitͰ͸YCbCrܗࣜ • mediapipe͸BGRAͷΈΛड͚औΕΔ →ม׵ͷඞཁ͕͋Δ 44
45. kCVPixelFormatType_32BGRA 1ϐΫηϧ32bitͰදݱ͢ΔϑΥʔϚοτ B:FF G:00 R:00 A:FF → ੺ 45
46. kCVPixelFormatType_420YpCbCr8BiPl anarFullRange • 2ຕͷY(8bit)+CbCr(16bit)ͷ૊Έ߹ΘͤͰ1ϑϨʔϜΛදݱ͢Δ ϑΥʔϚοτ • ܰྔͳͷͰϏσΦσʔλͰ࢖ΘΕΔ͜ͱ͕ଟ͍ 46
47. YCbCr͔ΒBGRA΁ͷม׵ • Accelerate • vImage • Metal • MSL GPUΛ࢖͏෼Metalͷํ͕ߴ଎ 47
48. YCbCr͔ΒBGRA΁ͷม׵ ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ R = Y + 1.402 × Cr G = Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48
49. BlueDress YCbCr͔ΒBGRAʹߴ଎ίϯόʔτ͢ΔϥΠϒϥϦ https://github.com/noppefoxwolf/BlueDress 49
50. Tracker΁ૹΔ let captureImage = frame.capturedImage let bgraCaptureImage = converter.convertToBGRA(captureImage) handTracker.send(bgraCaptureImage) 50
51. DelegateͰ݁ՌΛड͚औΔ ؔઅͷҐஔ͸X,Y,ZͰऔΕ·͢…͕ɺੈք࠲ඪܥͰ͸ແ͘ X,Y: Screen࠲ඪܥ Z: खट͔ΒͷԞߦ͖ ͔͠औΕ·ͤΜɻ࢒೦ʂ 51
53. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!) { let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53
54. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ private func press() { self.scene.notifications.press.post() count += 1 changeText("\(count)") } 54
57. Ͱ͖ΔࣄͰ͖ͳ͍ࣄ 57
58. ؾʹͳΔͱ͜Ζ • HandTracker͸Χϝϥ͔Βͷڑ཭͕औΕΔΘ͚Ͱ͸ͳ͍ࣄʹ஫ҙ →IrisͰ͸Ԟߦ͖͕औΕΔ • ֤ؔઅͷճస΋औΕͳ͍ →ࣗ෼Ͱܭࢉͯࣗ͠વʹݟͤΔ޻෉͕ඞཁ • ϑϨʔϜϫʔΫαΠζ͕·͊·͊Ͱ͔͍ →100MB͘Β͍ 58
59. ͍͍ͱ͜Ζ • ࣮૷ࡁΈͷάϥϑ͕ඇৗʹ࣭͕ߴ͍ • OSS • ϚϧνϓϥοτϑΥʔϜ 59
60. ΍ͬͯΈ͍ͨ͜ͱ • HairSegmentationͳͲΛiOSʹରԠ͍ͯ͠ͳ͍ͷ͸Կނͳͷ͔ௐ ΂͍ͨ • NSObjectΛܦ༝͠ͳ͚Ε͹ߴ଎Խ͕๬ΊΔͷͰ͸ • ϥϯυϚʔΫ͚ͩΛग़ྗ͢ΔάϥϑΛ࡞ͬͯΈ͍ͨ 60
61. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝ ·ͨ͠ 61