Upgrade to Pro — share decks privately, control downloads, hide ads and more …

google/mediapipe で始めるARアプリ開発/iOSDC2020

noppefoxwolf
September 21, 2020

google/mediapipe で始めるARアプリ開発/iOSDC2020

noppefoxwolf

September 21, 2020
Tweet

More Decks by noppefoxwolf

Other Decks in Programming

Transcript

  1. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

    ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 3
  2. 4

  3. 5

  4. noppe • גࣜձࣾσΟʔɾΤψɾΤʔ • ιʔγϟϧϥΠϒΞϓϦ Pococha • ݸਓ։ൃऀ • vear

    ReplayKitͰεϚϗ͔Β഑৴Ͱ͖Δ VTuberΞϓϦ • iOSDC18,19 • ͖ͭͶ͕޷͖ 6
  5. 9

  6. 10

  7. 12

  8. 18

  9. 27

  10. binarypbΛಡΈࠐΜͰMPPGraphΛ࡞ Δ + (instanceType) init { NSURL* url = [NSURL

    ...@"hand_landmarks.binarypb"]; NSData* data = [NSData dataWithContentsOfURL:url options:0 error:nil]; mediapipe::CalculatorGraphConfig config; config.ParseFromArray(data.bytes, data.length); self.graph = [[MPPGraph alloc] initWithGraphConfig:config]; } 28
  11. delegateͰ݁ՌΛड͚औΔ - (void)mediapipeGraph:(MPPGraph*)graph didOutputPacket:(const ::mediapipe::Packet&)packet fromStream:(const std::string&)streamName { const auto&

    timestamp = packet.Timestamp().Value(); const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>(); NSArray *landmarkObjects = [self toObject: landmarks]; [delegate didOutputLandmarks: landmarkObjects]; } 31
  12. objc_library( name = "HandTrackerLibrary", hdrs = ["HandTracker.h"], srcs = ["HandTracker.mm"],

    data = [ "hand_tracking:hand_tracking_mobile_gpu_binary_graph", ... ], deps = [ "//mediapipe/objc:mediapipe_framework_ios", "//mediapipe/objc:mediapipe_input_sources_ios", "//mediapipe/graphs/hand_tracking:mobile_calculators", "//mediapipe/framework/formats:landmark_cc_proto", ], ), ) 34
  13. 35

  14. 41

  15. ARFrame͔ΒcaptureImageΛऔΓग़͢ • captureImage = Χϝϥ͔Βͷө૾ arSession.delegate = self ... //

    ARSessionDelegate func session(_ session: ARSession, didUpdate frame: ARFrame) { let captureImage: CVPixelBuffer = frame.capturedImage } 43
  16. YCbCr͔ΒBGRA΁ͷม׵ ࣍ͷܭࢉࣜͰม׵͕Ͱ͖Δ R = Y + 1.402 × Cr G

    = Y - 0.344136 × Cb - 0.714136 × Cr B = Y + 1.772 × Cb A = 1 48
  17. 52

  18. RealityKitͷϘλϯΛ্ԼɾΧ΢ϯτΛ දࣔ ࠓճ͸ਓࠩ͠ࢦͷԆ௕ઢ্ʹEntity͕͋Ε͹ɺisPressΛtrueʹ͢Δ func handTracker(_ handTracker: HandTracker!, didOutputLandmarks landmarks: [Landmark]!)

    { let indexFinderPosition = landmarks[8] let screenLocation: CGPoint = ...(indexFinderPosition) self.isPress = arView .entities(at: screenLocation) .contains(where: { $0.id == self.buttonEntity.id }) } 53
  19. 55

  20. 56