MediaPipe と ML Kit ってどう ちがうの？ / What is the difference between MediaPipe and ML Kit?

GDG DevFest Tokyo 2024
https://gdg-tokyo.connpass.com/event/335192/

Android で image labeling や object detection など ML の機能を利用したいとなったとき、ML Kit だけでなく MediaPipe を使うという選択肢もあります。これらがどう違うのか、どのように使い分ければいいのか紹介します。

1. .FEJB1JQF
   ͱ
   .-
   ,JU
   ͬͯͲ͏ ͕ͪ͏ͷʁ :VLJ
   "O[BJ
   !ZBO[N
   %FW'FTU
   5PLZP
     1 2"
   TMJEP
2. :VLJ
   "O[BJ w (PPHMF
   %FWFMPQFS
   &YQFSU
   GPS
   "OESPJE
   w 9
   UXJUUFS
   
   !ZBO[N
   w CMPH
   
   ZBO[NCMPHTQPUDPN
   w V1IZDB
   *OD 2
3. "HFOEB w .FEJB1JQF
   ͱ͸
   w .-
   ,JU
   ͱ͸
   w .FEJB1JQF
   4PMVUJPOT
   ͱ
   .-
   ,JU
   ͷҧ͍ 3
4. .FEJB1JQF w 0QFO
   TPVSDF
   NBDIJOF
   MFBSOJOH
   GSBNFXPSL
   w POEFWJDF
   w DSPTTQMBUGPSN
   "OESPJE
   J04
   8FC
   1ZUIPO
   w .FEJB1JQF
   'SBNFXPSL
   w ޮ཰తͳ
   POEFWJDF
   .-
   QJQFMJOF
   Λߏங͢ΔͨΊʹ࢖༻͞ΕΔ
   MPXMFWFM
   ίϯϙʔ ωϯτ
   w .FEJB1JQF
   4PMVUJPOT
   w ΞϓϦέʔγϣϯʹػցֶशͷػೳΛ૊ΈࠐΉͨΊͷϥΠϒϥϦͱπʔϧ܈ 4 https://github.com/google-ai-edge/mediapipe
5. .FEJB1JQF w 0QFO
   TPVSDF
   NBDIJOF
   MFBSOJOH
   GSBNFXPSL
   w POEFWJDF
   w DSPTTQMBUGPSN
   "OESPJE
   J04
   8FC
   1ZUIPO
   w .FEJB1JQF
   'SBNFXPSL
   w ޮ཰తͳ
   POEFWJDF
   .-
   QJQFMJOF
   Λߏங͢ΔͨΊʹ࢖༻͞ΕΔ
   MPXMFWFM
   ίϯϙʔ ωϯτ
   w .FEJB1JQF
   4PMVUJPOT
   w ΞϓϦέʔγϣϯʹػցֶशͷػೳΛ૊ΈࠐΉͨΊͷϥΠϒϥϦͱπʔϧ܈ 5 https://github.com/google-ai-edge/mediapipe
6. .-
   ,JU w POEFWJDF
   .BDIJOF
   -FBSOJOH ػೳΛఏڙ͢Δ
   "OESPJE
   J04
   ޲͚ͷ
   4%,
   w ఏڙ͢Δػೳ
   w 7JTJPO
   "1*
   w /BUVSBM
   -BOHVBHF
   "1*
   w IUUQTEFWFMPQFSTHPPHMFDPNNMLJUHVJEFT 6
7. .FEJB1JQF
   4PMVUJPOT
   ͱ
   .-
   ,JU
   ͷҧ͍ 7
8. ରԠϓϥοτϑΥʔϜ 8 MediaPipe Solutions ML Kit w "OESPJE
   w J04
   w 8FC
   w 1ZUIPO w "OESPJE
   w J04
9. ػೳ 9 Media Pipe Solutions ML Kit
10. ػೳ 10 MediaPipe Solutions ML Kit
11. Ϟσϧબ୒ͷ༗ແ 11 MediaPipe Solutions ML Kit w ར༻͢ΔϞσϧΛࣗ෼ͰબͿ
    w ਪ঑Ϟσϧ͕༻ҙ͞Ε͍ͯΔ w CVJMEJO
    ͷϞσϧ͕࢖ΘΕΔ
    w Ұ෦ͷػೳͰ͸ΧελϜϞσϧͷར ༻͕Մೳ
12. Ϟσϧͷ഑෍ 12 MediaPipe Solutions ML Kit w ϞσϧΛΞϓϦ಺ʹ഑ஔ w CVJMEJO
    Ϟσϧ
    w ϥΠϒϥϦͱҰॹʹϞσϧΛΞϓϦʹ
    CVOEMF
    w (PPHMF
    QMBZ
    TFSWJDFT
    Ͱμ΢ϯϩʔυ
    w ΧελϜϞσϧʢରԠ͍ͯ͠ΔػೳͷΈʣ
    w ϞσϧΛΞϓϦ಺ʹ഑ஔ
    w 'JSFCBTF
    .-
    ʹϞσϧΛϗετ
    w (PPHMF
    QMBZ
    TFSWJDFT
    Ͱμ΢ϯϩʔυ
    w fi SFCBTFNMNPEFMEPXOMPBEFS
    Ͱμ΢ϯϩʔυ
13. 13 https://ai.google.dev/edge/mediapipe/solutions/vision/image_classi fi er#models MediaPipe Solutions
14. 14 val baseOptions = BaseOptions.builder() .setModelAssetPath("efficientnet_lite0.tflite") .build() val options = ImageClassifier.ImageClassifierOptions .builder() .setBaseOptions(baseOptions) … .build() imageClassifier = ImageClassifier.createFromOptions(context, options) MediaPipe Solutions αϯϓϧίʔυ https://github.com/google-ai-edge/mediapipe-samples AGP, Gradle ͕ݹ͍ͷͰ஫ҙ implementation("com.google.mediapipe:tasks-vision:0.20230731")
15. 15 ML Kit val options = ImageLabelerOptions.Builder() .setConfidenceThreshold(0.6f) .build() labeler = ImageLabeling.getClient(options) // bundled : About 5.7 MB size increase. implementation("com.google.mlkit:image-labeling:17.0.9") or // unbundled : About 200 KB size increase. // Model is dynamically downloaded via Google Play Services. implementation("com.google.android.gms:play-services-mlkit-image-labeling:16.0.8")
16. Ϟσϧͷҧ͍ 16 MediaPipe Solutions Image classi fi cation ML Kit Image Labeling w & ffi DJFOU/FU-JUF
    w *NBHF/FU
    Ͱֶशͨ͠
    
    छྨͷ Ϋϥε
    w IUUQTTUPSBHFHPPHMFBQJTDPN NFEJBQJQFUBTLTJNBHF@DMBTTJ fi FS MBCFMTUYU w 
    छྨͷϥϕϧ
    w IUUQTEFWFMPQFSTHPPHMFDPN NMLJUWJTJPOJNBHFMBCFMJOH MBCFMNBQ
17. 17 cup : 0.41 Cup : 0.98 Tableware : 0.95 Porcelain : 0.88
18. ઃఆͰ͖Δ߲໨ͷҧ͍ 18 MediaPipe Solutions Image classi fi cation ML Kit Image Labeling ImageClassifier.ImageClassifierOptions .builder() .setScoreThreshold(0.3f) .setMaxResults(3) .setRunningMode(RunningMode.LIVE_STREAM) .setDisplayNamesLocale("ja") .setCategoryAllowlist(…) or .setCategoryDenylist(…) ImageLabelerOptions.Builder() .setConfidenceThreshold(0.3f)
19. ΧελϜϞσϧ 19
20. 20 https://www.kaggle.com/models/google/aiy/tfLite/vision-classi fi er-birds-v1/3
22. 22 MediaPipe Solutions val baseOptions = BaseOptions.builder() .setModelAssetPath("vision_classifier_birds_v3.tflite") .build() val options = ImageClassifier.ImageClassifierOptions .builder() .setBaseOptions(baseOptions) … .build() imageClassifier = ImageClassifier.createFromOptions(context, options)
23. 23 ML Kit https://developers.google.com/ml-kit/custom-models#model-compatibility
24. 24 ML Kit https://developers.google.com/ml-kit/custom-models#model-compatibility
25. 25 ML Kit (Bundled) val localModel = LocalModel.Builder() .setAssetFilePath("vision_classifier_birds_v3.tflite") // or .setAbsoluteFilePath(absolute file path to model file) // or .setUri(URI to model file) .build() val options = CustomImageLabelerOptions.Builder(localModel) .setConfidenceThreshold(0.3f) .setMaxResultCount(5) .build() labeler = ImageLabeling.getClient(options) implementation("com.google.mlkit:image-labeling-custom:17.0.3")
26. 26 ML Kit (Bundled) val localModel = LocalModel.Builder() .setAssetFilePath("vision_classifier_birds_v3.tflite") // or .setAbsoluteFilePath(absolute file path to model file) // or .setUri(URI to model file) .build() val options = CustomImageLabelerOptions.Builder(localModel) .setConfidenceThreshold(0.3f) .setMaxResultCount(5) .build() labeler = ImageLabeling.getClient(options) 26 implementation("com.google.mlkit:image-labeling-custom:17.0.3")
27. 27 build-in model vision_classi fi er_birds_v3.t fl ite
28. w 'JSFCBTF
    .-
    ʹϞσϧΛϗετ͢Δʢ
    'JSFCBTF
    ͕ඞཁʣ
    w ΞϓϦαΠζΛ཈͑ΒΕΔ
    w ΞϓϦΛߋ৽ͤͣʹϞσϧΛߋ৽Ͱ͖Δ
    w Ϟσϧͷμ΢ϯϩʔυ͕ऴΘΔ·Ͱਪ࿦Ͱ͖ͳ͍
    w (PPHMF
    1MBZ
    4FSWJDFT
    ʹμ΢ϯϩʔυΛ͓೚ͤ
    PS
    fi SFCBTFNM NPEFMEPXOMPBEFS
    Ͱμ΢ϯϩʔυ 28 ML Kit (Unbundled)
32. 32 ML Kit (Unbundled : Google Play Services Ͱμ΢ϯϩʔυ) val remoteModel = CustomRemoteModel .Builder(FirebaseModelSource.Builder("vision_classifier_food").build()) .build() val downloadConditions = DownloadConditions.Builder().requireWifi().build() RemoteModelManager.getInstance() .download(remoteModel, downloadConditions) .addOnSuccessListener { val options = CustomImageLabelerOptions.Builder(remoteModel) .setConfidenceThreshold(0.1f) .setMaxResultCount(5) .build() labeler = ImageLabeling.getClient(options) } implementation("com.google.android.gms:play-services-mlkit-image-labeling-custom:16.0.0-beta5") implementation("com.google.mlkit:linkfirebase:17.0.0")
33. 33 ML Kit (Unbundled : fi rebase-ml-modeldownloader Ͱμ΢ϯϩʔυ) val conditions = CustomModelDownloadConditions.Builder().requireWifi().build() FirebaseModelDownloader.getInstance() .getModel("vision_classifier_food", DownloadType.LOCAL_MODEL, conditions) .addOnSuccessListener { model -> model?.file?.let { val localModel = LocalModel.Builder() .setAbsoluteFilePath(it.absolutePath) .build() val customImageLabelerOptions = CustomImageLabelerOptions.Builder(localModel) .setConfidenceThreshold(0.1f) .setMaxResultCount(5) .build() labeler = ImageLabeling.getClient(customImageLabelerOptions) } } implementation("com.google.firebase:firebase-ml-modeldownloader:25.0.1")
35. .FEJB1JQF
    .PEFM
    .BLFS 35
36. .FEJB1JQF
    .PEFM
    .BLFS w ϕʔεͷϞσϧΛసҠֶश
    w Ұ෦ͷػೳ͕ରԠʢ0CKFDU
    %FUFDUJPO
    ΍
    *NBHF
    $MBTTJ fi DBUJPO
    ͳͲʣ
    w $PMBC
    ͕༻ҙ͞Ε͍ͯΔ 36 https://ai.google.dev/edge/mediapipe/solutions/model_maker
37. 37 https://ai.google.dev/edge/mediapipe/solutions/customization/image_classi fi er
38. 38 w *NBHF
    $MBTTJ fi DBUJPO
    ͷΧελϜϞσϧ
    w छྨ
    
    ຕͩͱ͜ͷఔ౓ͷਫ਼౓ɺ࠷௿
     ຕ͸ཉ͍͠
    w 
    Y
    ʢ΋ͬͱখ͍͍ͯ͘͞ʣ
    ͷ
    
    ຕ
    Y
    
    
    ຕͰֶश࣌ؒ͸໿෼
39. ·ͱΊ w 8FC
    ΍
    1ZUIPO
    Ͱ࢖͍͍ͨ
    ˠ
    .FEJB1JQF
    4PMVUJPOT
    w ࢖͍͍ͨػೳ͕Ұํʹ͔͠ͳ͍
    ˠ
    ػೳ͕͋Δ΄͏
    w Ϟσϧͷѻ͍Λ͓·͔͍ͤͨ͠
    ˠ
    .-
    ,JU
    w 0CKFDU
    %FUFDUJPO
    *NBHF
    -BCFMJOH
    Ҏ֎ͰΧελϜϞσϧΛ࢖͍͍ͨ
    ˠ
    .FEJB1JQF
    4PMVUJPOT
    w 0CKFDU
    %FUFDUJPO
    *NBHF
    -BCFMJOH
    ͷΧελϜϞσϧΛΞϓϦʹόϯυϧͨ͘͠ͳ ͍
    ˠ
    .-
    ,JU
    w ͍ΖΜͳػցֶशػೳ͕؆୯ʹΞϓϦʹ૊ΈࠐΊΔΑ͏ʹͳͬͯΔ͔Βࢼͯ͠Έͯʂ 39
40. 5IBOL
    ZPV 40