CoreML3のオンデバイストレーニングでつくる母音推定

Transcript

1. CoreML3ͷΦϯσόΠετϨʔχϯάͰͭ͘Δ฼Իਪఆ
   Inference of vowel by on-device training of CoreML3
   Naruki Chigira @ Mirrativ, inc.
   _naru_jpn
   

   View Slide

2. σϞͰ࢖͏ΞϓϦͷίʔυ
   https://github.com/naru-jpn/InferenceVowel
   

   View Slide

3. ฼Իਪఆ
   /a/, /i/, /u/, /e/, /o/ ͷਪఆ
   

   View Slide

4. ΦϯσόΠετϨʔχϯά
   https://developer.apple.com/jp/machine-learning/core-ml/
   …ϢʔβʔσʔλΛ࢖༻ͨ͠ϞσϧͷΞοϓσʔτΛΦϯσό
   ΠεͰ࣮ߦͰ͖ΔΑ͏ʹͳͬͨͨΊɺϓϥΠόγʔΛ৵֐͢Δ
   ͜ͱͳ͘ɺϢʔβʔͷߦಈʹ߹ΘͤͯϞσϧΛΞοϓσʔτ͠
   ͍ͯ͘͜ͱ͕Ͱ͖·͢ɻ
   

   View Slide

5. View Slide

6. ํ਑
   

   View Slide

7. લॲཧ
   Preprocessing
   ֶश / ਪ࿦
   Training / Inference
   Ի੠σʔλ
   Audio
   

   View Slide

8. લॲཧ
   Preprocessing
   ֶश / ਪ࿦
   Training / Inference
   Ի੠σʔλ
   Audio
   let captureSession = AVCaptureSession()
   …
   captureSession.startSession()
   

   View Slide

9. લॲཧ
   Preprocessing
   ֶश / ਪ࿦
   Training / Inference
   Ի੠σʔλ
   Audio
   ϑʔϦΤม׵
   Fourier Transformation
   ϝϧϑΟϧλόϯΫ
   Mel Filter Bank
   ཭ࢄίαΠϯม׵
   Discrete Cosine Transformation
   

   View Slide

10. ϑʔϦΤม׵
    Fourier Transformation
    (ϩά)ϝϧϑΟϧλόϯΫ
    (Log) Mel Filter Bank
    ཭ࢄίαΠϯม׵
    Discrete Cosine Transformation
    Import Accelerate
    // ԻΛप೾਺͝ͱʹ෼ղ
    vDSP_fft_zrip(fftsetup, …
    // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ
    vDSP_mmul(melFilerBank, …
    // ΑΓগͳ͍σʔλ਺Ͱදݱ
    vDSP_DCT_Execute(dctsetup, …
    

    View Slide

11. ϑʔϦΤม׵
    Fourier Transformation
    (ϩά)ϝϧϑΟϧλόϯΫ
    (Log) Mel Filter Bank
    ཭ࢄίαΠϯม׵
    Discrete Cosine Transformation
    Import Accelerate
    // ԻΛप೾਺͝ͱʹ෼ղ
    vDSP_fft_zrip(fftsetup, …
    // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ
    vDSP_mmul(melFilerBank, …
    // ΑΓগͳ͍σʔλ਺Ͱදݱ
    vDSP_DCT_Execute(dctsetup, …
    1024ݸͷσʔλ
    16ݸͷσʔλ
    ~25μs
    

    View Slide

12. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    apple/coremltools
    updatable_nearest_neighbor_classifier.ipynb
    Jupyter notebook ্Ͱkۙ๣๏Λར༻ͯ͠෼
    ྨΛ͢ΔϞσϧΛ࡞੒͢Δαϯϓϧ
    

    View Slide

13. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ֶशʹ࢖͏σʔλͷ४උ
    let inputValue = snapshot.featureValue
    let outputValue = MLFeatureValue(
    string: snapshot.vowel.identifier
    )
    let dataPointFeatures: [String: MLFeatureValue] = [
    "input": inputValue,
    "label": outputValue
    ]
    let provider = try! MLDictionaryFeatureProvider(
    dictionary: dataPointFeatures
    )
    

    View Slide

14. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ֶशͷ࣮ߦ
    let batchProvider = MLArrayBatchProvider(
    array: featureProviders
    )
    let updateTask = try! MLUpdateTask(
    forModelAt: UpdatableKNN.urlOfModelInThisBundle,
    trainingData: batchProvider,
    configuration: nil,
    completionHandler: completion
    )
    updateTask.resume()
    

    View Slide

15. લॲཧ
    Preprocessing
    ֶश / ਪ࿦
    Training / Inference
    Ի੠σʔλ
    Audio
    https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb
    // ਪ࿦ͷ࣮ߦ
    let inputs = snapshots.compactMap({
    $0.featureValue.multiArrayValue
    }).map({
    UpdatableKNNInput(input: $0)
    })
    let model = try! UpdatableKNN(contentsOf: updatedModelUrl)
    // ฼Ի͝ͱͷ֬཰͕ग़ྗͱͯ͠ฦͬͯ͘Δ
    let outputs = try model.predictions(inputs: inputs)
    100~200μs
    

    View Slide

16. σϞ
    

    View Slide