CoreML3のオンデバイストレーニングでつくる母音推定

Transcript

1. CoreML3ͷΦϯσόΠετϨʔχϯάͰͭ͘Δ฼Իਪఆ Inference of vowel by on-device training of CoreML3 Naruki Chigira @ Mirrativ, inc. _naru_jpn
2. σϞͰ࢖͏ΞϓϦͷίʔυ https://github.com/naru-jpn/InferenceVowel
3. ฼Իਪఆ /a/, /i/, /u/, /e/, /o/ ͷਪఆ
4. ΦϯσόΠετϨʔχϯά https://developer.apple.com/jp/machine-learning/core-ml/ …ϢʔβʔσʔλΛ࢖༻ͨ͠ϞσϧͷΞοϓσʔτΛΦϯσό ΠεͰ࣮ߦͰ͖ΔΑ͏ʹͳͬͨͨΊɺϓϥΠόγʔΛ৵֐͢Δ ͜ͱͳ͘ɺϢʔβʔͷߦಈʹ߹ΘͤͯϞσϧΛΞοϓσʔτ͠ ͍ͯ͘͜ͱ͕Ͱ͖·͢ɻ
6. ํ਑
7. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio
8. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio let captureSession = AVCaptureSession() … captureSession.startSession()
9. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio ϑʔϦΤม׵ Fourier Transformation ϝϧϑΟϧλόϯΫ Mel Filter Bank ཭ࢄίαΠϯม׵ Discrete Cosine Transformation
10. ϑʔϦΤม׵ Fourier Transformation (ϩά)ϝϧϑΟϧλόϯΫ (Log) Mel Filter Bank ཭ࢄίαΠϯม׵ Discrete Cosine Transformation Import Accelerate // ԻΛप೾਺͝ͱʹ෼ղ vDSP_fft_zrip(fftsetup, … // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ vDSP_mmul(melFilerBank, … // ΑΓগͳ͍σʔλ਺Ͱදݱ vDSP_DCT_Execute(dctsetup, …
11. ϑʔϦΤม׵ Fourier Transformation (ϩά)ϝϧϑΟϧλόϯΫ (Log) Mel Filter Bank ཭ࢄίαΠϯม׵ Discrete Cosine Transformation Import Accelerate // ԻΛप೾਺͝ͱʹ෼ղ vDSP_fft_zrip(fftsetup, … // ਓ͕ฉ͖෼͚΍͍͢௿ԻΛΑΓৄࡉʹ vDSP_mmul(melFilerBank, … // ΑΓগͳ͍σʔλ਺Ͱදݱ vDSP_DCT_Execute(dctsetup, … 1024ݸͷσʔλ 16ݸͷσʔλ ~25μs
12. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb apple/coremltools updatable_nearest_neighbor_classifier.ipynb Jupyter notebook ্Ͱkۙ๣๏Λར༻ͯ͠෼ ྨΛ͢ΔϞσϧΛ࡞੒͢Δαϯϓϧ
13. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb // ֶशʹ࢖͏σʔλͷ४උ let inputValue = snapshot.featureValue let outputValue = MLFeatureValue( string: snapshot.vowel.identifier ) let dataPointFeatures: [String: MLFeatureValue] = [ "input": inputValue, "label": outputValue ] let provider = try! MLDictionaryFeatureProvider( dictionary: dataPointFeatures )
14. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb // ֶशͷ࣮ߦ let batchProvider = MLArrayBatchProvider( array: featureProviders ) let updateTask = try! MLUpdateTask( forModelAt: UpdatableKNN.urlOfModelInThisBundle, trainingData: batchProvider, configuration: nil, completionHandler: completion ) updateTask.resume()
15. લॲཧ Preprocessing ֶश / ਪ࿦ Training / Inference Ի੠σʔλ Audio https://github.com/apple/coremltools/blob/master/examples/updatable_models/updatable_nearest_neighbor_classifier.ipynb // ਪ࿦ͷ࣮ߦ let inputs = snapshots.compactMap({ $0.featureValue.multiArrayValue }).map({ UpdatableKNNInput(input: $0) }) let model = try! UpdatableKNN(contentsOf: updatedModelUrl) // ฼Ի͝ͱͷ֬཰͕ग़ྗͱͯ͠ฦͬͯ͘Δ let outputs = try model.predictions(inputs: inputs) 100~200μs
16. σϞ