Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Core ML 🏃 iOS Engineer

Core ML 🏃 iOS Engineer

at #tokyoiosmeetup
codes: https://github.com/shingt/BeerClassifier

Shinichi Goto

December 16, 2017
Tweet

More Decks by Shinichi Goto

Other Decks in Programming

Transcript

  1. Core ML ! iOS Engineer ɹ 2017/12/16 - Tokyo iOS

    meetup @shingt (Shinichi Goto)
  2. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning

    with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 7
  3. 8

  4. 9

  5. 10

  6. Core ML ɹ A new foundational machine learning framework used

    across Apple products, including Siri, Camera, and QuickType. https://developer.apple.com/machine-learning/ 12
  7. Core ML • Framework to integrate pre-trained models to apps

    • Inference only • Introduced Core ML model format (**.mlmodel) • Xcode automatically generates Swift interface for model • coremltools 13
  8. ɹ let animalModel = AnimalModel() if let prediction = try?

    animalModel.prediction(animalImage: image) { return prediction.animalType } 14
  9. 15

  10. Model conversion • coremltools (provided by Apple) • Enables converting

    models trained by major tools to mlmodel format • tools: Keras, Caffe, scikit-learn, etc. • TensorFlow model converter • MXNet Model converter 16
  11. BeerClassifier • Classify beer (bottles) in real time • B*dweiser

    / Co*ona / Hei*eken • Input: image • Output: beer name • https://github.com/shingt/BeerClassifier 18
  12. 21

  13. Step1: Collect images • Drink beer (and take photos) •

    Crawl beer (bottle) images • Bing Image Search API (v7) • Query: "B*dweiser bottle" • Need to exclude wrong images by myself • ~= 1,600 images (not big) • scraper.py 22
  14. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 23
  15. Step 2: Pre-process images • Process images for algorithms input

    • In my case • Resize (to 32x32) • Center-crop • preprocess.py 24
  16. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 25
  17. Step 3: Training • Using Keras • Construct the same

    network as cifar10_cnn.py • Available in Keras example networks • cifar10 (dataset): 80 million images, 10 classes • create_data_csv.py, train.py • (Skipped data augmentation) 26
  18. model = Sequential() model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(32,

    (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(64, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes)) model.add(Activation('softmax')) opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6) model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) 27
  19. • Step 1: Collect images • Step 2: Pre-process images

    • Step 3: Training • Step 4: Convert model to mlmodel format 28
  20. Step 4: Convert model to mlmodel format # convert.py coreml_model

    = coremltools.converters.keras.convert('./models/beer_model.h5', input_names = 'image', image_input_names = 'image', class_labels = './texts/beer_labels.txt' ) coreml_model.license = 'MIT' coreml_model.short_description = 'Beer Classifier' coreml_model.author = 'shingt' coreml_model.save('./models/BeerClassifier.mlmodel') 29
  21. To replace model • Update model and release app again

    ! OR • Download your model and compile it on device ! • MLModel#compileModel(at:) • Ref: Downloading and Compiling a Model on the User's Device | Apple Developer Documentation 32
  22. Download / compile model on device // On background thread

    // Assume in `do` block let url = URL(string: "https://your_model_url.mlmodel")! let data = try Data(contentsOf: url) try data.write(to: filePathUrl) let compiledUrl = try MLModel.compileModel(at: filePathUrl) let model = try MLModel(contentsOf: compiledUrl) // And use model // ... 33
  23. Learning cost • Great if you work with your team

    ! • And if Core ML satisfies your business requirements • if you're an individual developer? • Easy to copy-and-paste, but how to improve? • Understainding of statistics, learning algorithms, evaluation, ... • life is limited (it's great if you want to learn ML of course) 35
  24. "You do not need to be a machine learning expert

    to make use of these models. You continue to focus on the use case you're trying to enable and let Core ML handle those low-level details." https://developer.apple.com/videos/play/wwdc2017/703/ 36
  25. 37

  26. apple/turicreate • Application-oriented toolkit • Developers can focus on tasks

    instead of algorithms • Supported tasks • Image classification, Image similarity, Object detection • Recommender systems, Text classifier, Activity classification • Apple acquires Turi, a machine learning company | TechCrunch (posted Aug 5, 2016) 38
  27. # https://apple.github.io/turicreate/docs/userguide/image_classifier/introduction.html import turicreate as tc data = tc.SFrame('cats-dogs.sframe') train_data,

    test_data = data.random_split(0.8) # Automatically picks the right model based on your data. # Based on Transfer Learning model = tc.image_classifier.create(train_data, target='label') predictions = model.predict(test_data) metrics = model.evaluate(test_data) model.save('mymodel.model') model.export_coreml('MyCustomImageClassifier.mlmodel') 39
  28. Model size • BeerClassifier...5MB • Provided models by apple •

    ResNet50...102.6MB, VGG16...553.5 MB ! • Integrating / replacing is not realistic • No model compression technique in Core ML (currently) • Depends on your task / application • Reducing the Size of Your Core ML App | Apple Developer Documentation 40
  29. Other concerns • Unsupported machine learning components • You cannot

    convert your model if it include components Core ML doesn't support • Model data protection • Parameters are not encrypted / can be seen • etc. • Why Core ML will not work for your app (most likely) 41
  30. Recap • Core ML basics • BeerClassifier • Concerns •

    learning cost, model size, etc. • apple/turicreate? 42
  31. Reference • Introducing Core ML • Core ML in depth

    • Vision Framework: Building on Core ML • iOS 11: Machine Learning for everyone • Why Core ML will not work for your app (most likely) • CoreMLͰΞΠυϧإೝࣝΞϓϦΛ࡞Ζ͏ 43