TF open sourced 2016 TF Mobile 2017 TF Lite developer preview 2018 ML Kit 2019 - ML Kit new features - TF Mobile deprecated - TFLite new announcements!!!
TF.js ML directly inside React Native with WebGL acceleration • Load models from the web, or compile into your application Link to demo video | Link to github 6
Here are the base APIs you can use for ML Kit 9 9 Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply Object detection & Tracking Translation (56 languages) AutoML g.co/mlkit
model for your mobile app: • Download a pre-trained model (here): Inception-v3, mobilenet etc. • Transfer learning with a pre-trained model ◦ Feature extraction or fine tuning on pre-trained model ◦ TensorFlow hub (https://www.tensorflow.org/hub/) • Train your own model from scratch (example in this talk) 14
set and 10,000 test set • 28x28x1 grayscale images • 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 • Popular for computer vision ◦ “hello world” tutorial or ◦ benchmarking ML algorithms 17
on Colab → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define a model 3. Train a model 4. Save a Keras model (save as SavedModel) & convert to tflite 18
visualization tool: • (during training) TensorBoard • (after conversion) Netron - Drop the .tflite model into Netron and see the model visually Note: model metadata in a new TFLite tool (to be launched) will allow you to inspect the model & modify the metadata 21
as SavedModel or a Keras model? Note: In TensorFlow 2.0 , tf.keras.Model.save() and tf.keras.models.save_model() default to the SavedModel format (not HDF5). (link to doc) 23 SavedModel Keras Model Share pre-trained models and model pieces on TensorFlow Hub Train with tf.Keras and you know your deploy your target When you don’t know the deploy target
convert to TFLite file format • Interpreter - execute inference & optimized for small devices • Ops/Kernel - limited ops • Interface to hardware acceleration ◦ NN API ◦ Edge TPU 25
28 Protip: validate the tflite model in python after conversion - 28 TensorFlow result TFLite result Compare results # Test the TensorFlow model on random Input data. tf_result = model(tf.constant(input_data)) # Load TFLite model and allocate tensors. interpreter = tf.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() tflite_result = interpreter.get_tensor(output_details[0]['index']) # Compare the result. for tf_result, tflite_result in zip(tf_result, tflite_result): np.testing.assert_almost_equal(tf_result, tflite_result, decimal=5)
code DigitRecognizer, step by step: • Place tf.lite model under assets folder • Update build.gradle dependencies • Input image - custom view, gallery or camera • Data preprocessing • Classify with the model • Post processing • Display result in UI 30
include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 31
the classifier is an image, your options: 1. Draw on canvas from a custom View 2. Get image from Gallery or a 3rd party camera 3. Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects • 28x28x1- MNIST or FASHION_MNIST gray scale image • 299x299x3 - Inception V3 • 256x256x3 - MobileNet 32
to ByteBuffer • Normalize pixel values to be a certain range • Convert from color to grayscale, if needed Note: with the TFLite Android Support library announced today, this step is simplified 33
is easy • Model conversion to TFLite is easier • Android implementation is still challenging & error-prone: (getting improved with the support library) ◦ Validate tflite model before deploy to Android ◦ Image pre-processing ◦ Input tensor shape? ◦ Color or grayscale? ◦ Post processing My blog post: E2E tf.Keras to TFLite to Android 36
future of machine learning is tiny? - Pete Warden • Deploying to mobile and IoT will get much easier • TFLite new features • Federated learning • On device training 38