Android Developers Guide to Machine Learning with MLkit and TensorFlow

Presented at the Google Developer Agency Day Johannesburg 2018.

Spoke about MLKit, Firebase and Custom TensorFlow models.

1. Android Developers Guide to Machine Learning With MLKit, Tensorflow & Firebase
2. Rebecca Franks @riggaroo Google Developer Expert Android @ Over Pluralsight Author GDG Johannesburg Organiser
3. Who considers themselves an expert in Machine Learning?
5. What is Machine Learning?
6. Machine learning is an application of Artificial Intelligence in which we input a lot of data and let the machines learn “by themselves”
15. Face detection$% W orks offline
22. Demo
24. ⛩ Landmark detection
27. Demo
29. Image Labelling W orks offline
31. Barcode scanning W orks offline
34. val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it))
35. val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result)
36. val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) } }.addOnFailureListener{ textResult.postValue(it.message) }
38. OCR W orks offline
40. On Device vs Cloud
41. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result)
42. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) } } .addOnFailureListener{ Toast.makeText(/../“Error detecting Text $it”/../) }
43. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) } } .addOnFailureListener{ Toast.makeText(/../“Error detecting Text $it”/../) }
45. Custom Tensorflow Models W orks offline
46. TensorFlow
49. Retrain existing model 0 mobilenet_v1
50. What if I could tell what kind of chips I was eating?
51. Gather Training Data FFMPEG Folders of Images Retrain with new images Optimize for mobile Embed in app Store in Firebase App uses model
52. Gather training data
53. Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg
54. Folders of images
55. Retrain with new images python -m scripts.retrain \ --bottleneck_dir=tf_files/bottlenecks \ --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --architecture="${ARCHITECTURE}" \ --image_dir=training_data/south_african_chips
56. Optimize for mobile bazel-bin/tensorflow/contrib/lite/toco/toco \ --input_file=AgencyDay/retrained_graph.pb \ --output_file=AgencyDay/chips_optimized_graph.tflite \ --input_format=TENSORFLOW_GRAPHDEF \ --output_format=TFLITE \ --input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \ --input_array=input \ --output_array=final_result \ --inference_type=FLOAT \ --input_data_type=FLOAT
57. Insert into App https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ https://github.com/googlecodelabs/tensorflow-for-poets-2
58. Nik Nak 1 Or Not 2 bit.ly/mlkit-riggaroo
59. What if our model changes?
60. Ship an app update 3 And hope that people download it 4
61. Host on Firebase Updates automatically downloaded
62. val cloudSource = FirebaseCloudModelSource.Builder("my_cloud_model") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() FirebaseModelManager.getInstance() .registerCloudModelSource(cloudSource) ……
63. g.co/codelabs/mlkit-android-custom-model
64. You don’t need to be a ML Expert to take advantage of ML in your apps!
65. Thank you!
66. Resources - https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets - https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2-tflite/ - https://codelabs.developers.google.com/ codelabs/mlkit-android-custom-model/ #0 - https://github.com/riggaroo/android- demo-mlkit