Machine Learning for Android Developers

The slides are for a talk I gave at AnDevCon DC 2017:
http://www.andevcon.com/dc2017/sessions#MachineLearningforAndroidDevelopers

A screencast video of it is available here:
https://www.youtube.com/watch?v=fX7eQ9oRp-U

------------------------------------

Find out how to use machine learning on Android!

This talk explains what machine learning can do for you, how it works, and shows how to use a TensorFlow image classifier in your Android app.

We cover:
- Example machine learning applications, including sentiment analysis and artistic image filters.
- Machine learning APIs and frameworks.
- An overview of machine learning: training, evaluation and using models.
- Training data sets: NIST/MNIST, ImageNet, Kaggle.
- How to retrain a TensorFlow model and use it in your Android application: demo and sample code!

Transcript

1. Machine Learning for Android Developers Dan Jarvis
2. • Part 1 - Fundamentals ◦ Context and example applications ◦ Overview of training • Part 2 - Building stuff ◦ APIs and frameworks ◦ Using a pre-trained TensorFlow model ◦ Custom TensorFlow image classifier Agenda
3. Part 1 - Fundamentals
4. • 1997 - Deep Blue supercomputer beats chess world champion • 2006 - Personal computer beats chess world champion • 2011 - IBM Watson beats Jeopardy champions • 2015 - ML image classifiers start to beat humans • 2016 - AlphaGo beats Go Master Tipping Point
5. “I honestly can't think of any recent product development that Microsoft has been involved in that hasn't involved machine learning” “Everything we do now is influenced, one way or another, by machine learning.” - Peter Lee, Microsoft’s Director of Research (Feb 2015)
6. Jeff Dean, Google Senior Fellow - “Machine Learning - The Future Is Now” https://www.youtube.com/watch?v=UdHUDvxcH9o
7. Machine Learning Examples
12. Neural Style • Apply an artistic style to an existing image
13. Neural Style + = https://github.com/anishathalye/neural-style
14. Neural Style + = https://medium.com/@daj/how-does-prisma-work-f434273da92a
15. Sentiment Analysis • Does author feel positively, neutrally, or negatively? • Movie reviews, social media • Order matters
16. http://nlp.stanford.edu:8080/sentiment/rntnDemo.html
17. TensorFlow Android Demos https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
18. TensorFlow Android Demos https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
19. Image Classification • “TF Classify” • Works offline! • ~90MB model
21. Machine Learning Overview
22. When To Use Machine Learning • You cannot code the solution • You cannot scale with humans • You have training data
23. https://upxacademy.com/introduction-machine-learning/
24. • Need lots of data • Lots of excellent ones are online 1 - Get Data
25. • 814k handwritten character images • 128x128 pixels per character NIST
26. • Subset of NIST ◦ 60k for training ◦ 10k for testing • Normalized • 20x20 pixels per character MNIST - http://yann.lecun.com/exdb/mnist/
27. • English lexical database • Concepts = “Synsets” ◦ >114k synsets ◦ >80k are nouns • No longer maintained WORDNET - http://wordnet.princeton.edu
28. • Images for some WORDNET ◦ >14 million images ◦ >21k synsets • Annual competitions IMAGENET - http://imagenet.stanford.edu
29. Kaggle - https://www.kaggle.com
32. https://upxacademy.com/introduction-machine-learning/
33. • Convert into appropriate formats • Label the data (for supervised learning) • Distort your data to generalize your model ◦ Rotate, zoom in/out, brighten, crop … mirror? ◦ Takes longer to train, so make a good model first 2 - Clean & Prepare Data
34. • Very computationally expensive (CPU -> GPU -> TPU) • Only experts can configure training from scratch 3 - Train Model
36. Underfitting and Overfitting
37. • Typically save some training data for testing • Sometimes your accuracy is misleading! 4 - Test
38. • Iterate and increase accuracy 5 - Improve
39. Neural Networks
41. • Computer system modelled on the human brain Neural Networks http://home.agh.edu.pl/~vlsi/AI/intro/
42. • Computer system modelled on the human brain • Nonlinear processing units extract features Neural Networks https://stackoverflow.com/a/32527774
43. • Computer system modelled on the human brain • Nonlinear processing units extract features Neural Networks https://stackoverflow.com/a/32527774
44. https://blog.webkid.io/neural-networks-in-javascript/
45. Deep Learning https://blog.webkid.io/neural-networks-in-javascript/
54. Underfitting and Overfitting
55. Types of Machine Learning
56. • Input data are already labelled • Model trains by iterating 1 - Supervised
57. • Input data are already labelled • Model trains by iterating • Common examples: ◦ Regression 1 - Supervised
58. • Input data are already labelled • Model trains by iterating • Common examples: ◦ Regression ◦ Classification 1 - Supervised
59. • Input data are not labelled • Model deduces structures 2 - Unsupervised
60. • Input data are not labelled • Model deduces structures • Common examples: ◦ Clustering 2 - Unsupervised
61. • Input data are not labelled • Model deduces structures • Common examples: ◦ Clustering ◦ Dimensionality Reduction 2 - Unsupervised
62. • Learn by using a Reward function 3 - Reinforcement
63. • Learn by using a Reward function • Common examples: ◦ Games 3 - Reinforcement http://karpathy.github.io/2016/05/31/rl/
64. • Learn by using a Reward function • Common examples: ◦ Games 3 - Reinforcement https://nihit.github.io/resources/spaceinvaders.pdf
65. Part 2 - Building Stuff
66. Machine Learning APIs • Google Cloud https://cloud.google.com/products/machine-learning/ • IBM Watson https://www.ibm.com/watson/developercloud/discovery.html • Amazon AWS https://aws.amazon.com/machine-learning/ • Microsoft Azure https://docs.microsoft.com/en-us/rest/api/machinelearning/
67. Google Cloud APIs • Machine Learning Engine • Job Search • Video • Vision • Speech • Natural Language • Translation
68. Vision API https://cloud.google.com/vision/
69. https://cloud.google.com/vision/
70. Natural Language API https://cloud.google.com/natural-language/
74. Machine Learning Frameworks
79. Why TensorFlow? • Excellent tutorials, tools and demos • Great developer engagement • Cross platform ◦ Windows, Mac, Linux, Android, iOS, etc ◦ Android Things too! • Built to scale
80. • Training is done in Python or C++ • Requires NDK • Bazel is the build tool • Tutorials lag platform • Big model files (10MB to 90MB for image classifiers) ◦ TensorFlow Lite to the rescue? TensorFlow Limitations
81. Using Pre-Trained TensorFlow Models on Android
82. https://github.com/miyosuda/TensorFlowAndroidMNIST
83. https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/
92. Install Android NDK Add to your PATH Build native library from command line: cd jni-build make Building the Native Library
93. ndk-build Android NDK: WARNING:jni/Android.mk:tensorflow_mnist: non-system libraries in linker flags: jni/libs/armeabi-v7a/libprotos_all_cc.a jni/libs/armeabi-v7a/libprotobuf.a jni/libs/armeabi-v7a/libprotobuf_lite.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libgnustl_static.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libsupc++.a Android NDK: This is likely to result in incorrect builds. Try using LOCAL_STATIC_LIBRARIES Android NDK: or LOCAL_SHARED_LIBRARIES instead to list the library dependencies of the Android NDK: current module make[1]: Entering directory `/development/TensorFlowAndroidMNIST/jni-build' [armeabi-v7a] Compile++ arm : tensorflow_mnist <= tensorflow_jni.cc [armeabi-v7a] Compile++ arm : tensorflow_mnist <= jni_utils.cc [armeabi-v7a] SharedLibrary : libtensorflow_mnist.so [armeabi-v7a] Install : libtensorflow_mnist.so => libs/armeabi-v7a/libtensorflow_mnist.so make[1]: Leaving directory `/development/TensorFlowAndroidMNIST/jni-build'
94. ndk-build Android NDK: WARNING:jni/Android.mk:tensorflow_mnist: non-system libraries in linker flags: jni/libs/armeabi-v7a/libprotos_all_cc.a jni/libs/armeabi-v7a/libprotobuf.a jni/libs/armeabi-v7a/libprotobuf_lite.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libgnustl_static.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libsupc++.a Android NDK: This is likely to result in incorrect builds. Try using LOCAL_STATIC_LIBRARIES Android NDK: or LOCAL_SHARED_LIBRARIES instead to list the library dependencies of the Android NDK: current module make[1]: Entering directory `/development/TensorFlowAndroidMNIST/jni-build' [armeabi-v7a] Compile++ arm : tensorflow_mnist <= tensorflow_jni.cc [armeabi-v7a] Compile++ arm : tensorflow_mnist <= jni_utils.cc [armeabi-v7a] SharedLibrary : libtensorflow_mnist.so [armeabi-v7a] Install : libtensorflow_mnist.so => libs/armeabi-v7a/libtensorflow_mnist.so make[1]: Leaving directory `/development/TensorFlowAndroidMNIST/jni-build'
95. File Sizes Built: • APK = 17MB Raw: • libtensorflow_mnist.so = 14MB • expert-graph.pb = 13MB
96. Performance • Android build = ~20 seconds • JNI build = <10 seconds • Classification = ~70ms first time, 10-20ms thereafter (Nexus 5)
97. Training TensorFlow Models For Beginners
98. Training TensorFlow Models 1. Install TensorFlow 2. Find, download and run a training script
99. 1 - Install TensorFlow Install Docker from www.docker.com Download and start the TensorFlow binary image: docker run -it gcr.io/tensorflow/tensorflow:1.2.0 bash
100. 2 - Find Training Script
101. 2 - Find Training Script
103. 2 - Download Training Script Download the scripts into our Docker TensorFlow container: curl -O https://raw.githubusercontent.com/miyosuda/TensorFlowAndr oidMNIST/master/trainer-script/expert.py curl -O https://raw.githubusercontent.com/miyosuda/TensorFlowAndr oidMNIST/master/trainer-script/input_data.py
104. 2 - Run Training Script Run it: python expert.py
105. # python expert.py Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes. Extracting data/train-images-idx3-ubyte.gz Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes. Extracting data/train-labels-idx1-ubyte.gz Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes. Extracting data/t10k-images-idx3-ubyte.gz Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes. Extracting data/t10k-labels-idx1-ubyte.gz 2017-07-15 20:45:24.352372: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. ... step 0, training accuracy 0.14 step 100, training accuracy 0.86 step 200, training accuracy 0.96 step 300, training accuracy 0.88 step 400, training accuracy 1 step 500, training accuracy 0.94 ...
106. Docker Gotchas • Changes are lost unless you commit them • Only certain local folders can be mounted (e.g. $HOME) • Weird workarounds to avoid out of memory errors
107. ... step 10000, training accuracy 1 step 10100, training accuracy 1 step 10200, training accuracy 0.98 step 10300, training accuracy 1 step 10400, training accuracy 0.98 step 10500, training accuracy 1 step 10600, training accuracy 1 ...
114. ... step 19500, training accuracy 1 step 19600, training accuracy 1 step 19700, training accuracy 1 step 19800, training accuracy 1 step 19900, training accuracy 1 Killed
115. Viewing TensorFlow Models In TensorBoard
116. https://blog.altoros.com/visualizing-tensorflow-graphs-with-tensorboard.html
117. https://blog.altoros.com/visualizing-tensorflow-graphs-with-tensorboard.html
118. Installing TensorBoard • By default has a dependency on TensorFlow docker run -p 6006:6006 -it gcr.io/tensorflow/tensorflow:1.2.0 bash • Standalone version available: https://github.com/dmlc/tensorboard pip install tensorboard
119. Running TensorBoard • Easy to run: tensorboard --logdir=/tmp/tensorflow_logdir • Open http://localhost:6006 in your browser
120. TensorBoard - Limitations • No easy way to import a pretrained model :-( ◦ import_pb_to_tensorboard.py in master, but doesn’t work • Requires code into your training script • Try to find training examples built for TensorBoard
121. Custom Image Classifier on Android https://medium.com/@daj
124. Building From Source https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/README.md • Clone the repo • Install ◦ Bazel ◦ Android NDK ◦ Android SDK tools • Update ./WORKSPACE
125. # Uncomment and update the paths in these entries to build the demo. android_sdk_repository( name = "androidsdk", api_level = 23, build_tools_version = " 25.0.2", # Replace with path to Android SDK on your system path = "/android", ) android_ndk_repository( name="androidndk", path="/android/android-ndk-r12b ", api_level= 21)
126. ... ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:8:1: Traceback (most recent call last): File "/Users/Dan/Code/tensorflow/WORKSPACE", line 8 check_version("0.4.2") File "/Users/Dan/Code/tensorflow/tensorflow/workspace.bzl", line 33, in check_version fail(" Current Bazel version is {}, e...)) Current Bazel version is 0.3.2-homebrew, expected at least 0.4.2 . ERROR: Error evaluating WORKSPACE file. ERROR: error loading package 'external': Package 'external' contains errors. INFO: Elapsed time: 0.214s
127. ... ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:8:1: Traceback (most recent call last): File "/Users/Dan/Code/tensorflow/WORKSPACE", line 8 check_version("0.4.2") File "/Users/Dan/Code/tensorflow/tensorflow/workspace.bzl", line 33, in check_version fail(" Current Bazel version is {}, e...)) Current Bazel version is 0.3.2-homebrew, expected at least 0.4.2 . ERROR: Error evaluating WORKSPACE file. ERROR: error loading package 'external': Package 'external' contains errors. INFO: Elapsed time: 0.214s
128. WARNING: /Users/Dan/Code/tensorflow/tensorflow/core/BUILD:826:12: in srcs attribute of cc_library rule //tensorflow/core:android_tensorflow_lib_lite: please do not import '//tensorflow/core/util/tensor_bundle:tensor_bundle.h' directly. You should either move the file to this package or depend on an appropriate rule there. ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:11:1: no such package '@androidsdk//': Bazel requires Android build tools version 24.0.3 or newer, 23.0.1 was provided and referenced by '//external:android/sdk'. ERROR: Analysis of target '//tensorflow/examples/android:tensorflow_demo' failed; build aborted. INFO: Elapsed time: 28.826s
129. WARNING: /Users/Dan/Code/tensorflow/tensorflow/core/BUILD:826:12: in srcs attribute of cc_library rule //tensorflow/core:android_tensorflow_lib_lite: please do not import '//tensorflow/core/util/tensor_bundle:tensor_bundle.h' directly. You should either move the file to this package or depend on an appropriate rule there. ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:11:1: no such package '@androidsdk//': Bazel requires Android build tools version 24.0.3 or newer, 23.0.1 was provided and referenced by '//external:android/sdk'. ERROR: Analysis of target '//tensorflow/examples/android:tensorflow_demo' failed; build aborted. INFO: Elapsed time: 28.826s
130. • Difficult/tedious • Interferes with your development environment Problems
131. Solution 6.57 GB Docker container with everything in it: https://hub.docker.com/r/danjarvis/tensorflow-android/ Easy to use: docker run -it danjarvis/tensorflow-android:1.0.0
133. Building From Source Start our Docker container docker run -it danjarvis/tensorflow-android:1.0.0 From in the container bazel build -c opt --local_resources 4096,4.0,1.0 -j 1 //tensorflow/examples/android:tensorflow_demo
134. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
135. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
136. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
146. Image Classifier Model - Inception-v3 • https://github.com/tensorflow/models/tree/master/inception • Trained on 1000 IMAGENET categories from 2012 https://medium.com/towards-data-science/transfer-learning-using-keras-d804b2e04ef8
147. Transfer Learning • https://www.tensorflow.org/tutorials/image_retraining • Retraining a classifier is easy! https://medium.com/towards-data-science/transfer-learning-using-keras-d804b2e04ef8
151. docker run -it -v $HOME/tf_files:/tf_files danjarvis/tensorflow-android:1.0.0 Retraining The Model
152. docker run -it -v $HOME/tf_files:/tf_files danjarvis/tensorflow-android:1.0.0 python tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=/tf_files/bottlenecks \ --how_many_training_steps 500 \ --model_dir=/tf_files/inception \ --output_graph=/tf_files/retrained_graph.pb \ --output_labels=/tf_files/retrained_labels.txt \ --image_dir /tf_files/photos Retraining The Model
153. ... 2017–02–27 02:39:58.461678: Step 499: Train accuracy = 100.0% 2017–02–27 02:39:58.461841: Step 499: Cross entropy = 0.036179 2017–02–27 02:39:59.081111: Step 499: Validation accuracy = 99.0% (N=100) Final test accuracy = 97.2% (N=145) Converted 2 variables to const ops.
157. bazel build --local_resources 4096,4.0,1.0 -j 1 tensorflow/python/tools:strip_unused Prepare The Model
158. bazel build --local_resources 4096,4.0,1.0 -j 1 tensorflow/python/tools:strip_unused bazel-bin/tensorflow/python/tools/strip_unused \ --input_graph=/tf_files/retrained_graph.pb \ --output_graph=/tf_files/stripped_retrained_graph.pb \ --input_node_names="Mul" \ --output_node_names="final_result" \ --input_binary=true Prepare The Model
159. cp /tf_files/stripped_retrained_graph.pb bazel-bin/tensorflow/examples/android/assets/stripped_output _graph.pb Rebuilding From Source
160. cp /tf_files/stripped_retrained_graph.pb bazel-bin/tensorflow/examples/android/assets/stripped_output _graph.pb Rebuilding From Source
161. cp /tf_files/stripped_retrained_graph.pb bazel-bin/tensorflow/examples/android/assets/stripped_output _graph.pb cp /tf_files/retrained_labels.txt bazel-bin/tensorflow/examples/android/assets/imagenet_comp_g raph_label_strings.txt Rebuilding From Source
162. cp /tf_files/stripped_retrained_graph.pb bazel-bin/tensorflow/examples/android/assets/stripped_output _graph.pb cp /tf_files/retrained_labels.txt bazel-bin/tensorflow/examples/android/assets/imagenet_comp_g raph_label_strings.txt Rebuilding From Source
163. cp /tf_files/stripped_retrained_graph.pb bazel-bin/tensorflow/examples/android/assets/stripped_output _graph.pb cp /tf_files/retrained_labels.txt bazel-bin/tensorflow/examples/android/assets/imagenet_comp_g raph_label_strings.txt bazel build -c opt --local_resources 4096,4.0,1.0 -j 1 //tensorflow/examples/android:tensorflow_demo Rebuilding From Source
164. python label_image.py <input_image> Command Line Testing
165. python label_image.py test_02_hotdogs.jpg Command Line Testing
166. python label_image.py test_02_hotdogs.jpg legs (score = 0.59678) hot dogs (score = 0.40322) Command Line Testing
167. hot dogs (score = 0.89646) legs (score = 0.10354)
168. legs (score = 0.59678) hot dogs (score = 0.40322)
169. legs (score = 0.91612) hot dogs (score = 0.08388)
170. legs (score = 0.50959) hot dogs (score = 0.49041)
171. Recap
172. From easiest to hardest: 1. Use machine learning cloud APIs 2. Embed pre-trained models 3. Follow instructions to train a model 4. Use transfer learning to customize a pre-trained model 5. Train your own model from scratch 6. Train your model dynamically on the device How To Use Machine Learning
173. • Blog posts: https://medium.com/@daj • Docker container: https://hub.docker.com/r/danjarvis/ • Legs or hot dogs images: https://github.com/daj/legs-or-hotdogs-images • Slides: https://speakerdeck.com/daj Links
174. Thank You!