Machine Learning for Android Developers

Ever wanted to build smart Android apps the easy way? This presentation covers all there is to know about Face Detection, Barcode Detection and Text Recognition with the Mobile Vision API.

1. Machine Learning for Android Developers Moyinoluwa Adeyemi November 25, 2016
2. Android @SwiftaSystems GDG Lagos Co-organizer Android Developer Nanodegree Graduate Half Marathoner
3. How Can You Get Started with Machine Learning? Three ways, with varying complexity: (1) Use a Cloud-based or Mobile API (Vision, Natural Language, etc.) (2) Use an existing model architecture, and retrain it or fine tune on your dataset (3) Develop your own machine learning models for new problems More flexible, but more effort required
4. Mobile Vision API Providing on-device vision for applications
5. Face API faces, facial landmarks, eyes open, smiling Barcode API 1D and 2D barcodes Text API Latin-based text / structure Common Mobile Vision API Support for fast image and video on-device detection and tracking. NEW!
6. Googly Eyes Android App Video credit Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray<Face> faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image
7. Face API Photo credit developers.google.com/vision
8. Easy to use Java API image detected items Detector 1. Create a detector object 2. detectedItems = detector.detect(image) Photo credit developers.google.com/vision
9. Text Detection Latin based language Understand text structure Photo credit Getty Images
10. Text Structure Blocks Lines Words Lines Words Words Words
11. Barcode Detection 1D barcodes EAN-13/8 UPC-A/E Code-39/93/128 ITF Codabar 2D barcodes QR Code Data Matrix PDF-417 AZTEC UPC DataMatrix QR Code PDF 417 Video and image credit Google
12. Combined Vision & Translation
13. Code Walkthrough
14. Import the Google Play Services SDK for the Mobile Vision API compile 'com.google.android.gms:play-services-vision:9.8.0'
15. Face Detection manifest file meta-data <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="face"/>
16. Barcode Detection manifest file meta-data <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="barcode"/>
17. Text Recognition manifest file meta-data <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="ocr"/>
18. Create simple layout with static image Process
19. Convert the static image into a Bitmap Bitmap barcodeBitmap = BitmapFactory.decodeResource(getResources(), R.drawable.cute_cat_image);
20. Using the FaceDetector API FaceDetector faceDetector = new FaceDetector.Builder(this) .setTrackingEnabled(false) .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build();
21. Using the BarcodeDetector API BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(this) .setBarcodeFormats(Barcode.ALL_FORMATS) .build();
22. Using the TextRecognizer API TextRecognizer textRecognizer = new TextRecognizer.Builder(this) .build();
23. Check if the face detector is ready if (!faceDetector.isOperational()) { new AlertDialog.Builder(this) .setMessage("Face Detector could not be set up on your device") .show(); return; }
24. Check if the barcode detector is ready if (!barcodeDetector.isOperational()) { new AlertDialog.Builder(this) .setMessage("Barcode Detector could not be set up on your device") .show(); return; }
25. Check if the text recognizer is ready if (!textRecognizer.isOperational()) { new AlertDialog.Builder(this) .setMessage("Text Recognizer could not be set up on your device") .show(); return; }
26. Create a frame using the face bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray<Face> face = faceDetector.detect(frame);
27. Create a frame using the barcode bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray<Barcode> barcode = barcodeDetector.detect(frame);
28. Create a frame using the text bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray<TextBlock> text = textRecognizer.detect(frame);
29. Use the values from the SparseArray for (int i = 0; i < sparseArray.size(); i++) { Face face = sparseArray.valueAt(i); float left = face.getPosition().x; float top = face.getPosition().y; float right = left + face.getWidth(); float bottom = right + face.getHeight(); float cornerRadius = 2.0f; RectF rectF = new RectF(left, top, right, bottom); canvas.drawRoundRect(rectF, cornerRadius, cornerRadius, rectPaint); }
30. Use the values from the SparseArray for (int i = 0; i < sparseArray.size(); i++) { Face face = sparseArray.valueAt(i); float left = face.getPosition().x; float top = face.getPosition().y; float right = left + face.getWidth(); float bottom = right + face.getHeight(); float cornerRadius = 2.0f; RectF rectF = new RectF(left, top, right, bottom); canvas.drawRoundRect(rectF, cornerRadius, cornerRadius, rectPaint); }
31. Use the values from the SparseArray if (size == 0) { textView.setText("No information available"); } else { for (int i = 0; i < size; i++) { barcodeValue += (barcode.valueAt(i).displayValue + "\n"); } textView.setText(barcodeValue); }
32. for (int i = 0; i < text.size(); i++) { TextBlock textBlock = text.valueAt(i); if (textBlock != null && textBlock.getValue() != null) { detectedText += textBlock.getValue(); } detectedTextView.setText(detectedText); } Use the values from the SparseArray
33. faceDetector.release(); Remember to release resources
34. barcodeDetector.release(); Remember to release resources
35. textRecognizer.release(); Remember to release resources
36. Results - Face Detection API
37. Results - Barcode Detection API
38. Results - Text Recognition API
39. Mobile Vision: Codelabs and Samples Googly Eyes Code Sample github.com/googlesamples/android-vision/tree/master/visionSamples/googly-eyes Codelabs codelabs.developers.google.com/codelabs/face-detection/ codelabs.developers.google.com/codelabs/bar-codes/ codelabs.developers.google.com/codelabs/mobile-vision-ocr/ Mobile Vision Developers developers.google.com/vision/ GitHub Code Samples github.com/googlesamples/android-vision Stack Overflow Find and ask questions under the android-vision tag.
40. Questions?
41. @moyheen @moyinoluwa @moyheen Thank you! Moyinoluwa Adeyemi Android @SwiftaSystems GDG Lagos co-organizer