Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AI with Flutter

Rishit Dagli
February 15, 2020

AI with Flutter

A talk at Flutter Mumbai to show how AI algorithms can be deployed on edge using Flutter and some Firebase SDKs

Rishit Dagli

February 15, 2020
Tweet

More Decks by Rishit Dagli

Other Decks in Technology

Transcript

  1. Rishit Dagli ML, DL? ML - capability to learn without

    being explicitly programmed DL - Learning based on DNNs
  2. Rishit Dagli Edge? • Local or near Local processing •

    Not just anywhere in cloud • Low latency • No network availability • Best for real time decision making
  3. Rishit Dagli Cloud vs Edge - Cloud? • Get data

    locally • Send it to cloud • Process it in cloud • Send response back
  4. Rishit Dagli Cloud vs Edge - Edge? • No need

    to send to cloud • Secure • Less impact on network
  5. Rishit Dagli Cloud vs Edge? • Does not mean no

    cloud whatsoever • Cloud can be used for training of models • Inference is on edge
  6. Rishit Dagli Why care about edge? Network • Expensive in

    cost, bandwidth or power • Can be impossible • Sending audio/ video is data intensive
  7. Rishit Dagli How to do edge deployment? • Can I

    use a normal model? • Shrink the model • Use a .tflite model • Or some pretrained models too
  8. Rishit Dagli Convert models • Get the TFLite model •

    You can use the package tflite • You cannot use pre trained models with it • Save the model in Firebase
  9. Rishit Dagli ML Kit • Part of Firebase • Allows

    you to use pre trained models • And custom TF models too
  10. Rishit Dagli ML Kit • Part of Firebase • Allows

    you to use pre trained models • Easily use barcode scanning, text, landmark, label detectors • And custom TF models too
  11. Rishit Dagli Create a simple Scaffold class FacePage extends StatefulWidget

    { @override _FacePageState createState() => _FacePageState(); } class _FacePageState extends State<FacePage> { @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text('Face Detector') ), body: Container(), ); }
  12. Rishit Dagli Create a simple Scaffold class FacePage extends StatefulWidget

    { @override _FacePageState createState() => _FacePageState(); } To track the selected images and any faces detected
  13. Rishit Dagli Add a FAB floatingActionButton: FloatingActionButton( onPressed: (){}, tooltip:

    'Pick Image', child: Icon(Icons.add_a_photo) • You made a FAB with a suitable icon
  14. Rishit Dagli WorkFlow • Select an image • Load image

    for processing • Perform face detection
  15. Rishit Dagli WorkFlow • Select an image • Load image

    for processing • Perform face detection
  16. Rishit Dagli Selecting images • Very easy with the image_picker

    final imageFile = await ImagePicker.pickImage( source: ImageSource.gallery, ); • Can also use camera
  17. Rishit Dagli Selecting images final imageFile = await ImagePicker.pickImage( source:

    ImageSource.gallery, ); Get output of this asynchronous function
  18. Rishit Dagli WorkFlow • Select an image • Load image

    for processing • Perform face detection
  19. Rishit Dagli Loading images final imageFile = await ImagePicker.pickImage( source:

    ImageSource.gallery); final image = FirebaseVisionImage.fromFile(imageFile); Loads and store image in a format suitable for feature detection
  20. Rishit Dagli WorkFlow • Select an image • Load image

    for processing • Perform face detection
  21. Rishit Dagli Face detection - continue • Also provide optional

    parameters • Control accuracy • Speed • Look for ears, eyes, nose
  22. Rishit Dagli Face detection - continue final faceDetector = FirebaseVision.instance.faceDetector(

    FaceDetectorOptions( mode: FaceDetectorMode.fast, enableLandmarks: true, )
  23. Rishit Dagli Face detection - continue • Now asynchronously pass

    your image • And get a list of face coordinates • And any other parameters
  24. Rishit Dagli Face detection - continue class Face{ final Rectangle<int>

    boundingBox; final double headEulerAngleY; final double headEulerAngleZ; final double leftEyeOpenProbability; final double rightEyeOpenProbability; final double smilingProbability; final int trackingId; FaceLadmark getLandmark( FaceLandmarkType landmark, ) => _landmarks[landmark]
  25. Rishit Dagli Face detection - continue • So I will

    just add all of this in a function • Make a call to setState • Mark the state as updated once face detection is complete • Add this function in the FAB on pressed we talked about earlier
  26. Rishit Dagli A problem? • We have 2 await- Getting

    image from user Getting faces list • Face page widget might not be active or mounted
  27. Rishit Dagli A problem? • Can close the app •

    Or navigate away from page • Wasted processing or battery • Solution to this?
  28. Rishit Dagli mounted • Check if widget is mounted •

    Before setting the state if (mounted) { setState(() { _imageFile = imageFile; _faces = faces; });
  29. Rishit Dagli What now? • I have my image •

    Faces list with coordinnates • Faces are detected properly • But I still need to modify the image itself to show a bounding box over face • Any guesses, what should I use?
  30. Rishit Dagli CustomPaint widget • Gives you a canvas •

    Allows you to draw almost anything • Execute simple paint commands
  31. Rishit Dagli CustomPaint widget final customPaint = CustomPaint( painter: myPainter(),

    child: AnyWidget(), ) class MyPainter extends CustomPainter{ @override void paint(ui.Canvas canvas, ui.Size size){ // implement paint } @override void shouldRepaint(CustomPainter oldDelegate){ // implement shoudRepaint } }
  32. Rishit Dagli CustomPaint widget class MyPainter extends CustomPainter{ @override void

    paint(ui.Canvas canvas, ui.Size size){ // implement paint } @override void shouldRepaint(CustomPainter oldDelegate){ // implement shoudRepaint } } Real drawing takes place
  33. Rishit Dagli CustomPaint widget class MyPainter extends CustomPainter{ @override void

    paint(ui.Canvas canvas, ui.Size size){ // implement paint canvas.drawCircle(Offset.zero, Offset(50, 50), 20, Paint()) } Draw figures easily
  34. Rishit Dagli CustomPaint widget final Paint paint = Paint() ..style

    = PaintingStyle.stroke ..strokeWidth = 15.0 ..color = Colors.blue; Easily customise properties
  35. Rishit Dagli CustomPaint widget @override void shouldRepaint(CustomPainter oldDelegate) => false;

    • Controls when painter should redraw • No mutable properties for customPainter • Return false • We cannot change what is drawn
  36. Rishit Dagli Paint over images Future<ui.Image> _loadImage(File file) async {

    final data = await file.readAsBytes(); return await decodeImageFromList(data); } • Load image in a format Canvas can understand
  37. Rishit Dagli Paint over images Future<ui.Image> _loadImage(File file) async {

    final data = await file.readAsBytes(); return await decodeImageFromList(data); } Load image as array of raw bytes
  38. Rishit Dagli Paint over images Future<ui.Image> _loadImage(File file) async {

    final data = await file.readAsBytes(); return await decodeImageFromList(data); } Decode image using a Flutter’s function
  39. Rishit Dagli Paint over images • Create a custom painter

    • Pass it our decoded image • Pass Coordinates for face
  40. Rishit Dagli Paint over images class FacePainter extends CustomPainter{ FacePainter(this.image,

    this.faces); final ui.Image image; final List<Rect> faces; @override void paint(ui.Canvas canvas, ui.Size size){} @override void shouldRepaint(CustomPainter oldDelegate){ return null; }
  41. Rishit Dagli Paint over images @override void paint(ui.Canvas canvas, ui.Size

    size) { canvas.drawImage(image, Offset.zero, Paint()); for (var i = 0; i < faces.length; i++) { canvas.drawRect(rects[i], paint); } } Draw the original image
  42. Rishit Dagli Paint over images @override void paint(ui.Canvas canvas, ui.Size

    size) { canvas.drawImage(image, Offset.zero, Paint()); for (var i = 0; i < faces.length; i++) { canvas.drawRect(rects[i], paint); } } Draw an unfilled rectangle on each face
  43. Rishit Dagli Paint over images @override void paint(ui.Canvas canvas, ui.Size

    size) { canvas.drawImage(image, Offset.zero, Paint()); for (var i = 0; i < faces.length; i++) { canvas.drawRect(rects[i], paint); } } Draw an unfilled rectangle on each face
  44. Rishit Dagli Paint over images @override bool shouldRepaint(FacePainter oldDelegate) {

    return image != oldDelegate.image || faces != oldDelegate.faces; } Repaint if image or list of faces change
  45. Rishit Dagli Paint over images • Reference it from a

    custom paint widget final facePaint = FacePaint( painter: myPainter()); • Are we done?
  46. Rishit Dagli Still a problem • Flutter is not good

    when you try to draw something out of canvas • Image can go outside of your canvas SizedBox( width: _image.width.toDouble(), height: _image.height.toDouble(), child: CustomPaint( painter: FacePainter(_image, _faces), ),
  47. Rishit Dagli Still a problem • Placing custom paint in

    a container • And trying to size it • Bad idea!!! • Use another widget
  48. Rishit Dagli FittededBox • Fit and scale sizeBox inside it

    • Allows other widgets to constrain dimensions FittedBox( child: SizedBox( width: _image.width.toDouble(), height: _image.height.toDouble(), child: CustomPaint( painter: FacePainter(_image, _faces), ),