Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AI for Art and Design - GDG Seattle Android ML ...

AI for Art and Design - GDG Seattle Android ML Talks

A brief overview of how AI can be used to assist and inspire the artists and designers in their creative space. Discuss a few use cases of on-device ML for creating artistic Android apps.

This is a talk at GDG Seattle event: Android ML Talks https://www.meetup.com/gdg-seattle/events/291984330/

Margaret Maynard-Reid

March 15, 2023
Tweet

More Decks by Margaret Maynard-Reid

Other Decks in Technology

Transcript

  1. @margaretmz | #AIArt | #TensorFlow | #TFLite Aspiring 3D Fashion

    Designer 4 • Study at Parsons School of Design • Creating fashion designs with Clo3D
  2. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Topics: generative & on-device •

    Deep dream • Style transfer • GANs & diffusion models • Image to image • Text to image… Generative AI for Art & Design Training TensorFlow 2 / Keras Inference TensorFlow Lite • What is TF Lite? • Sample apps • Awesome TFLite • E2E TFLite Tutorials • Sequential API • Functional API • Model subclassing - custom train_step 5
  3. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Art & Design • Drawing

    • Painting • Photography • Music • Videos • Product design • Fashion design • Graphic design • UI/UX design • ... 7
  4. Computer Vision Tasks Complexity Image classification (single-label) Classify an image

    to a class Examples Painting style or artist Van Gogh Image classification (multi-label) Classify an image to multiple classes Examples Movie poster genre action, sci-fi Feature extraction Extracting latent features of an image with CNN models Examples Visual search find similar fashion Object detection Identify one or multiple objects within an image and their locations with bounding boxes. detect UI elements Segmentation Classify whether each pixel of the image belongs to a certain class segment UI elements Generative models (GANs & Stable Diffusion Two or more models trained simultaneously Examples - Generate new images - Super res - Image-to-image 8
  5. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Deep Dream 2015 9 Source:

    Deep Dream generator https://deepdreamgenerator.com/#gallery
  6. Image Style Transfer (2015) Use CNN neural networks Content image

    + style image = styled image Content Style Generated Neural Style Transfer: 50 Shades of Miaw (link) A few more samples of generated images… 10 Style Transfer 2015
  7. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Models Source: Lilian Weng

    blog (link) 11 “Generative models: take a machine, observe many samples from a distribution and generate more samples from that same distribution”. - Ian Goodfellow 2016
  8. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Adversarial Networks (GANs) GANs

    have at least two network models which compete against each other ... 12
  9. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Diffusion Models 1. Gradually add

    gaussian noise to training data 2. Learn how to reverse the process to generate images from noise. 13 Source: Nvidia developer blog (link) Forward image diffusion Generative reverse denoise
  10. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Text-to-image: VQGAN + CLIP Use

    a text prompt to generate images • VQGAN - Vector Quantized GAN • CLIP Contrastive Language-Image Pre-training model from OpenAI 20 Image Source: @hannahjdotca on Twitter
  11. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Keras Stable Diffusion Tutorials Excellent

    Keras tutorials: • A walk through latent space with Stable Diffusion • High-performance image generation using Stable Diffusion in KerasCV • DreamBooth with Keras 22
  12. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative art & design… A

    lot of sites for trying out generative art and design for free… • https://dreamstudio.com/create/ • https://discord.com/invite/midjourney • https://designer.microsoft.com/ • https://techcrunch.com/2021/12/23/wombo-dream-app/ 24
  13. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative models: key takeaways •

    Diffusion models democratize generative AI • AI will assist the artists and designers • Gives artists and designers inspiration • None engineers will have access to the AI tools • Everyone can create art and designs 25
  14. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign GANs in Art & Design

    My tutorial series with Colab notebooks: https://github.com/margaretmz/GANs-in-Art-and-Design 27
  15. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign TensorFlow 2.x 29 Model •

    tf.Keras (TensorFlow) • Python libraries: Numpy, Matplotlib etc SavedModel or Keras model Serving • Cloud • Web • Mobile • IoT • Micro controllers • Edge TPU Training Inference Data
  16. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign What is TensorFlow Lite? A

    framework with a set of tools for deploying ML to mobile & embedded: • A converter • An interpreter • Ops / kernels • Interface to hardware accelerations: ◦ e.g. NN AIP & Edge TPU • A bunch of other tools: quantization, pruning, benchmarking... ◦ 30
  17. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Android app with TensorFlow Lite

    • ML model: ◦ Train / find a model ◦ Convert to .tflite format ◦ Run inference in Python ◦ Benchmark (optional) ◦ Add model metadata (for using task APIs) • Build Android app ◦ Create UI navigation ◦ Input image: canvas, gallery or camera ◦ Load model & run inference ◦ Display results in UI 31
  18. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign How to implement a GAN

    model? • You only need need the generator model for inference 32
  19. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign ESRGAN on Android with TFLite

    • Enhanced Super Resolution GAN • Pre-trained ESRGAN model from TF Hub • Convert to TFLite model ◦ Dynamic range quantization ◦ Input image 50x50 • TensorFLow C API, GPU library • Android NDK 33 https://blog.tensorflow.org/2020/12/how-to-generate-super -resolution-images-using-tensorflow-lite-on-android.html
  20. @margaretmz | #AIArt | #TensorFlow | #TFLite Selfie2Anime • U-GAT-IT

    paper • Image to image translation • Written in TF1 code • Unpaired selfie and anime images • 50 & 100 epochs checkpoints • Collaboration with Sayak & Khanh • Links: project repo | blog 34
  21. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Cartoonizer • Model: Whitebox CartoonGAN

    (authors Xinrui Wang & Jinze Yu) ◦ Surface, structure and texture representations ◦ Faster & smaller ◦ Not just for selfies • Collaboration with Googlers from TFLite team & another ML GDE • Links: project repo | blog 35
  22. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Segmentation + Style Transfer Demo

    by Khanh LeViet TFLite inference in Python, Already implemented in Android (and iOS)... (project repo) 36
  23. @margaretmz | #AIArt | #TensorFlow | #TFLite Awesome TensorFlow Lite

    A curated list of models, apps, tutorials and learning resources… Thanks to the ML GDE and Android community! Support of TensorFlow Lite team https://github.com/margaretmz/awesome-tflite 37
  24. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Follow me on Twitter, Medium

    or GitHub to learn more about generative art and design, TensorFlow and on-device ML. @margaretmz @margaretmz @margaretmz Thank you! 38