AI for Art and Design - GDG Seattle Android ML Talks

A brief overview of how AI can be used to assist and inspire the artists and designers in their creative space. Discuss a few use cases of on-device ML for creating artistic Android apps.

This is a talk at GDG Seattle event: Android ML Talks https://www.meetup.com/gdg-seattle/events/291984330/

Transcript

1. AI for Art & Design (with TensorFlow Lite) Margaret Maynard-Reid 3/15/2023 @margaretmz
2. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign About me 2
3. margaretmz.art
4. @margaretmz | #AIArt | #TensorFlow | #TFLite Aspiring 3D Fashion Designer 4 • Study at Parsons School of Design • Creating fashion designs with Clo3D
5. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Topics: generative & on-device • Deep dream • Style transfer • GANs & diffusion models • Image to image • Text to image… Generative AI for Art & Design Training TensorFlow 2 / Keras Inference TensorFlow Lite • What is TF Lite? • Sample apps • Awesome TFLite • E2E TFLite Tutorials • Sequential API • Functional API • Model subclassing - custom train_step 5
6. Generative Art & Design 6
7. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Art & Design • Drawing • Painting • Photography • Music • Videos • Product design • Fashion design • Graphic design • UI/UX design • ... 7
8. Computer Vision Tasks Complexity Image classification (single-label) Classify an image to a class Examples Painting style or artist Van Gogh Image classification (multi-label) Classify an image to multiple classes Examples Movie poster genre action, sci-fi Feature extraction Extracting latent features of an image with CNN models Examples Visual search find similar fashion Object detection Identify one or multiple objects within an image and their locations with bounding boxes. detect UI elements Segmentation Classify whether each pixel of the image belongs to a certain class segment UI elements Generative models (GANs & Stable Diffusion Two or more models trained simultaneously Examples - Generate new images - Super res - Image-to-image 8
9. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Deep Dream 2015 9 Source: Deep Dream generator https://deepdreamgenerator.com/#gallery
10. Image Style Transfer (2015) Use CNN neural networks Content image + style image = styled image Content Style Generated Neural Style Transfer: 50 Shades of Miaw (link) A few more samples of generated images… 10 Style Transfer 2015
11. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Models Source: Lilian Weng blog (link) 11 “Generative models: take a machine, observe many samples from a distribution and generate more samples from that same distribution”. - Ian Goodfellow 2016
12. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative Adversarial Networks (GANs) GANs have at least two network models which compete against each other ... 12
13. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Diffusion Models 1. Gradually add gaussian noise to training data 2. Learn how to reverse the process to generate images from noise. 13 Source: Nvidia developer blog (link) Forward image diffusion Generative reverse denoise
14. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Stable Diffusion https://jalammar.github.io/illustrated-stable-diffusion/ 14
15. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign 15
16. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Fashion++ Facebook AI: “Fashion++: Minimal Edits for Outfit Improvement” (news | paper) 16
17. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Icons 8 - AI powered face generation 17 Image Source: icons8.com
18. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Photoshop - Neural Filters 18 Image Source: Adobe Photoshop tour
19. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Text-to-image: StyleCLIP Text-Driven Manipulation of StyleGAN Imagery 19
20. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Text-to-image: VQGAN + CLIP Use a text prompt to generate images • VQGAN - Vector Quantized GAN • CLIP Contrastive Language-Image Pre-training model from OpenAI 20 Image Source: @hannahjdotca on Twitter
21. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Stable Diffusion Link to tweet here 21 Source: stability.ai
22. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Keras Stable Diffusion Tutorials Excellent Keras tutorials: • A walk through latent space with Stable Diffusion • High-performance image generation using Stable Diffusion in KerasCV • DreamBooth with Keras 22
23. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign 23 Lexica.art
24. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative art & design… A lot of sites for trying out generative art and design for free… • https://dreamstudio.com/create/ • https://discord.com/invite/midjourney • https://designer.microsoft.com/ • https://techcrunch.com/2021/12/23/wombo-dream-app/ 24
25. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Generative models: key takeaways • Diffusion models democratize generative AI • AI will assist the artists and designers • Gives artists and designers inspiration • None engineers will have access to the AI tools • Everyone can create art and designs 25
26. Train a GAN model with TensorFlow 2 / Keras 26
27. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign GANs in Art & Design My tutorial series with Colab notebooks: https://github.com/margaretmz/GANs-in-Art-and-Design 27
28. Model Inference with TensorFlow Lite 28
29. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign TensorFlow 2.x 29 Model • tf.Keras (TensorFlow) • Python libraries: Numpy, Matplotlib etc SavedModel or Keras model Serving • Cloud • Web • Mobile • IoT • Micro controllers • Edge TPU Training Inference Data
30. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign What is TensorFlow Lite? A framework with a set of tools for deploying ML to mobile & embedded: • A converter • An interpreter • Ops / kernels • Interface to hardware accelerations: ◦ e.g. NN AIP & Edge TPU • A bunch of other tools: quantization, pruning, benchmarking... ◦ 30
31. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Android app with TensorFlow Lite • ML model: ◦ Train / find a model ◦ Convert to .tflite format ◦ Run inference in Python ◦ Benchmark (optional) ◦ Add model metadata (for using task APIs) • Build Android app ◦ Create UI navigation ◦ Input image: canvas, gallery or camera ◦ Load model & run inference ◦ Display results in UI 31
32. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign How to implement a GAN model? • You only need need the generator model for inference 32
33. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign ESRGAN on Android with TFLite • Enhanced Super Resolution GAN • Pre-trained ESRGAN model from TF Hub • Convert to TFLite model ◦ Dynamic range quantization ◦ Input image 50x50 • TensorFLow C API, GPU library • Android NDK 33 https://blog.tensorflow.org/2020/12/how-to-generate-super -resolution-images-using-tensorflow-lite-on-android.html
34. @margaretmz | #AIArt | #TensorFlow | #TFLite Selfie2Anime • U-GAT-IT paper • Image to image translation • Written in TF1 code • Unpaired selfie and anime images • 50 & 100 epochs checkpoints • Collaboration with Sayak & Khanh • Links: project repo | blog 34
35. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Cartoonizer • Model: Whitebox CartoonGAN (authors Xinrui Wang & Jinze Yu) ◦ Surface, structure and texture representations ◦ Faster & smaller ◦ Not just for selfies • Collaboration with Googlers from TFLite team & another ML GDE • Links: project repo | blog 35
36. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Segmentation + Style Transfer Demo by Khanh LeViet TFLite inference in Python, Already implemented in Android (and iOS)... (project repo) 36
37. @margaretmz | #AIArt | #TensorFlow | #TFLite Awesome TensorFlow Lite A curated list of models, apps, tutorials and learning resources… Thanks to the ML GDE and Android community! Support of TensorFlow Lite team https://github.com/margaretmz/awesome-tflite 37
38. @margaretmz @GDGSeattle #AI #Art #3DFashionDesign Follow me on Twitter, Medium or GitHub to learn more about generative art and design, TensorFlow and on-device ML. @margaretmz @margaretmz @margaretmz Thank you! 38