Client-Side Machine Learning - Bringing AI to the Frontend

Transcript

1. C L I E N T- S I D E M A C H I N E L E A R N I N G B R I N G I N G A I T O T H E F R O N T E N D @markusingvarssn
2. https://www.linkedin.com/posts/david-kriesel_nice-one-activity-7062736038179262464-OWdU/
3. Qualcomm image - https://www.qualcomm.com/news/onq/2024/02/what-is-an-npu-and-why-is-it-key-to-unlocking-on-device-generative-ai
4. Qualcom image - https://www.qualcomm.com/news/onq/2024/02/what-is-an-npu-and-why-is-it-key-to-unlocking-on-device-generative-ai Apple Neural Engine image - https://www.youtube.com/watch?v=zXJbdtxh0XE
5. Apple Intelligence on iPhone in 5 minutes - https://www.youtube.com/watch?v=PugKQZHPut8
6. B R I N G I N G A I TO T H E F R O N T E N D After this talk, we will know about:
7. B R I N G I N G A I TO T H E F R O N T E N D After this talk, we will know about: • Recent innovation that enables client-side machine learning
8. B R I N G I N G A I TO T H E F R O N T E N D After this talk, we will know about: • Recent innovation that enables client-side machine learning • What client-side machine learning can do for us
9. B R I N G I N G A I TO T H E F R O N T E N D After this talk, we will know about: • Recent innovation that enables client-side machine learning • What client-side machine learning can do for us • How we can start leveraging it today
10. HELLO! BRINGING AI TO THE FRONTEND I’m Markus @markusingvarssn
11. HELLO! BRINGING AI TO THE FRONTEND I’m Markus @markusingvarssn
12. HELLO! BRINGING AI TO THE FRONTEND I’m Markus @markusingvarssn
13. M A C H I N E L E A R N I N G I N 1 0 0 S E C O N D S https://www.linkedin.com/pulse/machine-learning-image-detectioncats-vs-dogs-amrith-kumar/
14. M A C H I N E L E A R N I N G I N 1 0 0 S E C O N D S https://www.linkedin.com/pulse/machine-learning-image-detectioncats-vs-dogs-amrith-kumar/
15. ” R E C E N T ” M AC H I N E L E A R N I N G I N N OVAT I O N S
16. ” R E C E N T ” M AC H I N E L E A R N I N G I N N OVAT I O N S • Hardware improvements – The GPU (and the NPU, and the TPU etc)
17. ” R E C E N T ” M AC H I N E L E A R N I N G I N N OVAT I O N S • Hardware improvements – The GPU (and the NPU, and the TPU etc) • Software improvements • TensorFlow (and TensorFlow.js), PyTorch
18. P R E V I O U S TA L K S O N M L @ J S - P O L A N D
19. ” R E C E N T ” M AC H I N E L E A R N I N G I N N OVAT I O N S • Hardware improvements – The GPU (and the NPU, and the TPU etc) • Software improvements • TensorFlow (and TensorFlow.js), PyTorch • WebGPU
20. W E B G P U B R O W S E R S U P P O RT
21. ” R E C E N T ” M AC H I N E L E A R N I N G I N N OVAT I O N S • Hardware improvements – The GPU (and the NPU, and the TPU etc) • Software improvements • TensorFlow (and TensorFlow.js), PyTorch • WebGPU • Technique improvements https://arxiv.org/pdf/1706.03762
22. W H Y B R I N G I N G A I TO T H E C L I E N T ?
23. W H Y B R I N G I N G A I TO T H E C L I E N T ? • No more network delay!
24. W H Y B R I N G I N G A I TO T H E C L I E N T ? • No more network delay! https://www.keycdn.com/support/network-latency
25. B O N U S – O F F L I N E S U P P O RT https://developer.chrome.com/blog/improved-pwa-offline-detection
26. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E https://github.com/tensorflow/tfjs- models/tree/master/hand-pose-detection
27. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E • We will use TensorFlow.js and MediaPipe
28. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E • We will use TensorFlow.js and MediaPipe • Developed by Google
29. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E • We will use TensorFlow.js and MediaPipe • Developed by Google • MediaPipe Models gives us tons of light-weight ML models for free
30. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E • We will use TensorFlow.js and MediaPipe • Developed by Google • MediaPipe Models gives us tons of light-weight ML models for free • Can be customizable through MediaPipe Model Maker
31. F A S T O N - D E V I C E M L W I T H T FJ S A N D M E D I A P I P E • We will use TensorFlow.js and MediaPipe • Developed by Google • MediaPipe Models gives us tons of light-weight ML models for free • Can be customizable through MediaPipe Model Maker • We will use the @mediapipe/hands library in this demo.
32. W H Y B R I N G I N G A I TO T H E C L I E N T ?
33. W H Y B R I N G I N G A I TO T H E F R O N T E N D ? • No more network delay! • Privacy https://pub.towardsai.net/43-memorag-rag-agent-rag-fusion-and-more-7b8463590747
34. P R I V A T E A I W I T H A S K - M Y- P D F https://github.com/nico-martin/ask-my-pdf
36. S C E N A R I O – P E R S O N W I T H C H R O N I C B A C K P A I N
37. S C E N A R I O – P E R S O N W I T H C H R O N I C B A C K P A I N
41. https://www.youtube.com/watch?v=VAfy26xs6e0
45. W H Y B R I N G I N G A I TO T H E F R O N T E N D ? • No more network delay! • Privacy • Cost
46. M A C H I N E L E A R N I N G A N D C O S T https://ux- news.com/google- announces-new-virtual- try-on-tools/
47. M A C H I N E L E A R N I N G A N D C O S T https://www.reddit.com/r/Programm erHumor/comments/1cxfjws/tenouto ftendentistsrecommend/?rdt=61385
48. ng-pet-cam • Heavily inspired by Jason Mayes pet-cam app • Uses @tensorflow- models/coco-ssd for object- detection • An app to monitor your pet 🐶 • Is live on https://ngpetcam.web.app and code can be accessed on https://github.com/markusingv arsson/aiayn-ng-pet-cam
49. S U M M A RY
50. S U M M A RY • Dedicated hardware is required for client-side machine learning
51. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense:
52. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense: • Smooth on-device performance
53. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense: • Smooth on-device performance • Offline Support
54. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense: • Smooth on-device performance • Offline Support • Privacy
55. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense: • Smooth on-device performance • Offline Support • Privacy • Enable ML when Server-Side ML is impractical (e.g. cost)
56. S U M M A RY • Dedicated hardware is required for client-side machine learning • When client-side machine learning makes sense: • Smooth on-device performance • Offline Support • Privacy • Enable ML when Server-Side ML is impractical (e.g. cost) • Optimistic future
57. G E T STA RT E D ?
58. G E T STA RT E D ?
59. G E T STA RT E D ?
60. G E T STA RT E D ?
61. C L I E N T- S I D E M A C H I N E L E A R N I N G B R I N G I N G A I T O T H E F R O N T E N D Connect with me!