Has your puppy joined the dark side?

Machine learning is permeating mobile and web app development. To create modern, interactive apps that can evolve over time, it's a good idea to learn how to build and use machine learning models. What better way to super-charge an App than to integrate some AI? I'll step you through how to build a web app, embed an ML model that is trained using the Custom Vision AI service to tell if the furry animals living in your house are Shihtzu or Ewoks. In this talk I'll share my experience of building a progressive web app, integrating ML and adding in a bit of Continuous Integration/Continuous Delivery (CI/CD) to push updates to GitHub. By the end of the session you should be able to build your own app [Note: There will be gratuitous puppy and Ewok pictures throughout].
You can play with the App here: https://aka.ms/ewok
You can find the code here: https://aka.ms/GitHubEwok
You can find the deck with the side-by-side code comparison: https://aka.ms/PWAChange
Demo of the Custom Vision + ML here: https://youtu.be/_S5rYsLMd2s
Demo of the push from VSCode to GitHub and Azure: https://youtu.be/J9nyKkN6sWk
You can find a LiveCode video here: https://youtu.be/ZaV6tGFNL8g
Technorama BE was not recorded

Transcript

1. Has your puppy joined the dark side? Michelle Sandford Microsoft Developer Engagement Lead Emerging Communities (Students/Career-Change/Data- Science+AI)
2. Michelle Sandford Developer Engagement Lead www.linkedin.com/in/michellesandford/ https://github.com/codess-aus Twitter: @codess_aus Instagram: @codess_aus TikTok: @codess_aus #Technorama
3. #Technorama @codess_aus
4. These are my puppies #Technorama
5. Those are not Puppies Bing Image Search #Technorama @codess_aus
6. Choices: Uh, what kind of App do I build? Low Code/No Code Native Web – Static/Dynamic Progressive Web App Mobile Can we just skip to the good part? #Technorama @codess_aus
7. Progressive Web Apps Discoverable Installable Safe Re-engageable Offline Responsive Linkable Progressive #Technorama @codess_aus
8. Progressive Web App Concept HTTPS Web App Manifest Service Worker #Technorama @codess_aus
9. Scaffold the App #Technorama @codess_aus
10. Has Your Puppy Joined the Dark Side? #Technorama @codess_aus
11. Train your machine learning model #Technorama @codess_aus
12. Conceptual Overview HTTPS Web App Manifest Service Worker CICD #Technorama @codess_aus
13. #Technorama @codess_aus Step 1 – Retitle the App
14. App.vue <template> <div id="app"> <img alt="Vue logo" src="./assets/logo.png"> <HelloWorld msg="Welcome to Your Vue.js App"/> </div> </template> <script> import HelloWorld from './components/HelloWorld.vue' export default { name: 'App', components: { HelloWorld } } </script> <template> <div id="app"> <DetectImage msg=“Has your puppy joined the dark side"/> </div> </template> <script> import DetectImage from './components/DetectImage.vue' export default { name: 'App', components: { DetectImage } } </script> Rename HelloWorld.vue and change all references to it #Technorama @codess_aus
15. Step 2 – Replace image and links #Technorama @codess_aus
16. Create images Folder and replace links with image and button DetectImage.vue <template> <div class="hello"> <h1>{{ msg }}</h1> <p> For a guide and recipes on how to configure / customize this project,<br> check out the <a href="https://cli.vuejs.org" target="_blank" rel="noopener">vue-cli documentation</a>. </p> <h3>Installed CLI Plugins</h3> <ul> <li><a href="https://github.com/vuejs/vue- cli/tree/dev/packages/%40vue/cli-plugin-babel" target="_blank" rel="noopener">babel</a></li> <template> <div class="hello"> <h1>{{ msg }}</h1> <!-- Display an image --> <div> <img class="image" ref="img" :src="require('../assets/images/' + getImgIndex + '.jpg')"/> </div> <!-- Display a button --> <div> <button class="button" @click="next()" :disabled="disable">Next</button> </div> </div> </template> #Technorama @codess_aus
17. Add a data method DetectImage.vue <script> export default { name: 'HelloWorld', props: { msg: String } } </script> <script> export default { name: "DetectImage", props: { msg: String }, // create a placeholder for the data data() { return { image: 0, numImages: 21 }; }, }; </script> #Technorama @codess_aus
18. Create a Computed Property to return the index of an image DetectImage.vue data() { return { image: 0, numImages: 21 }; }, data() { return { image: 0, numImages: 21 }; }, // return the index of the image computed: { getImgIndex() { return this.image.toString(); }, disable() { if (this.image == this.numImages) { return true; } else return false; }, }, } </script> #Technorama @codess_aus
19. Create a Methods Object to capture user interaction DetectImage.vue // return the index of the image computed: { getImgIndex() { return this.image.toString(); }, disable() { if (this.image == this.numImages) { return true; } else return false; } }, computed: { getImgIndex() { return this.image.toString(); }, disable() { if (this.image == this.numImages) { return true; } else return false; } }, // capture user interaction methods: { next() { this.image++; this.predictions = []; setTimeout(this.predict, 500); } } }; </script> #Technorama @codess_aus
20. Let’s update those styles DetectImage.vue <!-- Add "scoped" attribute to limit CSS to this component only --> <style scoped> h3 { margin: 40px 0 0; } ul { list-style-type: none; padding: 0; } li { display: inline-block; margin: 0 10px; } a { color: #42b983; } </style> <!-- bling bling baby --> <style scoped> h3 { margin: 40px 0 0; } .image { min-height: 500px; max-height: 500px; max-width: 100%; } .button { width: 200px; height: 50px; border-radius: 5px; background-color: blueviolet; color: white; font-size: 20pt; margin: 10px; } .button:disabled, .button[disabled] { border: 1px solid #999999; background-color: #cccccc; color: #666666; } </style> #Technorama @codess_aus
21. Install the custom vision package Package.json "dependencies": { "core-js": "^3.6.5", "register-service-worker": "^1.7.1", "vue": "^2.6.11" }, "devDependencies": { "@vue/cli-plugin-babel": "~4.5.15", "@vue/cli-plugin-eslint": "~4.5.15", "@vue/cli-plugin-pwa": "~4.5.15", "@vue/cli-service": "~4.5.15", "babel-eslint": "^10.1.0", "eslint": "^6.7.2", "eslint-plugin-vue": "^6.2.2", "vue-template-compiler": "^2.6.11" }, "dependencies": { "core-js": "^3.6.5", "register-service-worker": "^1.7.1", "customvision-tfjs": "^1.0.1", "vue": "^2.6.11" }, "devDependencies": { "@vue/cli-plugin-babel": "~4.5.15", "@vue/cli-plugin-eslint": "~4.5.15", "@vue/cli-plugin-pwa": "~4.5.15", "@vue/cli-service": "~4.5.15", "babel-eslint": "^10.1.0", "eslint": "^6.7.2", "eslint-plugin-vue": "^6.2.2", "raw-loader": "^4.0.0", "webpack-cli": "^3.3.10", "vue-template-compiler": "^2.6.11" }, #Technorama @codess_aus
22. Integrate the custom vision package DetectImage.vue <script> export default { name: "DetectImage", props: { msg: String }, <script> //integrate the customvision packages import * as cvstfjs from "customvision-tfjs"; import labels from "raw-loader!../../public/models/labels.txt"; export default { name: "DetectImage", props: { msg: String }, #Technorama @codess_aus
23. Read the labels.txt file Webpack.config.js module.exports = { module: { rules: [ { test: /\.txt$/i, use: 'raw-loader', }, ], }, }; #Technorama @codess_aus
24. Complete the Data elements section DetectImage.vue // create a placeholder for the data data() { return { image: 0, numImages: 21 }; }, // data elements data() { return { labels: labels, model: null, predictions: [], image: 0, numImages: 21 }; }, #Technorama @codess_aus
25. Add a mounted lifecycle hook DetectImage.vue // return the index of the image computed: { getImgIndex() { return this.image.toString(); }, disable() { if (this.image == this.numImages) { return true; } else return false; } }, // return the index of the image computed: { getImgIndex() { return this.image.toString(); }, disable() { if (this.image == this.numImages) { return true; } else return false; } }, // load the model async mounted() { this.image++; //load up a new model this.model = new cvstfjs.ClassificationModel(); await this.model.loadModelAsync("models/model.json"); //parse labels this.labels = labels.split("\n").map(e => { return e.trim(); }); //run prediction this.predict(); }, #Technorama @codess_aus
26. Add the asynchronous method DetectImage.vue // capture user interaction methods: { next() { this.image++; this.predictions = []; setTimeout(this.predict, 500); } } }; </script> methods: { async predict() { //execute inference let prediction = await this.model.executeAsync(this.$refs.img); let label = prediction[0]; //build up a predictions object by parsing details to labels and probability this.predictions = label.map((p, i) => { return { index: i, label: this.labels[i], probability: p * 100 }; }); }, // capture user interaction next() { this.image++; this.predictions = []; //this.predict(); setTimeout(this.predict, 500); } } }; </script> }, #Technorama @codess_aus
27. Show the confidence level in the UI DetectImage.vue <template> <div class="hello"> <h1>{{ msg }}</h1> <!-- Display an image --> <div> <img class="image" ref="img" :src="require('../assets/images/' + getImgIndex + '.jpg')" /> </div> <!-- Display a button --> <div> <button class="button" @click="next()" :disabled="disable">Next</button> </div> </div> </template> <template> <div class="hello"> <h1>{{ msg }}</h1> <div> <img class="image" ref="img" :src="require('../assets/images/' + getImgIndex + '.jpg')" /> </div> <div> <button class="button" @click="next()" :disabled="disable">Next</button> </div> <!-- display the interactions in the UI --> <div v-for="pred in predictions" :key="pred.index">{{ pred.label }}: {{ pred.probability.toFixed(0) + '%' }}</div> <div v-if="!predictions.length">hmm.....</div> </div> </template> #Technorama @codess_aus
28. Run on windows Web.config <?xml version="1.0" encoding="utf-8"?> <configuration> <system.webServer> <staticContent> <remove fileExtension=".json"/> <mimeMap fileExtension=".json" mimeType="application/json"/> </staticContent> </system.webServer> </configuration> #Technorama @codess_aus
29. GitHub Actions #Technorama @codess_aus
30. Create a Static Web App and Deploy to Azure with GitHub Actions #Technorama @codess_aus
31. Static Web Apps #Technorama @codess_aus
32. https://lively-ocean-00408a000.1.azurestaticapps.net/ #Technorama @codess_aus
33. Resources Code https://aka.ms/GitHubEwok Ewok or Chouleke App https://aka.ms/ewok Side by Side Code Changes https://aka.ms/PWAChange MS Learn Path (long version) https://aka.ms/LearnPWA #Technorama @codess_aus
34. Michelle Sandford Developer Engagement Lead www.linkedin.com/in/michellesandford/ https://github.com/codess-aus Twitter: @codess_aus Instagram: @codess_aus TikTok: @codess_aus #Technorama