IoT with the Google Assistant

Introductory talk held at the Voice meets IoT Hackathon:
https://www.meetup.com/de-DE/Munich-Voice-Interfaces/events/246064221/

3. Voice UX
4. Voice UX • Voice UX is special – Discoverability of app – Discoverability of features – Onboarding • Avoid repetitions – Past behaviour of user
5. Voice UX • You need to think about your app‘s persona • Try to keep to user expectations of a dialog – Script it – Practise script with a partner
6. Voice UX • Some recommendations – provide feedback – elicit answers – Give helpful comments – Repair broken conversations – Provide help – Avoid the usage of the same phrases over and over again
7. Voice UX • Loads of UX recommendations by Google • See the explanations about the IO Assistant app
8. Developing
10. Assistant SDK • If you want to add the Assistant to your device • Enables you to control some aspects of the device
11. Assistant SDK • Communicate with Google‘s backend using gRPC • Change the trigger for the assistant – No direct support for a different hotword (use snowboy or pocketSphinx) • On device support for different traits – Brightness, color... • Either use a lib (Python) or the service directly • For proper devices; you must (!) contact Google; it‘s for development only
12. Actions on Google – Smart Home https://en.wikipedia.org/wiki/Philips_Hue
13. Actions on Google – Smart Home • If you want your users to control your device using the Google Assistant – „Turn on the light in the living room“
14. Actions on Google – Smart Home • Consists of two parts: – OAuth2 server to authenticate users and issue a token – Some backend to • Query the state of the device • Trigger changes to the device • User adds a device via Google Home app – Here the Oauth flow comes into play • Digg into sample app – https://github.com/actions-on-google/smart-home-nodejs
19. Actions https://assistant.google.com/services/a/uid/0000008011b5e5f2?hl=en-US
20. Actions • If you want your users to use a voice app • „Ask MVV, when‘s the next bus due at Hofmannstraße“ • Use one of those options – Actions SDK – A service like Dialogflow
21. Actions • If you want your users to use a voice app • „Ask MVV, when‘s the next bus due at Hofmannstraße“ • Use Dialogflow
22. Action SDK
24. Dialogflow
26. App-Structure
27. Base concepts • Intents – Predefined and custom ones – Special ones for smart home • Entities • Fulfillment • Actions • Context
28. Permissions • For some things you need the user‘s consent – User name – Fine or coarse location – Push actions – Repeated actions
29. Visual Elements • Only on devices with visual interface – Cards with images, text and links – List selectors – Carousel – Suggestion chips
30. Fulfillment • It‘s just JSON going back and forth • Use Cloud Functions / Lambda or whatever you like • You can use ngrok to make your localhost server available
31. Demo
32. Cross Platform
33. Libs • AssitantJS https://github.com/webcomputing/AssistantJS • Jovo Framework https://github.com/jovotech/jovo-framework-nodejs
34. Limitations
35. Limitations of the Platform • No implicit intents – Probably the main beef on G+ • Limited options for visual UI elements • SSML missing important features – Though some have been added recently • Testing • Account switching is a PITA, language switching as well • No private actions (store apps, inhouse apps)
36. But • Google is eagerly changing and adding stuff – what might be a limitation now might work tomorrow
37. There‘s more out there
38. Resources • Talks at Google I/O https://events.google.com/io/schedule/?section=may-18&track=assistant https://events.google.com/io/schedule/?section=may-19&track=assistant • Actions on Google Dev Home: https://developers.google.com/actions/ • Actions on Google Console: https://console.actions.google.com • Dialogflow Console https://console.dialogflow.com