Hey Google, how can I build something like Circle to search?

Some premium devices offer a cool feature called Circle to search. It lets you identify objects, songs, and more just by pointing your phone's camera. Now, what might be needed to build something like that? What technologies, libraries and tools could be involved? Let's find out. In this talk, I will introduce you to key components and show how to build a simplified version of Circle to Search.

Transcript

1. Thomas Künneth Hey Google, how can I build something like Circle to search? Are you sure you want to say Hey Google in a room full of Android devices?
2. https://github.com/tkuenneth/ viewfAInder Invoke the Digital Assistant Take a picture Draw on screen Ask Gemini Appreciate the answer
3. ➢Taking pictures (sort of) ➢On a … role ➢Drawing on screen ➢Using Gemini Topics
4. Required for CameraX Make sure the preview shows something Main UI including camera preview
5. Setting up the ImageAnalysis use case Setting up the Preview use case Putting all pieces together
7. • ImageAnalysis is great for repeatedly doing something with incoming images • Not a real capture • viewfAInder processes the image only upon clicking an icon (so, why is it using ImageAnalysis?)
8. ➢Taking pictures (sort of) ➢On a … role ➢Drawing on screen ➢Using Gemini Topics
9. • Unique name within the system associated with certain privileges / capabilities • Some existing roles: ROLE_ASSISTANT, ROLE_BROWSER, ROLE_CALL_REDIRECTION, ROLE_CALL_SCREENING, ROLE_DIALER, ROLE_EMERGENCY, ROLE_HOME, ROLE_NOTES, ROLE_SMS, ROLE_WALLET • Check availability of a role with isRoleAvailable What is a role?
10. • While multiple applications may qualify for a role, only a subset can become role holders • To qualify for a role, apps must meet certain requirements, for example defining certain components in the manifest • Requirements can be found in androidx.core.role.RoleManagerCompat
11. • Name and documentation for roles that might be available in the system • Latest update: December 15, 2021 • Release Candidate: 1.1.0-rc01 (androidx.core:core- role:1.1.0-rc01) • Package: androidx.core.role Jetpack Core-Role
12. One of the smallest Jetpack libraries
14. https://github.com/tkuenneth/ unclutter-your-jetpack
15. Handle the result of role requests • Get a RoleManager instance • Check role availability (isRoleAvailable) • Check if role is held (isRoleHeld) Request the role
16. Request the role Update upon state changes
17. Why did you explain this using ROLE_BROWSER?
19. Update the role status Initial setup Jump into system settings
21. • Since the last Jetpack library update, the framework class RoleManager received additions, for example ROLE_NOTES (API level 34) • RoleManagerCompat lacks factory functions that ensure a consistent behavior on older Android versions (RoleManager was added with API level 29) • In its current form Core-Role looks more than some sort of documentation than a component
22. ➢Taking pictures (sort of) ➢On a … role ➢Drawing on screen ➢Using Gemini Topics
24. Handle pointer events Draw a closed path Draw individual circles
25. Convert points to match image size Create a Path from the scaled points Create a Canvas based on a Bitmap Draw the Path onto the Canvas Ask Gemini
26. ➢Taking pictures (sort of) ➢On a … role ➢Drawing on screen ➢Using Gemini Topics
31. • Default format of Gemini answers is Markdown • Can be changed, but is nice if the answers are shown on screen • Cool Open-Source library by Jeziel Lago: compose-markdown (https://github.com/jeziellago/compose-markdown)
32. • Why not taking advantage of the continuous image stream? • How about using on-device LLMs? Pretty cool, but …
33. https://github.com/tkuenneth/ viewfAInder https://github.com/tkuenneth/ unclutter-your-jetpack
34. Thank you @tkuenneth.bsky.social @tkuenneth @tkuenneth