Win the hearts of Chrome OS users - How to adapt your Android app

Android has been supporting pointer/mouse and keyboard inputs from almost day one. But mouses and keyboards have not really been a top priority in the past years, at least for most applications. Recently, a whole new category of devices - Chromebooks - puts these types of input back to the front of the stage. Built-in touchpads and keyboards, or free-form multi-window support are just two examples of what should be taken into account when publishing an Android app with Chromebook users in mind.

Despite the increasing success of Chromebooks, a lot of Android applications are not optimised and provide poor user experiences. The good news is that there are quick and easy actions that we can take to change that in our apps.

In this session, we will talk about two key points to optimize and improve the user experience and win the hearts of our users:
- How to provide more context by adapting the mouse cursor.
- How to navigate in an application without leaving the keyboard. We will share the tips and tricks we learnt while adapting edjing mix for Chromebooks: https://youtu.be/2pJwC8wdk88

Transcript

1. Win the hearts of Chrome OS users By MWM | April 2022 How to adapt your Android app Android Makers 2022 Today we will see why and how you should adapt your Android app for Chrome OS devices. At MWM, we already worked on this subject for one of our app, so we will talk about what we achieved. During this presentation we will see why a non optimized application leads to a bad user experience and how to fix it to win the hearts of Chrome OS users.
2. Why adapt your app for Chrome OS ? So now, i will try to convince you that you should consider adapting your app on Chrome OS 2
3. • Linux kernel Chrome OS First of all, what is Chrome OS ? • Gentoo linux based operatin system 3
4. • Linux kernel • proprietary licence Chrome OS ChromeSO is a Gentoo linux based operatin system • owned by Google • a part of code is open source and available in Chromium OS project 4
5. Chrome OS 5
6. Chrome OS 6
7. Chrome OS 7
8. Chrome OS 8
9. Chrome OS 9
10. Key numbers • 2020 : 30M • 2021 : 40M • ChromeOS > MacOS In 2020 the amount of chromebooks solds was about 30M. The same year, Chrome OS outsold MacOS We can explain this by the fact that google is targeting the student market, ChromeBooks are convinients for taking notes particulary with the evernote app and so Chrome OS is more and more used on American campus 10
11. Rating policy New Play store rating policy One more reason for considering Chrome OS app adaptation : • In april 2022, Google will launch a new rating policy on the Google play store. • Users will be able to see app rating depending on the device they use ; phone, tablet, watch or Chrome OS. • So if Chrome OS users have a bad experience on your app and give bad ratings, the ratings from other devices will not compensate it. 11
12. How adapt your app on ChromeOS edjing case Now that you want to adapt your app on chrome os, we will talk about how you can adapt it At MWM we worked on the chrome os adaptation of our app edjing For the rest of presentation we will take it as a reference. But first, let’s see what we achieved with edjing on ChromeBook. 12
14. Built-it keyboard Main difference with Android device is the physical keyboard The user may want to use it in order to interact with your app. 14
15. Keyboard shortcuts • Intercept key : override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { if (keyCode == KeyEvent.KEYCODE_W) { playPause(DISC_LEFT) return true } if (keyCode == KeyEvent.KEYCODE_O) { playPause(DISC_RIGHT) return true } return false } • A good practice is to use keyboard shortcuts. • On an activity, fragment, or view you can intercept keys by overriding onKeyUp. You wil be able to know each up actions on each keys. 15
16. Keyboard shortcuts • Intercept key : override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { if (keyCode == KeyEvent.KEYCODE_W) { playPause(DISC_LEFT) return true } if (keyCode == KeyEvent.KEYCODE_O) { playPause(DISC_RIGHT) return true } return false } • For exemple in edjing, 16
17. Keyboard shortcuts • Combine keys : override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { return when (keyCode) { KeyEvent.KEYCODE_S -> { if (event.isCtrlPressed) { save() true } else false } else false } } • There are several key combinations commmonly used on laptop, for exemple “Ctrl + Z” for undo. • For exemple “Ctrl + S” shortcut for saving 17
18. Keyboard shortcuts • Combine keys : override fun onKeyUp(keyCode: Int, event: KeyEvent): Boolean { return when (keyCode) { KeyEvent.KEYCODE_S -> { if (event.isCtrlPressed) { save() true } else false } else false } } • There are several key combinations commmonly used on laptop, for exemple “Ctrl + Z” for undo. • For exemple “Ctrl + S” shortcut for saving 18
19. Keyboard shortcuts • Input method for EditText : searchEditText.setOnKeyListener((v, keyCode, event) -> { if ((keyCode == KeyEvent.KEYCODE_ENTER) && (event.getAction() == KeyEvent.ACTION_DOWN)) { performSearch(..); return true; } return false; }); • When using an edittext with a hardware keyboard, a frustating behavior is to not start an action when typing the enter key but add a new line. This leads to a bad user experiance. • To resolve this we can set an onKeyListener on the edittext which allows us to know each actions on each keys 19
20. Keyboard shortcuts • Input method for EditText : searchEditText.setOnKeyListener((v, keyCode, event) -> { if ((keyCode == KeyEvent.KEYCODE_ENTER) && (event.getAction() == KeyEvent.ACTION_DOWN)) { performSearch(..); return true; } return false; }); • For exemple, in edjing we can start research on enter key pression. 20
21. Navigation keys • Directions : <Button android:id="@+id/buttonTopLeft" android:nextFocusRight="@id/buttonTopRight" android:nextFocusDown="@id/buttonBottomLeft" … /> <Button android:id="@+id/buttonTopRight" android:nextFocusLeft="@id/buttonTopLeft" android:nextFocusDown="@id/buttonBottomRight" … /> <Button android:id="@+id/buttonBottomLeft” android:nextFocusRight="@id/buttonBottomRight" android:nextFocusUp="@id/buttonTopLeft" … /> <Button android:id="@+id/buttonBottomRight" android:nextFocusLeft="@id/buttonBottomLeft" android:nextFocusUp="@id/buttonTopRight" … /> • An other good practice is to let the user use navigation keys in order to navigate through your app. • The operating system handle this natively but you may want to adapt this if you plan to optimise the user experiance. • We can do this by using nextFocus attributes on views in xml code 21
22. Navigation keys • Tab : <Button android:id="@+id/buttonFxLeft" android:nextFocusForward="@id/buttonLoopLeft" … /> <Button android:id="@+id/buttonLoopLeft" android:nextFocusForward="@id/buttonEqLeft" … /> <Button android:id="@+id/buttonEqLeft” android:nextFocusForward="@id/buttonHotCuesLeft" … /> <Button android:id="@+id/buttonHotCuesLeft" android:nextFocusForward="@id/buttonPlayPauseLeft" … /> • We can also use the TAB key • Also handle natively by using the xml definition order • We can improve the behavior by using nextfocus attribute on the xml code 22
23. Manage touchpad Now we will speak about the built-in touchpad and the pointer. 23
24. Provide more context by adapting the mouse cursor To explain the modifications we applied to our app, we will follow the flow of a user who wants to mix some tracks. • First, we want to display the tracks library screen. To do that, we have to click on the button inside the red square. 24
25. Single click -> PointerIcon.TYPE_HAND Provide more context by adapting the mouse cursor This button is a custom View. A single click on it displays the library. To give more context on the action to do on it, we change the “arrow” icon into a “hand” icon. 25
26. Provide more context by adapting the mouse cursor For classical “android.widget.Button”, we have nothing to do , it’s the normal behavior. This “PointerIcon change” should only be done for custom views. 26
27. <com.mwm.edjing.OpenTracksLibraryButton android:layout_width=”wrap_content” android:layout_height=”wrap_content” ... /> android:pointerIcon=”hand” Provide more context by adapting the mouse cursor The first way to change the pointer icon is inside a layout file by using “pointerIcon” attribute with one of the icons provided by the system. Here “hand” value. But it exists dozens of other icons. 27
28. openTracksLibraryButton.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) Provide more context by adapting the mouse cursor We can also change it programmatically. Call setPointerIcon method 28
29. Provide more context by adapting the mouse cursor openTracksLibraryButton.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) To get one of the system icon, we call PointerIcon.getSystemIcon. 29
30. Provide more context by adapting the mouse cursor openTracksLibraryButton.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) 1. We apply “hand” icon like in XML to do what we want. 2. It’s also possible to use any Drawable to create a custom pointer. val dollarBitmap = Bitmap.createScaledBitmap( BitmapFactory.decodeResource( this.resources, R.drawable.dollar_sign ), CURSOR_WIDTH, CURSOR_HEIGHT, false ) // Creating the pointer icon and sending clicks from the center of the mouse icon PointerIcon.create(dollarBitmap, (CURSOR_WIDTH/2).toFloat(), (CURSOR_HEIGHT/2).toFloat()) 30
31. Provide more context by adapting the mouse cursor You have to keep in mind: all views are rectangles. In our example, our button has a round shape but the pointer is changed when it is in the corners of the view. It’s not very nice 31
32. Provide more context by adapting the mouse cursor openTracksLibraryButton.setOnHoverListener { v, event -> if (isInsideCircle(event)) { v.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) } else { // restore arrow icon } } To improve that, we can be notified of pointer movement above our view by listening to OnHoverListener. 32
33. Provide more context by adapting the mouse cursor openTracksLibraryButton.setOnHoverListener { v, event -> if (isInsideCircle(event)) { v.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) } else { // restore arrow icon } } We receive MotionEvent and with coordinates, we can check if the event happened inside our circle. 33
34. Provide more context by adapting the mouse cursor openTracksLibraryButton.setOnHoverListener { v, event -> if (isInsideCircle(event)) { v.pointerIcon = PointerIcon.getSystemIcon( context, PointerIcon.TYPE_HAND ) } else { // restore arrow icon } } And after, applied the PointerIcon change. 34
35. Provide more context by adapting the mouse cursor Before to display the track library, I want to reduce the volume of the left side to prepare the platine. 35
36. Provide more context by adapting the mouse cursor Long click & move -> PointerIcon.TYPE_GRAB and PointerIcon.TYPE_GRABBING Here, you can see our custom view responsible of managing the volume. We can select the cursor and move it vertically. Like you can see, the Pointer icon has 2 different states: • when it’s above the view • the user selects the cursor We can’t use the XML attribute to do that because the scenario is too complex. So how to do that ? 1. We programmatically set the pointer to “TYPE_GRAB” -> open hand icon 2. Our view overrides “onTouchEvent method” to be notified of pointer movement and apply the volume change. 3. When motion event action is “ACTION_DOWN” and the click action occurred in the cursor area, we change the pointer to “TYPE_GRABBING” -> closed hand icon 4. And finally, we restore “TYPE_GRAB” when MotionEvent action is “ACTION_UP” or “ACTION_CANCEL”. Some scenarios need a little more code than just adding a XML attribute. 36
37. Manage right click on list items Ok, so, now I can clicked on the button to display the track library. In this new screen, I want to • display a menu on one of the list item. • And after that, I will click on an item to load a track. 37
38. Manage right click on list items Classical smartphone action to display menu: • click on overflow button. • long press on item list. Here we focus on 2 track items. It’s on the top item that I want to display a menu to access some features. • On smartphones, we are used to click on an overflow button or long click on a list item to display a menu. • On computer, we right click on an item to display a menu and it appears where the click occurred. Keep smartphone behavior on ChromeBook would be quite frustrating because it takes time to do this simple action. I have to move the cursor on a big distance to select the overflow button 38
39. Manage right click on list items trackListItem.setOnContextClickListener { view -> showContextMenu(view) true } To implement the computer behavior on a ChromeBook, I have to set an “OnContextClickListener” on my list items to be notified of the right click event. 39
40. Manage right click on list items trackItemList.setOnContextClickListener { view -> showContextMenu(view) true } After that, I can call my method to display a context menu. ⚠ I said ContextMenu and not PopupMenu. On smartphone, in edjing, we use PopupMenu to display the menu next to the overflow button. On Chromebook, we need a ContextMenu to display it at the position of the right click event... which is not possible with a PopupMenu because it takes an anchor view to be displayed next to it and not coordinates. ContextMenu allows it. 40
41. Cursor caption I was in the library and I clicked on a track item to load a new track. I come back on the platine and now I want to • Do a smooth transition between tracks. I have to interact with the view in the red rectangle: the crossfader. 41
42. Cursor caption A view can capture the cursor to get all cursor events. Its role is to manage if it’s the left disc or the right disc we can hear. But as a user, as a DJ, I don’t want to loose time to move the pointer to select the cursor, I want to move the crossfader now. To do that, I can “capture” the pointer. • Wherever the pointer is on the screen, a view can capture it in order to receive all motion events. • This “mode” can be triggered with a keyboard shortcut (click on “ctrl”, like PIerre explained before). • The fact to capture it make it disappears from the screen. • In this mode, the pointer can’t be stuck on the side of a screen if he goes to far in one direction -> used for video game with first personal camera • When you end this mode, pointer is displayed again at the same position it was before the capture. 42
43. Cursor caption crossfader.requestPointerCapture() crossfader.releasePointerCapture() To enter in this mode, the view, which needs to receive motion events, has to call requestPointerCapture. To leave this mode, the view can call releasePointerCapture. 43
44. Cursor caption override fun onCapturedPointerEvent( motionEvent: MotionEvent ): Boolean { moveCrossfader(motionEvent.x) // delta since last event return true } My custom view has to override onCapturePointerEvent to be notified of pointer movements. 44
45. Cursor caption override fun onCapturedPointerEvent( motionEvent: MotionEvent ): Boolean { moveCrossfader(motionEvent.x) // delta since last event return true } Coordinates inside the MotionEvent are not coordinate on the screen, the pointer is not displayed anymore on it, they are relative to the movement since the last event. With this delta value, I can move the crossfader and change the music produced by the app. 45
46. (+) Quick and easy to implement (+) Improves the user experience (+) Get better user ratings (+) Increases app usage and number of downloads Conclusion 46
47. Conclusion Windows 11 now supports Android applications One last argument to motivate you to adapt your app for desktop: is the release of Windows eleven which can run Android applications through the Amazon Store. Our applications can now run on several billions of desktops and laptops and we should consider these users to increase our user base. 47
48. Sources - Android documentation: Optimize apps for Chrome OS https://developer.android.com/topic/arc/optimizing - Google I/O 2021: Input matters for Chrome OS https://io.google/2021/session/99ebd796-0ee2-44bf-9b55-3f9cdf7 75532?lng=en - Android dev summit 2021: Enable great input support for all devices https://www.youtube.com/watch?v=piLEZYTc_4g&list=PLWz5rJ2 EKKc99PA-mKk2rz0jYXshN94sM&index=5 Here are some of the sources we used to prepare this presentation. A lot of this content (documentation and videos) has been created by Emilie Roberts, a developer advocate for ChromeOS and I would like to thanks her for this great work. 48
49. Questions? By MWM | April 2022 Android Makers 2022 To go further: PointerIcon change available since API 24 Pointer capture available since API 26 • What is the Android version running on Chrome OS ? ◦ For now, Chrome OS embeds an Android 11 (API 31) emulator • Resizable • One thing we didn’t talked about is the fact that Android application are forced to be resizable by the user. It means, you must be sure to correctly manage configuration change to restore the state of your view when the size change. • Due to resizable feature, we should make our layouts responsive. • What is the hardware of a ChromeBook ◦ touchable / not touchable screen ◦ Processor X86 -> If you have native code, I mean C/C++ code, you will have to build X86 “.so” file ◦ Some can be converted to tablet but not all (keyboard removed 49
50. ◦ or fully opene 360°) • Some ChromeBook don’t have: ◦ GPS ◦ some sensors. ◦ Non touchable screen (by default, when not specified, all applications can run on non touchable device) Make them required in your manifest if you really need these sensors. • Our ChromeBook rating is the highest (smartphone, tablet, Chromebook) on the last 28 days. (+0.05 compared to smartphone, + 0.1 compared to tablet). Compared to peers, we get +0.5. • Activities are displayed in a window system. To avoid any problem (security or failure to understand what happened), the back Window is not fully transparent. That mean, if the root Activity is “transparent”, we won’t see through it.