Screenshot Testing for Android: Which Library is Right for You?

Slides of my lighting talk "Screenshot Testing for Android: Which Library is Right for You?" at Droidcon Lisbon 2023
-------------------------------------------------------------------------------
In recent years, screenshot testing has become an increasingly popular way to automate UI testing in Android applications, with new libraries emerging regularly. That makes hard to determine which one is the best fit for your project.

In this tech-talk, we'll explore the top screenshot testing libraries, from on-device to JVM-based libraries, including the newest player in the game: Roborazzi, based on Google's Robolectric.

We'll dive deep into each library's unique features and compare their pros and cons to help you make an informed decision about which one is right for your project.

Whether you're new to screenshot testing or have experience with it, this tech-talk will take your UI testing skills to the next level. You'll leave this talk with the knowledge to confidently choose the best screenshot testing library for your projects. Don't miss out on this opportunity to become a screenshot testing Ninja in Android!
---------------------------------------------------------------
Repos of the lighting talk:
https://github.com/sergio-sastre/Android-screenshot-testing-playground
https://github.com/sergio-sastre/AndroidUiTestingUtils

Transcript

1. BY SERGIO SASTRE SCREENSHOT TESTING WHICH LIBRARY IS RIGHT FOR YOU? FOR ANDROID
2. WHAT Screenshot testing is @Gio_Sastre
3. Special type of UI tests take a screenshot of it, compare it to its reference we inflate a UI component, where and @Gio_Sastre
4. WORKS HOW Screenshot testing @Gio_Sastre
5. record AND verify @Gio_Sastre
6. write test CI @Gio_Sastre <code> +
7. record CI @Gio_Sastre
8. = CI @Gio_Sastre
9. CI new <code> PR + @Gio_Sastre
10. approve CI @Gio_Sastre
11. merge CI @Gio_Sastre
12. CI new <code> PR verify @Gio_Sastre
13. CI new <code> PR @Gio_Sastre
14. CI new <code> PR @Gio_Sastre
15. CI new <code> PR @Gio_Sastre
16. CI new <code> PR @Gio_Sastre
17. EXIST? Which Screenshot testing libraries @Gio_Sastre
18. Screenshot testing libraries android-testify Paparazzi screenshot-tests 
 for-android Shot Dropshots Snappy Roborazzi @Gio_Sastre
19. on-device Screenshot testing libraries @Gio_Sastre
20. jvm layoutlib robolectric robolectric * Needs to be con f igured * Screenshot testing libraries @Gio_Sastre
21. in this talk jvm jvm on-device Screenshot testing libraries @Gio_Sastre on-device on-device
22. On-device Screenshot testing @Gio_Sastre
23. Dropshots Shot tolerance record & verify gradle managed devices test reports general value for all tests only from CLI @Gio_Sastre standard xml configuration problematic Windows support Kotlin gradle scripts ⚠ ⚠ super easy per test from CLI & AS very nice HTML with diffs not supported not supported + diffs android-testify easy per test CLI or AS Plugin supported YAML + diffs (experimental) max 1 snapshot per test
24. Dropshots Shot tolerance record & verify gradle managed devices test reports general value for all tests only from CLI @Gio_Sastre standard xml configuration problematic Windows support Kotlin gradle scripts ⚠ ⚠ super easy per test from CLI & AS very nice HTML with diffs not supported not supported + diffs android-testify easy per test CLI or AS Plugin supported YAML + diffs (experimental) max 1 snapshot per test
25. Jvm Screenshot testing @Gio_Sastre
26. Paparazzi Windows OS not supported Roborazzi Compose Multiplatform (as desktop) Dynamic Feature Modules not supported supported @Gio_Sastre supported 2 1 Due to Robolectric Native Graphics 2 Due to Robolectric supported 2 Bazel record & verify from CLI & AS only from CLI unofficial AS plugin supported 1 Via Skiko or not supported 3 3 not supported
27. Paparazzi Windows OS not supported Roborazzi Compose Multiplatform (as desktop) Dynamic Feature Modules not supported supported @Gio_Sastre supported 2 1 Due to Robolectric Native Graphics 2 Due to Robolectric supported 2 Bazel record & verify from CLI & AS only from CLI unofficial AS plugin supported 1 Via Skiko or not supported 3 3 not supported
28. Paparazzi complex animations (views) Side effects (compose) render elevation not supported Roborazzi different 
 API levels supported not supported not supported not fully supported not supported supported @Gio_Sastre supported2 1 2 1 Due to layoutlib 2 Due to Robolectric 2 supported supported2 2 Activities 
 & Fragments
29. Paparazzi complex animations (views) Side effects (compose) render elevation not supported Roborazzi different 
 API levels supported not supported not supported not fully supported not supported supported @Gio_Sastre supported2 1 2 1 Due to layoutlib 2 Due to Robolectric 2 supported supported2 2 Activities 
 & Fragments
30. DIFFERENT CONFIGS? SCREENSHOT TESTS under @Gio_Sastre
31. @Gio_Sastre Configuration matters 1 3 Locale UI Mode 2 Orientation 4 6 Custom themes Display Size 5 Font Size
32. Settings Display Font size > > Font Size @Gio_Sastre
33. @Gio_Sastre Font Size
34. Display Size Settings Display Display size > > @Gio_Sastre
35. Display Size @Gio_Sastre
36. Font Size vs. Display Size @Gio_Sastre
37. Font Size + Display Size @Gio_Sastre
38. MANY CONFIGS? HOW to screenshot test so @Gio_Sastre
39. Android Ui Testing Utils @Gio_Sastre 1 Screenshot Testing under different configurations https://github.com/sergio-sastre/AndroidUiTestingUtils 2 Cross-library screenshot testing
40. @Gio_Sastre possible Display size Font size + Locale UI Mode Orientation out of the box out of the box out of the box 1 via updateConfiguration() possible switch height & width 1 Activities under test must be open, otherwise still possible with AndroidUiTestingUtils out of the box Custom theme out of the box possible via updateConfiguration() out of the box out of the box out of the box robolectric based
41. JVM ? ON-DEVICE or @Gio_Sastre
42. complex animations (views) Side effects (compose) render elevation not supported different 
 API levels Activity & Fragment not supported not supported supported not fully supported not supported supported @Gio_Sastre supported2 1 2 1 Due to layoutlib 2 Due to Robolectric supported supported 2 robolectric based 2 2
43. supported supported supported supported supported complex animations (views) Side effects (compose) render elevation not supported different 
 API levels Activity & Fragment not supported not supported supported not fully supported not supported supported @Gio_Sastre supported2 1 2 1 Due to layoutlib 2 Due to Robolectric supported supported 2 robolectric based 2 2
44. @Gio_Sastre * Only possible with Paparazzi and Roborazzi On-device on-device vs. jvm Best fidelity Foldable support Special configuration (adb) Jvm Very fast Less flaky Accessibility services * (no issues with emulators)
45. Resources 1 @Gio_Sastre Github Repo https://github.com/sergio-sastre/Android-screenshot-testing-playground 2 Screenshot testing companion library https://github.com/sergio-sastre/AndroidUiTestingUtils
46. attention for your THANKS
47. GitHub @SergioSastre sergio-sastre-florez @sergio-sastre @Gio_Sastre Hashnode blogs Sergio Sastre Flórez Lead & Senior Android developer appdev.de Linkedin