Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mastering Screenshot Testing for Android Apps

Bing
September 12, 2024
87

Mastering Screenshot Testing for Android Apps

Screenshot testing helps in verifying that your UI renders correctly across various devices and configurations. Also, can directly display the UI difference after any code changes.

In this talk, we will dive into the world of screenshot testing using popular tools like paparazzi, roborazzi and Google’s experimental plugin: Compose Preview Screenshot Testing.

Starting with the basics, we’ll explore the implementation of screenshot tests, how to set up and configure these tools, and integrate them into CI pipeline. I’ll share experience about weird issues our team faced and how to address them, and best practices for testing adaptive layouts, ensuring your app looks great on any screen size.

As Compose Preview Screenshot Testing is still in Alpha phase, I will catch up with the issues/pain point under disucssing, and share an ideal world of screenshot testing.

Bing

September 12, 2024
Tweet

Transcript

  1. Slide QR Code Sample App QR Code Sample app link:

    https://github.com/bingningO/screenshot- testing-sample-app 2
  2. Self-Intro Bing X: Bing76869213 Github: bingningO Senior Android Engineer at

    From China! Favorites: Anime, Snowboarding, Tennis, Hiking 3
  3. UI Change in big refactoring? 9 • Multi-Activities -> Single

    Activity yurihondo: Road to Single Activity at Hedgehog 9/13 14:20~ • Material Design M2 -> M3 Planning…
  4. Screenshot Testing with other Testing Unit Testing UI Testing E2E

    Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing 13
  5. Screenshot Testing with other Testing Unit Testing UI Testing E2E

    Testing Screenshot Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing Ensure visual consistency 14
  6. Key Benefits 2 Automation of UI Verification 1 Visual Regression

    Detection 3 Consistency Across Devices 4 Enhanced Confidence in Refactoring 15
  7. Agenda 01 | Implementation of Screenshot Testing ◦ Roborazzi ◦

    Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ◦ Test for Adaptive Layout ◦ Test for Animations 16
  8. Roborazzi • An open-source library • Introduced by nowinandroid, droidkaigiApp2023,

    droidkaigiApp2024 • Works with Robolectric • Render previews: by Robolectric Native Graphics 21
  9. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7,

    ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 23
  10. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7,

    ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } Configure 24
  11. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7,

    ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 25 Test Rule
  12. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7,

    ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 26 Capture screenshot
  13. Roborazzi – Single Test 27 Record Record current screenshots as

    reference images Compare Record current screenshots as new images, and compare with reference ones
  14. What if, we want to run screenshot testing for all

    composables? Don't try to write all tests by hands 30
  15. Roborazzi – Automatic Testing • Get all @Preview list ◦

    Showkase • Take screenshot for them ◦ Roborazzi 32
  16. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) {

    @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 33
  17. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) {

    @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 34 Configure
  18. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) {

    @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 35 Showkase: Get component list Showkase: Provide component list
  19. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) {

    @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 36 Capture screenshot
  20. • By Google, Experimental, mentioned in Google IO 2024 •

    Works with Compose, automatically test all previews • Works as easy as composable previews CPST refer: google android developer document 39
  21. • By Google, Experimental, mentioned in Google IO 2024 •

    Works with Compose, automatically test all previews • Works as easy as composable previews CPST refer: google android developer document 40
  22. 43 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso

    Mocks Hamcrest Junit 5 Instrumentation No More Test Code
  23. CPST - Setup refer: google android developer document • Ensure

    Environment • Configure gradle • Add @Preview under ./src/screenshotTest/ 44
  24. CPST - Setup refer: google android developer document • Ensure

    Environment • Configure gradle • Add @Preview under ./src/screenshotTest/ 45 DONE!
  25. CPST - More Discussion refer: issuetracker.google • Feature request ◦

    Support @Preview in main source set, link ◦ Customize configuration for screenshot report ◦ Support for more complex composables ◦ Support for Enhanced integration with other testing frameworks • Other ◦ The future of Compose preview screenshot testing v.s. Current solutions (e.g. Paparazzi, Roborazzi…), link1 , link2 47
  26. Compare Common Feature Strengths Limitations Roborazzi • Support for Jetpack

    Compose • Capture and validate the UI difference including 1-dp change Flexiable • Both Compose and traditional Views • Support customize • Requires setup and integration • May have a steeper learning curve Compose Preview Screenshots Testing Fast • Integrated with the IDE • Compatible with Compose • Experimental • Limited to static @Preview composables 48
  27. Screenshot Gradle Plugin 52 Configure for Multi-Module app ui domain

    network local database Data module Screenshot Testing ShowKase @Preview
  28. 53 Configure for CI New pull request New merge into

    main Record reference images refer: the companion branch approach Compare new changes Comment in the PR
  29. 55 Running Status • Started over 4 months • 520

    images, CI jobs take around 10 min in busy days
  30. 57 Feedbacks • Good ◦ Easy to see UI change

    ◦ Give confidence to PR owners ◦ Good preparation for big refactoring • Encountered Issues ◦ Please check my speech in Shibuya.apk
  31. Test for Adaptive Layout 63 • Multiple device types •

    Orientations • Themes • Fonts • … Saiki Iijima: 使って知るCustomLayout. vs DailyScheduler at Hedgehog 9/13 15:20~ Custom Adaptive Layout Speech:
  32. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor

    = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } CPST: natively supports it @Preview Code: Just add annotations 64
  33. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor

    = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } The results nearly the same by @PreviewScreenS izes issue link 😫 67 CPST: issues
  34. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor

    = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } Got build error by setting device type 😫, issue link 68 CPST: issues
  35. @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val

    filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 69 Roborazzi: set different test environments refer: robolectric#device configuration
  36. @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val

    filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 70 Roborazzi: set different test environments More Details
  37. Test for Adaptive Layout Summary • Compose Preview Screenshot Testing

    ◦ Good: Natively supports adaptive layouts with simple annotations ◦ Pain: Still experimental, with unresolved bugs 73
  38. Test for Adaptive Layout Summary • Roborazzi ◦ Good: Highly

    configurable for different test environments ◦ Pain: Requires manual setup and has some limitations 74
  39. Animation Type • Android build-in animations ◦ e.g. progressbar, shared

    element transitions • Value-based animations ◦ e.g. translating, rotating, alpha changing • Animated vector images ◦ e.g. Lottie Animation 79
  40. • Control the animation state manually • Capture screenshots at

    specific points ——————————————————————————— • Roborazzi ◦ Set state in single test code ◦ Or add state in @Preview • Compose Preview Testing ◦ add state in @Preview Challenge 1: Capture Animation States 80
  41. @Test fun loadingContentTest_start() { composeTestRule.setContent { // set different progress

    state LoadingContent(progress = 0.0f) } composeTestRule.onRoot().captureRoboImage() } 81 Challenge 1: Test Code
  42. • Lottie Compose Code: set iterations as 1 • Test

    Code: ◦ wait until the composition is idle ◦ take screenshot ——————————————————————————— • Only works for Roborazzi • Compose Screenshot Testing issue link 83 Challenge 2: Capture Lottie Animation
  43. val progress by animateLottieCompositionAsState( ... iterations = if (Build.FINGERPRINT ==

    "robolectric") 1 else LottieConstants.IterateForever, ) LottieAnimation( ... composition = composition, progress = { progress }, ) 84 Challenge 2: Lottie Compose Code
  44. @Test fun animateLottieTest() { composeTestRule.setContent { AnimateLottie() } // wait

    until the composition is idle composeTestRule.waitForIdle() testDispatcher.scheduler.advanceUntilIdle() // take a screenshot composeTestRule.onRoot().captureRoboImage() } 85 Challenge 2: Test Code
  45. • Simulate user interactions, by compose-ui-test • Capture the sequence

    of screenshots as a GIF ——————————————————————————— • Works as a feature of Roborazzi, refer ◦ Can’t be compared Challenge 3: Capture as a GIF 87
  46. 88 Challenge 3: Test Code @get:Rule val roborazziRule = RoborazziRule(

    captureRoot = onView(ViewMatchers.isRoot()), options = RoborazziRule.Options( captureType = RoborazziRule.CaptureType.Gif(), ) ) @Test fun animateLottieTest() { composeTestRule.setContent { AppTheme { AnimateTab() } } waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) composeTestRule.onNodeWithText("WORK").performClick() waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) }
  47. • Almost There: ◦ The solutions are mostly sufficient •

    Ideal Goal: ◦ A simpler, more streamlined approach. ◦ Smoother GIF production. ◦ Support for custom duration, frame rates, and other animation parameters. 91 Test for Animations Summary
  48. • Roborazzi ◦ Set different screen sizes ◦ Take screenshot

    after clicking one item • CPST ◦ Does not work because of mentioned issues 94 Use Case: Material3#ListDetailPaneScaffold
  49. 95 Use Case: Test Code @Test fun contactsListScreenTest_tablet() { RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi")

    ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() } @Test fun contactsListScreenTest_phone() { RuntimeEnvironment.setQualifiers("w411dp-h891dp-port") ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() }
  50. Recap 01 | Implementation of Screenshot Testing ◦ Roborazzi ◦

    Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ◦ Test for Adaptive Layout ◦ Test for Animations 97
  51. 98 Feel free to check today’s examples in the Sample

    App bingningO/screenshot-testing-sample-app
  52. Roborazzi – Summary • Good Points ◦ Fast and Light

    ◦ Free to do customize • Pain Points ◦ Hard to configure gradle ◦ Hard to analyse bugs ◦ Showkase has several limitations
  53. CPST – Summary • Good Points ◦ Google official support

    ◦ Super easy to use ◦ Super good compatibility with compose @Preview 102
  54. CPST – Summary • Pain Points ◦ Have to put

    all @Preview you want to test into ./screenshotTest source set ◦ Report always fail if having difference ◦ No customize settings at present ◦ Still experimental with some bugs 103