Screenshot Testing Report -- 最近触ってきた経験談

English follows Japanese. (The slide is written in English)

最近ScreenshotTesting を導入したみたの話になります。主に以下の内容を含めます。
* 簡単のscreenshot testingの紹介
* チーム内Roborazzi + Showkaseで自動テストさせた実装経験、色々Tips、感想のシェア
* Google Experimental の Compose Preview Screenshot Testingを触った感想、現状行われた議論のまとめ

This talk is about the experience of recently importing Screenshot Testing. It mainly includes the following:
* A brief introduction to screenshot testing.
* Sharing implementation experiences, various tips, and impressions from using Roborazzi and Showkase for automated testing compose @Preview within the team.
* Impressions of trying out Google's experimental Compose Preview Screenshot Testing and a summary of the discussions held so far.

Transcript

1. Screenshot Testing Report 最近触ってきた経験談 2024.6
2. Intro @Bing（X） • Born in China, Sichuan 🌶 🐼 🍲 • Joined U-NEXT for 4 years • Work as Android Engineer for 8 years • Recently catch chiikawa at night
3. Intro @Bing（X） • Point ◦ Test plays ▪ Find the center of gravity ◦ Choose right shape and size one ▪ actually bigger one is easier ◦ Choose right claw ◦ Find good timing to ask for repositioning But the machines for chiikawa became harder and harder 😫
4. Today’s Topic Compose @Preview Take screenshots Compare before & after
5. Agenda • Simple Intro of Screenshot Testing • Current Best Practise ◦ Roborazzi + Showkase • Experimental Leading Edge ◦ Compose preview screenshot testing • Further Discussion
6. Screenshot Testing Overview
7. 1-dp-padding diff Layout diff
8. Font size diff Color diff
9. Key Benefits • UI Verification ◦ Ensure the UI elements are displayed correctly across different devices and configurations ◦ Ensure the UI elements changes as you expected in each PR • Regression Testing ◦ Helps detect visual regressions introduced by code changes, especially for large scale refactoring/library migration • Easy of Maintenance ◦ Relatively easy to write and maintain, running quickly compared with other tests
10. Libraries on Android • Paparazzi ◦ By CashApp ◦ Use LayoutLib (private library) to render views • Roborazzi ◦ By takahirom-san ◦ Compared with Paparazzi, it works with Robolectric, which allows it to interact with the Android framework ◦ Use Robolectric Native Graphics (supported API) to render views ◦ Used in nowinandroid and DroidKaigi2023, other google samples… • Compose Preview Screenshot Testing ◦ By Google, Experimental, mentioned in Google IO 2024 ◦ “is designed to be as easy to use as composable previews.” (quote from doc)
11. • Paparazzi ◦ By CashApp ◦ Use LayoutLib (private library) to render views • Roborazzi ◦ By takahirom-san ◦ Compared with Paparazzi, it works with Robolectric, which allows it to interact with the Android framework ◦ Use Robolectric Native Graphics (supported API) to render views ◦ Used in nowinandroid and DroidKaigi2023, other google samples… • Compose Preview Screenshot Testing ◦ By Google, Experimental, mentioned in Google IO 2024 ◦ “is designed to be as easy to use as composable previews.” (quote from doc) Libraries on Android
12. Roborazzi (roborazzi + showkase -> run automatically)
13. • @Preview Views • Take screenshots • CI configuration Implementation
14. Implementation • @Preview Views ◦ Showkase ◦ Helps get a list of all @Preview composable views • Take screenshots ◦ Roborazzi ◦ By captureRoboImage() function, take screenshots of all views passed by ParameterizedRobolectricTestRunner • CI configuration ◦ Use github Actions to call roborazzi commands, add comments directly to PRs ◦ The companion branch approach to store images in a new branch and delete them after specific time. (e.g. 1 week)
15. Tips 💡
16. Tips 1 – skip other unit tests • Please do it at first, or will take >> 1 hour locally • Refer: roborazzi github#FAQ // define the test category name interface ScreenshotTestCategory // set the category in screenshot test @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { … } // configure in gradle, then run ./gradlew recordRoborazziDebug -Pscreenshot to skip UT if (target.hasProperty("screenshot")) { target.logger.lifecycle("Screenshot tests are included, ${it.name}, ${it.classpath}") includeCategories("....ScreenshotTestCategory") }
17. Tips 2 – skip AndroidViewBinding • Any @Preview including AndroidViewBinding fails 🤔 • Hard to analyze the log • java.lang.reflect.InvocationTargetException -> can’t find a solution to fix it
18. Tips 2 – skip AndroidViewBinding @Composable fun SomeAndroidViewBindingWrapContent( modifier: Modifier = Modifier, ) { ... // just skip rendering AndroidViewBinding if under testing if (Build.FINGERPRINT == "robolectric") { return } ... } とりあえずSKIP！
19. Tips 3 – layout issues for large screens For large screens, have some layout issues 🤔 （表示崩れ） • Reason ◦ Can only be captured as phone size Refer1: slide of StudyApp Refer2: solution in nowinandroid
20. // @Preview annotation settings @Preview(device = Devices.TABLET) @Preview(device = Devices.DESKTOP) @Preview(device = Devices.LONG_PHONE_DEVICE) // component generated by Showkase, can't transfer `device`, can transfer `group`/`heightDp`/`widthdp` public val PreviewHogeHogeContentDefaultGroupPreviewHogeHogeContent1: ShowkaseBrowserComponent = ShowkaseBrowserComponent( group = "Default Group", componentName = "PreviewHogeHogeContent", componentKDoc = "", componentKey = """modulename.hoge.hoge.description_PreviewHogeHogeContent_null_DefaultGroup_PreviewHogeHogeContent_1_null" "", isDefaultStyle = false, component = @Composable { PreviewHogeHogeContent() } ) Tips 3 – identify the problem
21. Tips 3 – configure for large screens // @Preview annotation settings, add group parameter manually @Preview(device = Devices.TABLET, group = "TABLET") @Preview(device = Devices.DESKTOP, group = "DESKTOP") @Preview(device = Devices.LONG_PHONE_DEVICE, group = "LONG_PHONE_DEVICE") // inside screenshot test, before calling captureRoboImage() when { showkaseBrowserComponent.group.contains("TABLET") -> { RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") } showkaseBrowserComponent.group.contains("DESKTOP") -> { RuntimeEnvironment.setQualifiers("w1920dp-h1080dp-160dpi") } showkaseBrowserComponent.group.contains("LONG_PHONE_DEVICE") -> { RuntimeEnvironment.setQualifiers("w411dp-h1200dp-420dpi") } }
22. // // @Preview annotation settings, add group parameter manually @Preview(device = Devices.TABLET, group = "TABLET") @Preview(device = Devices.DESKTOP, group = "DESKTOP") @Preview(device = Devices.LONG_PHONE_DEVICE, group = "LONG_PHONE_DEVICE") // inside screenshot test, before calling captureRoboImage() when { showkaseBrowserComponent.group.contains("TABLET") -> { RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") } showkaseBrowserComponent.group.contains("DESKTOP") -> { RuntimeEnvironment.setQualifiers("w1920dp-h1080dp-160dpi") } showkaseBrowserComponent.group.contains("LONG_PHONE_DEVICE") -> { RuntimeEnvironment.setQualifiers("w411dp-h1200dp-420dpi") } } Tips 3 – configure for large screens Same configuration
23. Tips 3 – configure for large screens Pain point: • Have to write each screen type manually 😵
24. Other Tips
25. Item Reason Solution @Preview to test have to be internal/public Required by showkase, see their readme Change all @Preview to internal, and configure to skip private ones Always get diff when using current time in @Preview Time displayed is the timing when running the testing Give up using current time in all @Preview Crash for Dialog @Preview Not support by default captureRoboImage() Refer Use captureScreenRoboImage(), Experimental Can’t load image url (e.g. AsyncImage) Image url download is not stable in @Preview Use fixed image, without downloading Blank screen, not rendering UI items Display UI items depending on some UI state (hard to debug, 焦らずread the code ) Forcely display items, ignoring UI state Temporary companion_x branches(to store screenshot images) not deleted properly CI file didn’t check carefully --.yml 読みづらいのでご注意を Simple edition
26. Some more todo Item Reason Solution @Preview annotation (@PreviewScreenSizes, @PreviewFontScales, etc.) Showkase can’t read the annotation Add a new group • Would increase quite a lot of code @PreviewParameter Showkase can’t read it correctly No solution 😇 Refer Long list composable Can only capture the first page of a long list • Add device.LONG_PHONE_DEVICE • Add a new group and do scrolling before capturing Lottie animation Capture image of lottie composable is unstable Refer: droidkaigi2023 Prevent lottie running in background thread
27. Current Situation • Status ◦ Started for 1 month ◦ 469 images, CI job 10 ~ 20 min in busy days • Improved ◦ Easy to see UI change ◦ Easy to maintain • WIP ◦ Fix left issues
28. Summary of Impressions • Good Point ◦ Current best practise: Roborazzi + Showkase ◦ Better for UI check ▪ Easily to understand UI difference even only 1-dp change ▪ Helps to check if any unexpected UI changed ◦ Better for PR discussion ▪ No need to take screenshot manually any more in PR description ▪ Give confidence to PR owner ▪ Let PR reviewer easier to catch the difference ◦ Further ▪ A good preparation for big refactoring, e.g. Single-activity migration
29. • Pain Points ◦ Hard to configure gradle ◦ Hard to fix issues occurred in the testing ▪ Log hard to analysis ◦ Showkase has several limitations ▪ Can’t load private @Preview ▪ Can’t transfer @Preview parameters, etc. Summary of Impressions
30. Plus: roborazzi plugin for Android Studio Release in 6/7 • Directly display screenshots along with the compose @Preview
31. Before: super long list After: quite easy to check related screenshots p.s. not hot reload
32. Compose Preview Screenshot Testing
33. Try it out! Refer: official doc • Prepare environment ◦ Android gradle > 8.5.0-beta01, kotlin > 1.9.20, Android Studio Koala • Add libraries ◦ compose.screenshot dependency ◦ ui-tooling dependency • Configure gradle ◦ Enable experimental property in gradle.properties ◦ Enable the experimental flag in build.gradle.kts • Add @Preview in ./src/screenshotTest/… source set • ✅ RUN!
34. List of screenshots Beautiful HTML report
35. Summary of Impressions • Good Point ◦ Google official support ◦ Configuration super easy ◦ Good compatibility with @Preview ▪ Support private @Preview ▪ Support @Previewparameter, @PreviewScreenSizes, @PreviewFontScale, etc. ▪ Support other @Preview(...) parameters
36. Summary of Impressions • Pain Point ◦ Have to put all @Preview into ./screenshotTest source set ▪ This point hard to implement in current project ◦ Report always fail if having difference ◦ No customize settings at present ◦ Got moshi error, downgrade it to 1.14.0 fixed, refer
37. Catch up current discussion • Feature request ◦ Support @Preview in main source set, link ◦ Customize configuration for screenshot report ◦ Support for more complex composables ◦ Support for Enhanced integration with other testing frameworks • Bugs ◦ Some screenshot accuracy ◦ Some rendering problems ◦ Support for multi-module • Other ◦ The future of Compose preview screenshot testing v.s. Current solutions (e.g. Paparazzi, Roborazzi…), link1 , link2
38. Further Discussion
39. Ideal screenshot testing in the future • An ideal world ◦ Easy of use ◦ Directly compare with design files ◦ No need for QA about UI changes • About current screenshot testing framework ◦ Hope to improve the solution of storing images in github ◦ Hope to have a better debug solution
40. Recap • What is screenshot testing • Roborazzi + showkase ◦ Implementation ◦ Issues ◦ Good / Pain points • Compose Preview Screenshot testing ◦ Implementation ◦ Current Discussion ◦ Good/ Pain points • Ideal world of screenshot testing
41. Refer • Basic ◦ Blog: screenshot testing with compose ◦ DroidKaigi Speech: build snapshot testing pipeline ◦ Droidcon Speech: easy screenshot testing with compose • Roborazzi ◦ Roborazzi github ◦ Roborazzi usage samples ◦ DroidKaigi 2023 app github ◦ スタディサプリ小学・中学講座に Roborazziを導入して半年が経過しました ◦ N予備校 Android アプリでスクリーンショットテストを導入した話 #Android - Qiita • Compose preview screenshot testing ◦ Official doc ◦ Issue tracker
42. ご清聴ありがとうご ざいました！ generated by ChatGPT