Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An introduction Effective Snapshot Testing on A...

An introduction Effective Snapshot Testing on Android

Talk on Snapshot testing at Droidcon Berlin 2021 & Droidcon London 2021
-------------------------------------------------------------------------------
UI tests are a powerful tool to detect regression bugs. However, they are tedious to write and run very slow. But they are not the only tool we got to test the UI:
snapshot tests are some specific type of UI tests, that are not only easier to write but also run much faster than standard UI tests. They even help detect visual regression bugs that Espresso/UiAutomator tests cannot. Indeed, they are more widely used than UI tests by big companies such as Airbnb, with ≈ 30.000 snapshot tests, and Uber, with thousand of snapshot tests vs. a handful of UI tests, among others.

While there are plenty of resources about writing UI tests on Android, the same doesn't exist for snapshot tests. There is no guidance.

So, what is actually snapshot testing? how do you write a valuable snapshot test? what are the best practices? what are the pitfalls you have to take into account?

If you want to know how to get the most out of snapshot testing, come and join this tech talk. You will learn:
- what is snapshot testing and how your app can benefit from it
- how to decide what to snapshot test
- how to write your first meaningful snapshot test
- tips and tricks for better snapshot testing and what pitfalls to avoid
-------------------------------------------------------------------------------
Repo with code examples:
https://github.com/sergio-sastre/Road-To-Effective-Snapshot-Testing

Sergio Sastre Flórez

October 30, 2021
Tweet

More Decks by Sergio Sastre Flórez

Other Decks in Programming

Transcript

  1. Special type of UI tests take a screenshot of it,

    compare it to its reference we inflate a view, where and
  2. Snapshot tests Ui tests thousand handful ≈ 1 600 ≈

    500 Source: "Building Mobile Apps at scale - 39 Engineering Challenges"
  3. Snapshot tests Ui tests thousand handful ≈ 1 600 ≈

    500 ≈ 2 300 ≈ 20 Source: "Building Mobile Apps at scale - 39 Engineering Challenges"
  4. Snapshot tests Ui tests thousand handful ≈ 1 600 ≈

    500 ≈ 2 300 ≈ 20 ≈ 30 000 ? Source: "Building Mobile Apps at scale - 39 Engineering Challenges"
  5. Ui tests Snapshot tests 5- 10 sec Icons made by

    “Freepik" from "Flaticon" writing speed
  6. Ui tests Snapshot tests 5- 10 sec Icons made by

    “Freepik" from "Flaticon" writing speed
  7. Ui tests Snapshot tests 5- 10 sec 1 sec <

    Icons made by “Freepik" from "Flaticon" writing speed
  8. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  9. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  10. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  11. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  12. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  13. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs which will expand till the end of the line and …
  14. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  15. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  16. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  17. Unexpected changes from library updates Spacing, styling, themes… Layout correctness

    Long text View overlapping Languages (RTL) System font size (HUGE) 1 2 3 visual bugs
  18. Font Size Largest 6,16% Small 8,06% Large 14,22% Default 71,56%

    TT Source: Android app with 80 000 monthly active users
  19. Font Size Small 8,06% Default 71,56% TT 20, 38 %

    Large + Largest Source: Android app with 80 000 monthly active users
  20. Dealing with asynchronicity Choosing test-worthy screen states More views, more

    prone to flakiness 1 2 3 snapshot testing full screen
  21. Dealing with asynchronicity Choosing test-worthy screen states More views, more

    prone to flakiness 1 2 3 snapshot testing full screen
  22. Dealing with asynchronicity Choosing test-worthy screen states More views, more

    prone to flakiness 1 2 3 snapshot testing full screen
  23. Dealing with asynchronicity Choosing test-worthy screen states More views, more

    prone to flakiness 1 2 3 snapshot testing full screen
  24. Dealing with asynchronicity Choosing test-worthy screen states More views, more

    prone to flakiness 1 2 3 snapshot testing full screen
  25. start small & simple Header Training View Text view to

    memorise 1 split into smaller testable views
  26. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages
  27. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size
  28. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size narrow screen
  29. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size narrow screen
  30. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size narrow screen trainingInfo =
 TrainingItem ( trainingByLang = wordsToTrainPerLang(999_999) , activeLangs = Language.values().toSet( ) ), TrainingTestItem( fontSize = FontScale.HUGE, viewWidth = ViewWidth.NARROW fun withWordsToTrainUnhappyPath() = )
 ) Mock Data
  31. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size narrow screen trainingInfo =
 TrainingItem ( trainingByLang = wordsToTrainPerLang(999_999) , activeLangs = Language.values().toSet( ) ), TrainingTestItem( fontSize = FontScale.HUGE, viewWidth = ViewWidth.NARROW fun withWordsToTrainUnhappyPath() = )
 )
  32. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size training 
 item narrow screen trainingInfo =
 TrainingItem ( trainingByLang = wordsToTrainPerLang(999_999) , activeLangs = Language.values().toSet( ) ), TrainingTestItem( fontSize = FontScale.HUGE, viewWidth = ViewWidth.NARROW ) fun withWordsToTrainUnhappyPath() = )
 )
  33. aim to break the layout 2 Identifying unhappy paths thousand

    of words to train words to train for all 7 languages huge system font size narrow screen trainingInfo =
 TrainingItem ( trainingByLang = wordsToTrainPerLang(999_999) , activeLangs = Language.values().toSet( ) ), TrainingTestItem( fontSize = FontScale.HUGE, viewWidth = ViewWidth.NARROW )
 ) external fun withWordsToTrainUnhappyPath() =
  34. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } } Test itself
  35. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } }
  36. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } } Wait for 
 Ui thread 
 idle + launch 
 ActivityScenario
  37. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } } Inflate view 
 & bind data Wait for 
 Ui thread 
 idle +
  38. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } } record screenshot
  39. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot private fun ScreenshotTest.snapViewHolder(testItem: TrainingTestItem, snapshotName: String){ val activity = FontsizeActivityScenario.launchWith(testItem.fontScale ) .waitForActivity(), val view = waitForView {
 val layout = activity.inflate(R.layout.training_row)
 TrainingViewHolder(layout).apply { bind ( trainingItem = testItem.trainingItem,
 languageClickedListener = null
 ) } } compareScreenshot {
 holder = view,
 widthInPx = testItem.viewWidth.widthInPx , name = snapshotName
 } }
  40. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } Parameterized 
 tests
  41. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) }
  42. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } set 
 Runner
  43. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } generate 
 test data
  44. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } test with 
 parameter
  45. aim to break the layout 2 writing the snapshot test

    with pedrovgs/Shot @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) }
  46. aim to break the layout 2 thousand of words to

    train words to train for all 7 languages huge system font size narrow screen recording unhappy path
  47. aim to break the layout 2 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val

    testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath()
 .. . )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } recording happy and unhappy paths
  48. aim to break the layout 2 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(private val

    testItem: TrainingTestItem): ScreenshotTest{ companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(), withWordsToTrainHappyPath()
 withoutWordsToTrainUnhappyPath(), withoutWordsToTrainHappyPath( ) )
 } @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } recording happy and unhappy paths
  49. aim to break the layout 2 recording happy and unhappy

    paths without WordsToTrain HappyPath() with WordsToTrain HappyPath() with WordsToTrain UnhappyPath() without WordsToTrain UnhappyPath()
  50. 2’ 4’ 8’ 16’ 800 400 200 100 happy +

    unhappy group tests by relevance 3 100%
  51. 2’ 4’ 8’ 16’ … 800 400 200 100 happy

    + unhappy 100% group tests by relevance 3
  52. 2’ 4’ 8’ 16’ 800 400 200 100 group tests

    by relevance 3 unhappy happy 90 - 100% 5 - 20% …
  53. 2’ … 1’ 3’ 40 80 160 320 6’ unhappy

    happy 90 - 100% 5 - 20% group tests by relevance 3
  54. night build run happy path tests ONLY catch most important

    bugs run on every PR run unhappy path tests ONLY catch less important bugs run once a day daily builds
  55. night build run happy path tests ONLY catch most important

    bugs run on every PR run unhappy path tests ONLY catch less important bugs run once a day daily builds fast development cycles
  56. 3 unhappy vs. happy path @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val

    testItem: TrainingTestItem
 ): ScreenshotTest{ @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(), withWordsToTrainHappyPath()
 withoutWordsToTrainUnhappyPath(), withoutWordsToTrainHappyPath( ) )
 } Filtered parameterized 
 tests
  57. 3 unhappy vs. happy path @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val

    testItem: TrainingTestItem
 ): ScreenshotTest{ @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(), withWordsToTrainHappyPath()
 withoutWordsToTrainUnhappyPath(), withoutWordsToTrainHappyPath( ) )
 }
  58. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(), withWordsToTrainHappyPath()
 withoutWordsToTrainUnhappyPath(), withoutWordsToTrainHappyPath( ) )
 } unhappy vs. happy path
  59. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(),
 withoutWordsToTrainUnhappyPath( ) )
 } companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainHappyPath()
 withoutWordsToTrainHappyPath( ) )
 } @RunWith(Parameterized::class)
 class TrainingItemHappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{ @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } unhappy vs. happy path
  60. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(),
 withoutWordsToTrainUnhappyPath( ) )
 } companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainHappyPath()
 withoutWordsToTrainHappyPath( ) )
 } @RunWith(Parameterized::class)
 class TrainingItemHappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{ @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } unhappy vs. happy path
  61. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @UnhappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(),
 withoutWordsToTrainUnhappyPath( ) )
 } companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainHappyPath()
 withoutWordsToTrainHappyPath( ) )
 } @RunWith(Parameterized::class)
 class TrainingItemHappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{ @HappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } unhappy vs. happy path
  62. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @UnhappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(),
 withoutWordsToTrainUnhappyPath( ) )
 } companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainHappyPath()
 withoutWordsToTrainHappyPath( ) )
 } @RunWith(Parameterized::class)
 class TrainingItemHappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{ @HappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } = path.to.annotation.UnhappyPath -Pandroid.testIntrumentationRunnerArguments.annotation ./gradlew -Precord unhappy vs. happy path
  63. 3 @RunWith(Parameterized::class)
 class TrainingItemUnhappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{

    @UnhappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } group tests by relevance companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainUnhappyPath(),
 withoutWordsToTrainUnhappyPath( ) )
 } companion object {
 @JvmStatic
 @Parameterized.Parameter s fun testData(): Array<TrainingTestItem> = arrayOf ( withWordsToTrainHappyPath()
 withoutWordsToTrainHappyPath( ) )
 } @RunWith(Parameterized::class)
 class TrainingItemHappyPath(
 private val testItem: TrainingTestItem
 ): ScreenshotTest{ @HappyPat h @Test
 fun snapTrainingItem(){ snapViewHolder(testItem)
 ) } = path.to.annotation.HappyPath -Pandroid.testIntrumentationRunnerArguments.annotation ./gradlew -Precord unhappy vs. happy path
  64. 1 Snapshot testing effectively 2 Start small & simple Aim

    to break the layout 3 Group tests by relevance