$30 off During Our Annual Pro Sale. View Details »

Screenshot Testing for Android: Which Library is Right for You?

Sergio Sastre Flórez
October 03, 2023
Programming
0
95

Screenshot Testing for Android: Which Library is Right for You?

Slides of my lighting talk "Screenshot Testing for Android: Which Library is Right for You?" at Droidcon Lisbon 2023
-------------------------------------------------------------------------------
In recent years, screenshot testing has become an increasingly popular way to automate UI testing in Android applications, with new libraries emerging regularly. That makes hard to determine which one is the best fit for your project.

In this tech-talk, we'll explore the top screenshot testing libraries, from on-device to JVM-based libraries, including the newest player in the game: Roborazzi, based on Google's Robolectric.

We'll dive deep into each library's unique features and compare their pros and cons to help you make an informed decision about which one is right for your project.

Whether you're new to screenshot testing or have experience with it, this tech-talk will take your UI testing skills to the next level. You'll leave this talk with the knowledge to confidently choose the best screenshot testing library for your projects. Don't miss out on this opportunity to become a screenshot testing Ninja in Android!
---------------------------------------------------------------
Repos of the lighting talk:
https://github.com/sergio-sastre/Android-screenshot-testing-playground
https://github.com/sergio-sastre/AndroidUiTestingUtils

Sergio Sastre Flórez

October 03, 2023
Tweet

More Decks by Sergio Sastre Flórez

See All by Sergio Sastre Flórez
Mastering Screenshot Testing Libraries for Android
gio_sastre
0
330
Writing bulletproof code with property-based testing
gio_sastre
0
390
An introduction Effective Snapshot Testing on Android
gio_sastre
0
480

Other Decks in Programming

See All in Programming
Data-centric視点で過去コンペを振り返る
inoichan
1
440
[Helvetic Ruby 2023] Profiling Ruby tests with Swiss precision
palkan
0
320
​고군분투 LLM 프로덕트 적용기: Blind Prompting 부터 Agent까지
huffon
2
960
10年モノのAndroidアプリのコード品質を改善していく、3つの取り組み
okuzawats
0
620
JSConfjp2023 Storybook駆動開発の再現性と効率化
kinocoboy2
1
1.9k
ちょうぜつ本の紹介 (クリーンアーキテクチャとパッケージ原則)
suzukimar
0
270
Open Source - A Journey of Contribution and Collaboration
ivargrimstad
0
470
Java 8 から 21 へのバージョンアップでどう変わる？：実アプリケーションで学ぶ実装のポイント / 20231111 Key Implementation Points from Java 8 to 21
mackey0225
6
530
if 式と switch 式による SwiftUI のプレビューエラー対策
ojun9
1
100
LLM Science Examにおけるデータ戦略まとめ
takashisomeya
1
260
動くコードを書こう / let's code a process
kishida
21
6k
LLMを使ったアプリケーション開発の基本とLangChain超入門
os1ma
16
6.3k

Featured

See All Featured
Ruby is Unlike a Banana
tanoku
93
10k
Side Projects
sachag
450
39k
Intergalactic Javascript Robots from Outer Space
tanoku
264
26k
Imperfection Machines: The Place of Print at Facebook
scottboms
257
12k
Clear Off the Table
cherdarchuk
81
300k
Practical Orchestrator
shlominoach
179
9.4k
From Idea to $5000 a Month in 5 Months
shpigford
376
45k
Design by the Numbers
sachag
272
18k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
185
15k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
5
740
Become a Pro
speakerdeck
PRO
8
3.6k
Building a Modern Day 
E-commerce SEO Strategy
aleyda
13
5.9k

Transcript

  1. BY SERGIO SASTRE
    SCREENSHOT TESTING
    WHICH LIBRARY


    IS RIGHT FOR YOU?
    FOR ANDROID

    View Slide

  2. WHAT
    Screenshot testing
    is
    @Gio_Sastre

    View Slide

  3. Special type of UI tests
    take a screenshot of it,
    compare it to its reference
    we inflate a UI component,
    where
    and
    @Gio_Sastre

    View Slide

  4. WORKS
    HOW
    Screenshot testing
    @Gio_Sastre

    View Slide

  5. record
    AND
    verify
    @Gio_Sastre

    View Slide

  6. write
    test
    CI
    @Gio_Sastre

    +

    View Slide

  7. record
    CI
    @Gio_Sastre

    View Slide

  8. =
    CI
    @Gio_Sastre

    View Slide

  9. CI
    new

    PR
    +
    @Gio_Sastre

    View Slide

  10. approve
    CI
    @Gio_Sastre

    View Slide

  11. merge
    CI
    @Gio_Sastre

    View Slide

  12. CI
    new
    PR
    verify
    @Gio_Sastre

    View Slide

  13. CI
    new
    PR
    @Gio_Sastre

    View Slide

  14. CI
    new
    PR
    @Gio_Sastre

    View Slide

  15. CI
    new
    PR
    @Gio_Sastre

    View Slide

  16. CI
    new
    PR
    @Gio_Sastre

    View Slide

  17. EXIST?
    Which
    Screenshot testing libraries
    @Gio_Sastre

    View Slide

  18. Screenshot testing libraries
    android-testify
    Paparazzi
    screenshot-tests
    

    for-android
    Shot
    Dropshots Snappy
    Roborazzi
    @Gio_Sastre

    View Slide

  19. on-device
    Screenshot testing libraries
    @Gio_Sastre

    View Slide

  20. jvm
    layoutlib robolectric
    robolectric
    *
    Needs to be con
    f
    igured
    *
    Screenshot testing libraries
    @Gio_Sastre

    View Slide

  21. in this talk
    jvm jvm
    on-device
    Screenshot testing libraries
    @Gio_Sastre
    on-device on-device

    View Slide

  22. On-device
    Screenshot testing
    @Gio_Sastre

    View Slide

  23. Dropshots
    Shot
    tolerance
    record


    &


    verify
    gradle


    managed


    devices
    test


    reports
    general value


    for


    all tests
    only


    from CLI
    @Gio_Sastre
    standard xml
    configuration
    problematic
    Windows support
    Kotlin gradle scripts
    ⚠
    ⚠
    super easy per test
    from


    CLI & AS
    very nice


    HTML


    with diffs
    not


    supported
    not


    supported
    +


    diffs
    android-testify
    easy per test
    CLI or


    AS Plugin
    supported
    YAML
    +


    diffs


    (experimental)
    max 1 snapshot


    per test

    View Slide

  24. Dropshots
    Shot
    tolerance
    record


    &


    verify
    gradle


    managed


    devices
    test


    reports
    general value


    for


    all tests
    only


    from CLI
    @Gio_Sastre
    standard xml
    configuration
    problematic
    Windows support
    Kotlin gradle scripts
    ⚠
    ⚠
    super easy per test
    from


    CLI & AS
    very nice


    HTML


    with diffs
    not


    supported
    not


    supported
    +


    diffs
    android-testify
    easy per test
    CLI or


    AS Plugin
    supported
    YAML
    +


    diffs


    (experimental)
    max 1 snapshot


    per test

    View Slide

  25. Jvm
    Screenshot testing
    @Gio_Sastre

    View Slide

  26. Paparazzi
    Windows


    OS
    not


    supported
    Roborazzi
    Compose


    Multiplatform


    (as desktop)
    Dynamic Feature


    Modules
    not


    supported supported
    @Gio_Sastre
    supported
    2
    1 Due to Robolectric Native Graphics
    2 Due to Robolectric
    supported
    2
    Bazel
    record


    &


    verify
    from


    CLI & AS
    only


    from CLI
    unofficial


    AS plugin
    supported
    1
    Via Skiko
    or
    not


    supported
    3
    3
    not


    supported

    View Slide

  27. Paparazzi
    Windows


    OS
    not


    supported
    Roborazzi
    Compose


    Multiplatform


    (as desktop)
    Dynamic Feature


    Modules
    not


    supported supported
    @Gio_Sastre
    supported
    2
    1 Due to Robolectric Native Graphics
    2 Due to Robolectric
    supported
    2
    Bazel
    record


    &


    verify
    from


    CLI & AS
    only


    from CLI
    unofficial


    AS plugin
    supported
    1
    Via Skiko
    or
    not


    supported
    3
    3
    not


    supported

    View Slide

  28. Paparazzi
    complex


    animations


    (views)
    Side


    effects


    (compose)
    render


    elevation
    not


    supported
    Roborazzi
    different
    

    API levels
    supported
    not


    supported
    not


    supported
    not fully


    supported
    not


    supported
    supported
    @Gio_Sastre
    supported2
    1
    2
    1 Due to layoutlib
    2
    Due to Robolectric
    2
    supported supported2
    2
    Activities
    

    &


    Fragments

    View Slide

  29. Paparazzi
    complex


    animations


    (views)
    Side


    effects


    (compose)
    render


    elevation
    not


    supported
    Roborazzi
    different
    

    API levels
    supported
    not


    supported
    not


    supported
    not fully


    supported
    not


    supported
    supported
    @Gio_Sastre
    supported2
    1
    2
    1 Due to layoutlib
    2
    Due to Robolectric
    2
    supported supported2
    2
    Activities
    

    &


    Fragments

    View Slide

  30. DIFFERENT CONFIGS?
    SCREENSHOT TESTS
    under
    @Gio_Sastre

    View Slide

  31. @Gio_Sastre
    Configuration matters
    1
    3
    Locale
    UI Mode
    2 Orientation
    4
    6
    Custom themes
    Display Size
    5 Font Size

    View Slide

  32. Settings Display Font size
    >
    >
    Font Size
    @Gio_Sastre

    View Slide

  33. @Gio_Sastre
    Font Size

    View Slide

  34. Display Size
    Settings Display Display size
    >
    >
    @Gio_Sastre

    View Slide

  35. Display Size
    @Gio_Sastre

    View Slide

  36. Font Size vs. Display Size
    @Gio_Sastre

    View Slide

  37. Font Size + Display Size
    @Gio_Sastre

    View Slide

  38. MANY CONFIGS?
    HOW
    to screenshot test so
    @Gio_Sastre

    View Slide

  39. Android Ui Testing Utils
    @Gio_Sastre
    1 Screenshot Testing under different configurations
    https://github.com/sergio-sastre/AndroidUiTestingUtils
    2 Cross-library screenshot testing

    View Slide

  40. @Gio_Sastre
    possible
    Display size
    Font size


    +


    Locale
    UI Mode
    Orientation
    out


    of the box
    out


    of the box
    out


    of the box
    1
    via


    updateConfiguration()
    possible
    switch height & width
    1
    Activities under test must be open, otherwise still possible with AndroidUiTestingUtils
    out


    of the box
    Custom


    theme
    out


    of the box
    possible
    via


    updateConfiguration()
    out


    of the box
    out


    of the box
    out


    of the box
    robolectric


    based

    View Slide

  41. JVM ?
    ON-DEVICE
    or
    @Gio_Sastre

    View Slide

  42. complex


    animations


    (views)
    Side


    effects


    (compose)
    render


    elevation
    not


    supported
    different
    

    API levels
    Activity


    &


    Fragment
    not


    supported
    not


    supported
    supported
    not fully


    supported
    not


    supported
    supported
    @Gio_Sastre
    supported2
    1
    2
    1 Due to layoutlib
    2 Due to Robolectric
    supported supported
    2
    robolectric


    based 2 2

    View Slide

  43. supported supported supported supported supported
    complex


    animations


    (views)
    Side


    effects


    (compose)
    render


    elevation
    not


    supported
    different
    

    API levels
    Activity


    &


    Fragment
    not


    supported
    not


    supported
    supported
    not fully


    supported
    not


    supported
    supported
    @Gio_Sastre
    supported2
    1
    2
    1 Due to layoutlib
    2 Due to Robolectric
    supported supported
    2
    robolectric


    based 2 2

    View Slide

  44. @Gio_Sastre
    * Only possible with Paparazzi and Roborazzi
    On-device
    on-device vs. jvm
    Best fidelity
    Foldable support
    Special configuration (adb)
    Jvm
    Very fast
    Less flaky
    Accessibility services *
    (no issues with emulators)

    View Slide

  45. Resources
    1
    @Gio_Sastre
    Github Repo
    https://github.com/sergio-sastre/Android-screenshot-testing-playground
    2 Screenshot testing companion library
    https://github.com/sergio-sastre/AndroidUiTestingUtils

    View Slide

  46. attention
    for your
    THANKS

    View Slide

  47. GitHub
    @SergioSastre
    sergio-sastre-florez
    @sergio-sastre
    @Gio_Sastre
    Hashnode blogs
    Sergio Sastre Flórez
    Lead & Senior
    Android developer appdev.de
    Linkedin

    View Slide