Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Android Testing Infrastructure
Search
Mohit S
August 02, 2022
Programming
1
390
Building Android Testing Infrastructure
Mohit S
August 02, 2022
Tweet
Share
More Decks by Mohit S
See All by Mohit S
Guide to Improving Compose Performance
heyitsmohit
0
160
Building Shared UIs across Platforms with Compose
heyitsmohit
1
550
Building Multiplatform Apps with Compose
heyitsmohit
2
420
Building StateFlows with Jetpack Compose
heyitsmohit
6
1.7k
Migrating to Kotlin State & Shared Flows
heyitsmohit
1
690
Using Square Workflow for Android & iOS
heyitsmohit
1
380
Building Android Infrastructure Teams at Scale
heyitsmohit
3
290
Strategies for Migrating to Jetpack Compose
heyitsmohit
2
510
Challenges of Building Kotlin Multiplatform Libraries
heyitsmohit
1
390
Other Decks in Programming
See All in Programming
17年周年のWebアプリケーションにTanStack Queryを導入する / Implementing TanStack Query in a 17th Anniversary Web Application
saitolume
0
250
PHPで学ぶプログラミングの教訓 / Lessons in Programming Learned through PHP
nrslib
2
170
テストケースの名前はどうつけるべきか?
orgachem
PRO
0
130
SymfonyCon Vienna 2025: Twig, still relevant in 2025?
fabpot
3
1.2k
20年もののレガシープロダクトに 0からPHPStanを入れるまで / phpcon2024
hirobe1999
0
450
あれやってみてー駆動から成長を加速させる / areyattemite-driven
nashiusagi
1
200
開発者とQAの越境で自動テストが増える開発プロセスを実現する
92thunder
1
180
create_tableをしただけなのに〜囚われのuuid編〜
daisukeshinoku
0
240
Semantic Kernelのネイティブプラグインで知識拡張をしてみる
tomokusaba
0
180
Keeping it Ruby: Why Your Product Needs a Ruby SDK - RubyWorld 2024
envek
0
180
テストコードのガイドライン 〜作成から運用まで〜
riku929hr
1
190
たのしいparse.y
ydah
3
120
Featured
See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
5
440
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
0
97
Being A Developer After 40
akosma
87
590k
Agile that works and the tools we love
rasmusluckow
328
21k
Gamification - CAS2011
davidbonilla
80
5.1k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
247
1.3M
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
95
17k
How To Stay Up To Date on Web Technology
chriscoyier
789
250k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Building an army of robots
kneath
302
44k
Optimising Largest Contentful Paint
csswizardry
33
3k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Transcript
Mohit Sarveiya Building Android Testing Infrastructure @heyitsmohit
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Testing
Challenges Teams Monorepo
Arch Fragmentation Codebase Legacy code Feature A Feature B
Testing Codebase 50% Code coverage 20% Code coverage
Test Coverage Gradle Builds Database UI API
Testing Infra Goals • Automate • Scale • Testing Gradle,
Database, UI
UI Test Automation
Test Case Test Suite Test Case Test Case Test Case
Test Case Test Case Test Case Test Case
Test Case Test Case Test Case Test Case Firebase Test
Lab
gCloud Tool to run tests on Firebase Integrates with CI
~ gCloud firebase test android models list
MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS Nexus4 LG Nexus 4
VIRTUAL 2560 x 1600 19,21,22 sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26 ~ gCloud firebase test android models list
~ gCloud firebase test android run —type robo tests instrumentation
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
—device model=Nexus6, version=21 local=en, orientation=portrait
Upload Google Cloud Storage
Google Cloud Storage Run Tests Report
None
~ gCloud firebase test android run —type robo
None
Get Report Test Lab gCloud Build APK CI Pipeline
UI Tests Time
Flank Run Android and iOS tests in parallel Uses Kotlin
Coroutines
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk device: - model: NexusLowRes
version: 28
Test Case Test Case Test Case Test Case Parallel
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2 ## shard time - the amount of time tests within a shard should take shard-time: 2
~ flank android test run
Test Case Test Case Test Case Test Case Parallel
Merged Results Shared 2 Parallel Shard 1
Test Case Test Case Test Case Test Case Problems Flaky
tests
Flaky Test Causes • Concurrency problems • Flaky third party
code • Resource idling.
Flank flank: ## Number of Flaky test attempts
num-flaky-test-attempts: 2
CI Setup Get Report Test Lab Flank Build APK
CI Setup Get Report Test Lab Flank Build APK Docker
Fladle Gradle plugin for using Flank Multi Module Testing support
Recipes Performance Regression
Recipes perfTests { devices.set([ ["model" : "Nexus5", "version" : "28"],
]) testTargets.set([ "class com.test.PerformanceTest" ]) }
Recipes regressionTests { devices.set([ [ "model" : "Nexus4", "version" :
"28"] ]) testTargets.set([ "class com.sample.MyRegressionTest" ]) }
Problems • Tests take too long to run • Flaky
tests • When do you run UI tests?
Nightly CI Job Running UI Tests Get Report Regression
Tests Fladle Build APK
Smoke Tests • Test basic functionality • Run on PRs
Smoke Tests @Retention(AnnotationRetention.RUNTIME) annotation class SmokeTest
Example - Firefox App • Open source • Nightly &
Smoke tests • Shards with Flank
https: / / github.com/mozilla-mobile/fenix
Firefox App Test Rails Smoke Tests Fladle Build APK
Test Rails • Documentation for your test suite • View
results over time • Collaborate with QA
None
None
UI Tests Infra • Run tests in parallel • Use
Flank with Firebase test lab • Flaky tests
Unit Testing Infrastructure
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
How do we structure unit tests?
Problem class Presenter( val repo: Repo, ... )
Problem interface Repo { fun getData(): Data
... }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Structure class PresenterTest { val fakeRepo = FakeRepository()
val presenter = Presenter(fakeRepo) }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Dev Tools • Auto Generate module scaffolding • Enforce rules
on how modules depends on each other
Unit Testing Lint Rules • Enforce best practices
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules • Do not mock data classes
Unit Tests Time
Problem bdae142 .. main 50157a .. d89f145 .. f0ddfb ..
None
Run only affected unit tests
Affected Module Dectector Gradle plugin for determine which files changed
Run only affected tests
Affected Module Detector App Module A Module B Module C
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" }
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" } Fork Commit SpecifiedBranchCommit
Affected Module Detector App Module A Module B Module C
assembleAndroidDebugTest connectedAndroidDebugTest testDebug
Affected Module Detector ~ ./gradlew runAffectedUnitTests // Runs jvm tests
Affected Module Detector ~ ./gradlew runAffectedAndroidTests // Runs UI tests
Run only affected unit tests
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
Problem • Run only affected unit tests • Snapshot testing
with unit tests
Problem • UI Regressions • Example - Constraint layout changes
Paparazzi Gradle plugin to generate screenshot with unit tests Supports
Compose UI
Paparazzi @get:Rule val paparazzi = Paparazzi( deviceConfig = PIXEL_5,
theme = “android.Theme.Material.Light.NoActionBar” )
Paparazzi @get:Rule val paparazzi = Paparazzi( ... ) @Test fun
testView() { paparazzi.snapshot { UiView(uiState) } }
Paparazzi Git (LFS) Snapshot
Run Paparazzi tests
Paparazzi ~ ./gradlew app:recordPaparazziDebug // Generate Report
Paparazzi ~ ./gradlew app:verifyPaparazziDebug // Run again previously recorded
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Database Migrations
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table Rename to “email”
Database Migrations @RenameColumn( tableName = "users", fromColumnName =
"user_email", toColumnName = "email" ) class RenameFromUserAddressToAddress : AutoMigrationSpec
Database Migrations @Database( version = 2, autoMigrations = [ AutoMigration(
from = 1, to = 2, spec = UserDatabase.RenameFromUserAddressToAddress :: class ), ], )
How do we test migrations?
Test Migrations @get:Rule val helper: MigrationTestHelper = MigrationTestHelper( UserDatabase ::
class.java, listOf( UserDatabase.RenameFromUserAddressToAddress() ), )
Test Migrations @Test fun migrate1To2() { db = helper.createDatabase(TEST_DB, 1).apply
{ execSQL( """ INSERT INTO users VALUES (1, ‘User 1', ‘
[email protected]
') """.trimIndent() ) close() } }
Test Migrations @Test fun migrate1To2() { db = helper.runMigrationsAndValidate(TEST_DB,
2, true) }
Test Migrations @Test fun migrate1To2() { val resultCursor =
db.query("SELECT * FROM users”) // Perform assertions }
Problem • Test migrations with unit tests
Database Migrations JDBC SQLite Driver
DB Tools Room for Android Test Room with databases Migration
testing
https: // github.com/jeffdcamp/dbtools-room
Test Migrations class MigrationTest: BaseMigrationTest<UserDatabase>()
Test Migrations val db = Room.databaseBuilder( ... ) .openHelperFactory(JdbcSQLiteOpenHelperFactory(…)) .build()
Test Migrations fun testMigration(fromVersion: Int, toVersion: Int) { migrationTestExtension.createDatabase(name, fromVersion)
migrationTestExtension.runMigrationsAndValidate( name, toVersion, *migrations ) }
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Test Coverage Gradle Builds Database UI API
Gradle Testing
Problem • Detect build regressions
Gradle Profiler Change
Gradle Profiler 1. Write a performance scenario 2. Specify number
of iterations
Gradle Profiler Iteration 1 Iteration 2 Iteration 3
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Abi Change
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Result
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” }
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” apply-android-resource-change-to = “strings.xml”
tasks = ["assemble"] }
Gradle Profiler gradle-profiler —benchmark —iterations=10 —warmups=6
Gradle Profiler Mean: 348 ms Min: 319 ms P25: 330
ms Median: 341 ms P75: 368 ms Std dev: 22.71 ms
Gradle Profiler
How do we automate profiling on CI?
CI Pipeline Docker image with Gradle Profiler
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios Run nightly
Use Cases • Introducing Anvil
Benchmarking Build • Benchmark with Anvil change • Benchmark without
Anvil change (Baseline) • Compare results
Results Module A Module B Module C Baseline Anvil
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Regressions
Thank You! www.codingwithmohit.com @heyitsmohit