Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Android Testing Infrastructure
Search
Mohit S
August 02, 2022
Programming
1
510
Building Android Testing Infrastructure
Mohit S
August 02, 2022
Tweet
Share
More Decks by Mohit S
See All by Mohit S
Guide to Improving Compose Performance
heyitsmohit
0
250
Building Shared UIs across Platforms with Compose
heyitsmohit
1
650
Building Multiplatform Apps with Compose
heyitsmohit
2
520
Building StateFlows with Jetpack Compose
heyitsmohit
6
1.9k
Migrating to Kotlin State & Shared Flows
heyitsmohit
1
800
Using Square Workflow for Android & iOS
heyitsmohit
1
440
Building Android Infrastructure Teams at Scale
heyitsmohit
3
340
Strategies for Migrating to Jetpack Compose
heyitsmohit
2
580
Challenges of Building Kotlin Multiplatform Libraries
heyitsmohit
1
450
Other Decks in Programming
See All in Programming
フロントエンド開発のためのブラウザ組み込みAI入門
masashi
6
3.2k
bootcamp2025_バックエンド研修_WebAPIサーバ作成.pdf
geniee_inc
0
120
CSC305 Lecture 05
javiergs
PRO
0
230
ソフトウェア設計の実践的な考え方
masuda220
PRO
4
620
Range on Rails ―「多重範囲型」という新たな選択肢が、複雑ロジックを劇的にシンプルにしたワケ
rizap_tech
0
6.7k
Webサーバーサイド言語としてのRustについて
kouyuume
0
680
タスクの特性や不確実性に応じた最適な作業スタイルの選択(ペアプロ・モブプロ・ソロプロ)と実践 / Optimal Work Style Selection: Pair, Mob, or Solo Programming.
honyanya
3
180
登壇は dynamic! な営みである / speech is dynamic
da1chi
0
350
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
500
Server Side Kotlin Meetup vol.16: 内部動作を理解して ハイパフォーマンスなサーバサイド Kotlin アプリケーションを書こう
ternbusty
3
230
スマホから Youtube Shortsを見られないようにする
lemolatoon
27
33k
iOSでSVG画像を扱う
kishikawakatsumi
0
140
Featured
See All Featured
Become a Pro
speakerdeck
PRO
29
5.6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.7k
Producing Creativity
orderedlist
PRO
347
40k
Making the Leap to Tech Lead
cromwellryan
135
9.6k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
21
1.2k
Optimizing for Happiness
mojombo
379
70k
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
Side Projects
sachag
455
43k
How GitHub (no longer) Works
holman
315
140k
Building a Modern Day E-commerce SEO Strategy
aleyda
44
7.8k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
Transcript
Mohit Sarveiya Building Android Testing Infrastructure @heyitsmohit
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Testing
Challenges Teams Monorepo
Arch Fragmentation Codebase Legacy code Feature A Feature B
Testing Codebase 50% Code coverage 20% Code coverage
Test Coverage Gradle Builds Database UI API
Testing Infra Goals • Automate • Scale • Testing Gradle,
Database, UI
UI Test Automation
Test Case Test Suite Test Case Test Case Test Case
Test Case Test Case Test Case Test Case
Test Case Test Case Test Case Test Case Firebase Test
Lab
gCloud Tool to run tests on Firebase Integrates with CI
~ gCloud firebase test android models list
MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS Nexus4 LG Nexus 4
VIRTUAL 2560 x 1600 19,21,22 sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26 ~ gCloud firebase test android models list
~ gCloud firebase test android run —type robo tests instrumentation
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
—device model=Nexus6, version=21 local=en, orientation=portrait
Upload Google Cloud Storage
Google Cloud Storage Run Tests Report
None
~ gCloud firebase test android run —type robo
None
Get Report Test Lab gCloud Build APK CI Pipeline
UI Tests Time
Flank Run Android and iOS tests in parallel Uses Kotlin
Coroutines
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk device: - model: NexusLowRes
version: 28
Test Case Test Case Test Case Test Case Parallel
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2 ## shard time - the amount of time tests within a shard should take shard-time: 2
~ flank android test run
Test Case Test Case Test Case Test Case Parallel
Merged Results Shared 2 Parallel Shard 1
Test Case Test Case Test Case Test Case Problems Flaky
tests
Flaky Test Causes • Concurrency problems • Flaky third party
code • Resource idling.
Flank flank: ## Number of Flaky test attempts
num-flaky-test-attempts: 2
CI Setup Get Report Test Lab Flank Build APK
CI Setup Get Report Test Lab Flank Build APK Docker
Fladle Gradle plugin for using Flank Multi Module Testing support
Recipes Performance Regression
Recipes perfTests { devices.set([ ["model" : "Nexus5", "version" : "28"],
]) testTargets.set([ "class com.test.PerformanceTest" ]) }
Recipes regressionTests { devices.set([ [ "model" : "Nexus4", "version" :
"28"] ]) testTargets.set([ "class com.sample.MyRegressionTest" ]) }
Problems • Tests take too long to run • Flaky
tests • When do you run UI tests?
Nightly CI Job Running UI Tests Get Report Regression
Tests Fladle Build APK
Smoke Tests • Test basic functionality • Run on PRs
Smoke Tests @Retention(AnnotationRetention.RUNTIME) annotation class SmokeTest
Example - Firefox App • Open source • Nightly &
Smoke tests • Shards with Flank
https: / / github.com/mozilla-mobile/fenix
Firefox App Test Rails Smoke Tests Fladle Build APK
Test Rails • Documentation for your test suite • View
results over time • Collaborate with QA
None
None
UI Tests Infra • Run tests in parallel • Use
Flank with Firebase test lab • Flaky tests
Unit Testing Infrastructure
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
How do we structure unit tests?
Problem class Presenter( val repo: Repo, ... )
Problem interface Repo { fun getData(): Data
... }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Structure class PresenterTest { val fakeRepo = FakeRepository()
val presenter = Presenter(fakeRepo) }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Dev Tools • Auto Generate module scaffolding • Enforce rules
on how modules depends on each other
Unit Testing Lint Rules • Enforce best practices
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules • Do not mock data classes
Unit Tests Time
Problem bdae142 .. main 50157a .. d89f145 .. f0ddfb ..
None
Run only affected unit tests
Affected Module Dectector Gradle plugin for determine which files changed
Run only affected tests
Affected Module Detector App Module A Module B Module C
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" }
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" } Fork Commit SpecifiedBranchCommit
Affected Module Detector App Module A Module B Module C
assembleAndroidDebugTest connectedAndroidDebugTest testDebug
Affected Module Detector ~ ./gradlew runAffectedUnitTests // Runs jvm tests
Affected Module Detector ~ ./gradlew runAffectedAndroidTests // Runs UI tests
Run only affected unit tests
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
Problem • Run only affected unit tests • Snapshot testing
with unit tests
Problem • UI Regressions • Example - Constraint layout changes
Paparazzi Gradle plugin to generate screenshot with unit tests Supports
Compose UI
Paparazzi @get:Rule val paparazzi = Paparazzi( deviceConfig = PIXEL_5,
theme = “android.Theme.Material.Light.NoActionBar” )
Paparazzi @get:Rule val paparazzi = Paparazzi( ... ) @Test fun
testView() { paparazzi.snapshot { UiView(uiState) } }
Paparazzi Git (LFS) Snapshot
Run Paparazzi tests
Paparazzi ~ ./gradlew app:recordPaparazziDebug // Generate Report
Paparazzi ~ ./gradlew app:verifyPaparazziDebug // Run again previously recorded
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Database Migrations
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table Rename to “email”
Database Migrations @RenameColumn( tableName = "users", fromColumnName =
"user_email", toColumnName = "email" ) class RenameFromUserAddressToAddress : AutoMigrationSpec
Database Migrations @Database( version = 2, autoMigrations = [ AutoMigration(
from = 1, to = 2, spec = UserDatabase.RenameFromUserAddressToAddress :: class ), ], )
How do we test migrations?
Test Migrations @get:Rule val helper: MigrationTestHelper = MigrationTestHelper( UserDatabase ::
class.java, listOf( UserDatabase.RenameFromUserAddressToAddress() ), )
Test Migrations @Test fun migrate1To2() { db = helper.createDatabase(TEST_DB, 1).apply
{ execSQL( """ INSERT INTO users VALUES (1, ‘User 1', ‘
[email protected]
') """.trimIndent() ) close() } }
Test Migrations @Test fun migrate1To2() { db = helper.runMigrationsAndValidate(TEST_DB,
2, true) }
Test Migrations @Test fun migrate1To2() { val resultCursor =
db.query("SELECT * FROM users”) // Perform assertions }
Problem • Test migrations with unit tests
Database Migrations JDBC SQLite Driver
DB Tools Room for Android Test Room with databases Migration
testing
https: // github.com/jeffdcamp/dbtools-room
Test Migrations class MigrationTest: BaseMigrationTest<UserDatabase>()
Test Migrations val db = Room.databaseBuilder( ... ) .openHelperFactory(JdbcSQLiteOpenHelperFactory(…)) .build()
Test Migrations fun testMigration(fromVersion: Int, toVersion: Int) { migrationTestExtension.createDatabase(name, fromVersion)
migrationTestExtension.runMigrationsAndValidate( name, toVersion, *migrations ) }
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Test Coverage Gradle Builds Database UI API
Gradle Testing
Problem • Detect build regressions
Gradle Profiler Change
Gradle Profiler 1. Write a performance scenario 2. Specify number
of iterations
Gradle Profiler Iteration 1 Iteration 2 Iteration 3
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Abi Change
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Result
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” }
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” apply-android-resource-change-to = “strings.xml”
tasks = ["assemble"] }
Gradle Profiler gradle-profiler —benchmark —iterations=10 —warmups=6
Gradle Profiler Mean: 348 ms Min: 319 ms P25: 330
ms Median: 341 ms P75: 368 ms Std dev: 22.71 ms
Gradle Profiler
How do we automate profiling on CI?
CI Pipeline Docker image with Gradle Profiler
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios Run nightly
Use Cases • Introducing Anvil
Benchmarking Build • Benchmark with Anvil change • Benchmark without
Anvil change (Baseline) • Compare results
Results Module A Module B Module C Baseline Anvil
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Regressions
Thank You! www.codingwithmohit.com @heyitsmohit