Building Android Testing Infrastructure

Transcript

1. Mohit Sarveiya Building Android Testing Infrastructure @heyitsmohit
2. Building Android Testing Infrastructure • Automation Tools • Building Infra pipelines • Snapshot Testing • Gradle Testing
3. Challenges Teams Monorepo
4. Arch Fragmentation Codebase Legacy code Feature A Feature B
5. Testing Codebase 50% Code coverage 20% Code coverage
6. Test Coverage Gradle Builds Database UI API
7. Testing Infra Goals • Automate • Scale • Testing Gradle, Database, UI
8. UI Test Automation
9. Test Case Test Suite Test Case Test Case Test Case
10. Test Case Test Case Test Case Test Case
11. Test Case Test Case Test Case Test Case Firebase Test Lab
12. gCloud Tool to run tests on Firebase Integrates with CI
13. ~ gCloud firebase test android models list
14. MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS Nexus4 LG Nexus 4 VIRTUAL 2560 x 1600 19,21,22 sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26 ~ gCloud firebase test android models list
15. ~ gCloud firebase test android run —type robo tests instrumentation
16. ~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
17. ~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk —device model=Nexus6, version=21 local=en, orientation=portrait
18. Upload Google Cloud Storage
19. Google Cloud Storage Run Tests Report
21. ~ gCloud firebase test android run —type robo
23. Get Report Test Lab gCloud Build APK CI Pipeline
24. UI Tests Time
25. Flank Run Android and iOS tests in parallel Uses Kotlin Coroutines
26. Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk
27. Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk device: - model: NexusLowRes version: 28
28. Test Case Test Case Test Case Test Case Parallel
29. Flank flank: 
 
 ## test shards - the amount of groups to split the test suite into max-test-shards: 2
30. Flank flank: 
 
 ## test shards - the amount of groups to split the test suite into max-test-shards: 2 ## shard time - the amount of time tests within a shard should take shard-time: 2
31. ~ flank android test run
32. Test Case Test Case Test Case Test Case Parallel
33. Merged Results Shared 2 Parallel Shard 1
34. Test Case Test Case Test Case Test Case Problems Flaky tests
35. Flaky Test Causes • Concurrency problems • Flaky third party code • Resource idling.
36. Flank flank: 
 
 ## Number of Flaky test attempts num-flaky-test-attempts: 2
37. CI Setup Get Report Test Lab Flank Build APK
38. CI Setup Get Report Test Lab Flank Build APK Docker
39. Fladle Gradle plugin for using Flank Multi Module Testing support
40. Recipes Performance Regression
41. Recipes perfTests { devices.set([ ["model" : "Nexus5", "version" : "28"], ]) testTargets.set([ "class com.test.PerformanceTest" ]) }
42. Recipes regressionTests { devices.set([ [ "model" : "Nexus4", "version" : "28"] ]) testTargets.set([ "class com.sample.MyRegressionTest" ]) }
43. Problems • Tests take too long to run • Flaky tests • When do you run UI tests?
44. Nightly CI Job Running UI Tests Get Report Regression 
 Tests Fladle Build APK
45. Smoke Tests • Test basic functionality • Run on PRs
46. Smoke Tests @Retention(AnnotationRetention.RUNTIME) annotation class SmokeTest
47. Example - Firefox App • Open source • Nightly & Smoke tests • Shards with Flank
48. https: / / github.com/mozilla-mobile/fenix
49. Firefox App Test Rails Smoke 
 Tests Fladle Build APK
50. Test Rails • Documentation for your test suite • View results over time • Collaborate with QA
53. UI Tests Infra • Run tests in parallel • Use Flank with Firebase test lab • Flaky tests
54. Unit Testing Infrastructure
55. Unit Testing Integration E2E Slower, 
 more expensive Faster, 
 cheaper
56. How do we structure unit tests?
57. Problem class Presenter( val repo: Repo, 
 ... )
58. Problem interface Repo { 
 
 fun getData(): Data 
 
 ... }
59. Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
60. Structure class PresenterTest { 
 val fakeRepo = FakeRepository() 
 
 val presenter = Presenter(fakeRepo) 
 }
61. Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
62. Dev Tools • Auto Generate module scaffolding • Enforce rules on how modules depends on each other
63. Unit Testing Lint Rules • Enforce best practices
64. Unit Testing Lint Rules • Enforce best practices • Coroutine lint test rules
65. Unit Testing Lint Rules • Enforce best practices • Coroutine lint test rules • Do not mock data classes
66. Unit Tests Time
67. Problem bdae142 .. main 50157a .. d89f145 .. f0ddfb ..
69. Run only affected unit tests
70. Affected Module Dectector Gradle plugin for determine which files changed Run only affected tests
71. Affected Module Detector App Module A Module B Module C
72. Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom = "PreviousCommit" }
73. Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom = "PreviousCommit" } Fork Commit SpecifiedBranchCommit
74. Affected Module Detector App Module A Module B Module C assembleAndroidDebugTest connectedAndroidDebugTest testDebug
75. Affected Module Detector ~ ./gradlew runAffectedUnitTests // Runs jvm tests
76. Affected Module Detector ~ ./gradlew runAffectedAndroidTests // Runs UI tests
77. Run only affected unit tests
78. Unit Testing Integration E2E Slower, 
 more expensive Faster, 
 cheaper
79. Problem • Run only affected unit tests • Snapshot testing with unit tests
80. Problem • UI Regressions • Example - Constraint layout changes
81. Paparazzi Gradle plugin to generate screenshot with unit tests Supports Compose UI
82. Paparazzi @get:Rule val paparazzi = Paparazzi( 
 deviceConfig = PIXEL_5, 
 theme = “android.Theme.Material.Light.NoActionBar” )
83. Paparazzi @get:Rule val paparazzi = Paparazzi( ... ) @Test fun testView() { paparazzi.snapshot { UiView(uiState) } }
84. Paparazzi Git (LFS) Snapshot
85. Run Paparazzi tests
86. Paparazzi ~ ./gradlew app:recordPaparazziDebug // Generate Report
87. Paparazzi ~ ./gradlew app:verifyPaparazziDebug // Run again previously recorded
88. Problem • Run only affected unit tests • Creating snapshot with unit testing • Unit testing database migrations
89. Database Migrations
90. Database Migrations id user_name user_email 1 User 1 user1@gmail.com 2 User 2 user2@gmail.com User Table
91. Database Migrations id user_name user_email 1 User 1 user1@gmail.com 2 User 2 user2@gmail.com User Table Rename to “email”
92. Database Migrations @RenameColumn( 
 tableName = "users", 
 fromColumnName = "user_email", 
 toColumnName = "email" 
 ) class RenameFromUserAddressToAddress : AutoMigrationSpec
93. Database Migrations @Database( version = 2, autoMigrations = [ AutoMigration( from = 1, to = 2, spec = UserDatabase.RenameFromUserAddressToAddress :: class ), ], )
94. How do we test migrations?
95. Test Migrations @get:Rule val helper: MigrationTestHelper = MigrationTestHelper( UserDatabase :: class.java, listOf( UserDatabase.RenameFromUserAddressToAddress() ), )
96. Test Migrations @Test fun migrate1To2() { db = helper.createDatabase(TEST_DB, 1).apply { execSQL( """ INSERT INTO users VALUES (1, ‘User 1', ‘user1@google.com') """.trimIndent() ) close() } }
97. Test Migrations @Test fun migrate1To2() { 
 db = helper.runMigrationsAndValidate(TEST_DB, 2, true) }
98. Test Migrations @Test fun migrate1To2() { 
 val resultCursor = db.query("SELECT * FROM users”) // Perform assertions }
99. Problem • Test migrations with unit tests
100. Database Migrations JDBC SQLite Driver
101. DB Tools Room for Android Test Room with databases Migration testing
102. https: // github.com/jeffdcamp/dbtools-room
103. Test Migrations class MigrationTest: BaseMigrationTest<UserDatabase>()
104. Test Migrations val db = Room.databaseBuilder( ... ) .openHelperFactory(JdbcSQLiteOpenHelperFactory(…)) .build()
105. Test Migrations fun testMigration(fromVersion: Int, toVersion: Int) { migrationTestExtension.createDatabase(name, fromVersion) migrationTestExtension.runMigrationsAndValidate( name, toVersion, 
 *migrations 
 ) }
106. Problem • Run only affected unit tests • Creating snapshot with unit testing • Unit testing database migrations
107. Test Coverage Gradle Builds Database UI API
108. Gradle Testing
109. Problem • Detect build regressions
110. Gradle Profiler Change
111. Gradle Profiler 1. Write a performance scenario 2. Specify number of iterations
112. Gradle Profiler Iteration 1 Iteration 2 Iteration 3
113. Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Abi Change
114. Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Result
115. Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” }
116. Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” apply-android-resource-change-to = “strings.xml” tasks = ["assemble"] }
117. Gradle Profiler gradle-profiler —benchmark —iterations=10 —warmups=6
118. Gradle Profiler Mean: 348 ms Min: 319 ms P25: 330 ms Median: 341 ms P75: 368 ms Std dev: 22.71 ms
119. Gradle Profiler
120. How do we automate profiling on CI?
121. CI Pipeline Docker image with 
 Gradle Profiler
122. CI Pipeline Docker image with 
 Gradle Profiler Performance 
 Scenarios
123. CI Pipeline Docker image with 
 Gradle Profiler Performance 
 Scenarios Run 
 nightly
124. Use Cases • Introducing Anvil
125. Benchmarking Build • Benchmark with Anvil change • Benchmark without Anvil change (Baseline) • Compare results
126. Results Module A Module B Module C Baseline Anvil
127. Building Android Testing Infrastructure • Automation Tools • Building Infra pipelines • Snapshot Testing • Gradle Regressions
128. Thank You! www.codingwithmohit.com @heyitsmohit