Beyond Modularization: Scaling your Android Build with Gradle

Tips and tricks on how to scale your Android build with Gradle.
How to measure performance improvements and gain insights using Gradle Enterprise.

https://gradle.com/enterprise/trial/

Video of this talk is available here: https://vimeo.com/430648415

Transcript

1. Beyond Modularization: Scaling Android Builds Nelson Osacky
2. Me • Previously Android Engineer • Large projects • SoundCloud • Square • Small startups • Solutions Engineer at Gradle
3. Me • Gradle Plugin Maintainer • Fladle - Easily Scale Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Delect - Replace Dagger with Dagger Reflect https://github.com/soundcloud/delect/ • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor
4. Solutions Engineer https://gradle.com/enterprise/trial/
6. Gradle Build Tool Gradle Enteprise
7. Android Gradle Plugin Android Studio
8. Kotlin Intellij
9. Android Development Feature Development Tech Debt (There are other things too)
10. Tech Debt Refactoring Build Speed
11. Build Speed is Tech Debt And it always pays off
12. Cost of Builds 60s waste * 50 builds / day * 50 devs = 42 hours lost / day
13. Cost of Builds 60s waste * 50 builds / day * 50 devs = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator
14. Fast Builds Matter
15. Cost of Builds 60s waste * 50 builds / day * 50 devs = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator
16. Build Speed is Tech Debt And it always pays off And is easy to justify working on it
17. Motivated? Where do we start?
19. Update All The Things ⬢ Gradle 6.5 ⬢ Android Gradle Plugin 4.0.0 ⬢ Gradle Enterprise Plugin 3.3.4 ⬢ Kotlin 1.3.72 ⬢ Third party plugins ⬢ Third party libraries New Performance APIs Caching Improvements Task Configuration Avoidance Background Scan Uploads Incremental Annotation Processors Compiler Perf Improvements
20. Update All The Things plugins { id "com.github.ben-manes.versions" version "0.28.0" } ./gradlew dependencyUpdates -Drevision=release https://github.com/ben-manes/gradle-versions-plugin
22. Build Lifecycle •Initialization •Configuration •Execution
23. Build Lifecycle Initialization Sets up the environment for the build and determines which projects will take part in it.
24. Build Lifecycle Configuration All build scripts of all projects are executed. Constructs and configures the task graph.
25. Build Lifecycle Execution Runs the tasks graph in order.
26. Where to start?
27. Check for Red Flags
28. What is a red flag?
29. If nothing changes, no tasks should execute.
30. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 2s 58 actionable tasks: 58 up-to-date
31. ./gradlew assembleDebug ./gradlew assembleDebug BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356 from cache
32. Check for Red Flags Regularly
33. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356 up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available
34. BUILD SUCCESSFUL in 5s 359 actionable tasks: 3 executed, 356 up-to-date Publishing build scan... https://gradle.com/s/yj5jtd4kh6ucc Build Analyzer results available Gradle Build Scans Android Studio Build Analyzer
35. Android Studio Build Analyzer
36. Run a build scan ./gradlew assembleDebug --scan https://scans.gradle.com/
38. Filter
41. 9 tasks ran Bugsnag
44. #perfmatters
47. 1 sec per project is high
49. 25 sec is red flag!
50. apply plugin: 'witness'
51. //apply plugin: 'witness'
53. 5.8s to 0.5s by disabling two plugins!
54. Configuration time savings affect every build
55. Configuration time should be roughly 5s per 100 projects
56. Disable plugins for local builds
57. engBuild pattern
58. engBuild != buildVariant
59. engBuild == Gradle property https://docs.gradle.org/current/userguide/build_environment.html#sec:gradle_configuration_properties
60. enabled by default ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()
61. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }
62. ./gradlew assembleDebug ./gradlew assembleRelease -PengBuild=false disable on CI or testing release builds
63. engBuild=false disable on CI or testing release builds gradle.properties
64. engBuild pattern
65. 16627 tasks is a red flag
66. Use Task Configuration Avoidance https://docs.gradle.org/current/userguide/task_configuration_avoidance.html
67. api vs implementation
68. Module A Module B Module C dependencies { implementation project('module-b') implementation project('module-c') } dependencies { implementation project('module-c') }
69. Module A Module B Module C dependencies { implementation project('module-b') } dependencies { api project('module-c') } module-c must always used when consuming module-b
70. api vs implementation convenience vs performance
71. Real World Example
72. https://github.com/k9mail/k-9 K9 Mail App
73. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')
74. K9 Mail App https://github.com/k9mail/k-9/blob/master/app/core/build.gradle#L12 apiaproject(':plugins:openpgp-api-lib:openpgp-api')
75. apiaproject(':plugins:openpgp-api-lib:openpgp-api') K9 Mail App
76. if (project.hasProperty('useImpl')) { implementation project(':plugins:openpgp-api-lib:openpgp-api') } else { apiaproject(':plugins:openpgp-api-lib:openpgp-api') } K9 Mail App
77. ./gradlew :app:k9mail:assembleDebug ./gradlew :app:k9mail:assembleDebug -PuseImpl K9 Mail App
78. Profile it!
79. https://github.com/gradle/gradle-profiler Gradle Profiler
80. Let's write scenarios Gradle Profiler
81. change_api_dep { } performance.scenarios Gradle Profiler
82. change_api_dep { title = "Change using api dependency" } performance.scenarios Gradle Profiler
83. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] } performance.scenarios Gradle Profiler
84. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
85. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
86. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
87. change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
88. change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
89. change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/ openintents/openpgp/OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
90. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
91. change_api_dep { title = "Change using api dependency" tasks = ["app:k9mail:assembleDebug"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } change_impl_dep { title = "Change using impl dependency" tasks = ["app:k9mail:assembleDebug"] gradle-args = ["-PuseImpl"] apply-non-abi-change-to = "openpgp-api/src/main/java/org/openintents/openpgp/ OpenPgpApiManager.java" } performance.scenarios Gradle Profiler
92. gradle-profiler --benchmark --scenario-file performance.scenarios Gradle Profiler
93. 1.stop daemon 2.clean 3.6 warm up builds 4.10 benchmark builds Gradle Profiler
95. html report and csv report
96. Build Time (milliseconds) 0 3250 6500 9750 13000 #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 Change using api dependency Change using impl dependency
97. scenario Change using api dependency Change using impl dependency mean 11603.0 10151.7 median 11529.5 10049.5 confidence 0.0 99.98429477157690
98. ~15% improvement
99. why?
100. gradle-profiler --profile buildscan --scenario-file performance.scenarios
101. gradle-profiler --profile buildscan --scenario-file performance.scenarios * Results written to /Users/no/workspace/k-9/profile-out-14 Scenario Change using api dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/ozev5czodllsk Scenario Change using impl dependency using Gradle 6.5 - Build scan for measured build #1: https://gradle.com/s/rxujhuyxqlybu
102. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency
103. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency
104. Which tasks ran?
105. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency
106. https://scans.gradle.com/s/rxujhuyxqlybu/timeline?outcomeFilter=SUCCESS implementation Dependency
107. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency
108. https://scans.gradle.com/s/ozev5czodllsk/timeline?outcomeFilter=SUCCESS api Dependency
109. What exactly is the difference?
110. Enterprise Feature Alert
111. Capture Task Input Files https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files
112. gradleEnterprise { buildScan { captureTaskInputFiles = true } } https://docs.gradle.com/enterprise/gradle-plugin/#capturing_task_input_files
113. Build Scan Comparison
114. Build Scan Comparison
115. Build Scan Comparison Build B uses api
116. Adding unnecessary inputs to tasks.
117. How do we fix it?
118. https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin Dependency Analysis Plugin plugins { id "com.autonomousapps.dependency-analysis" version "0.49.0" }
120. Annotation Processors
121. Make sure they are incremental!
122. w: [kapt] Incremental annotation processing requested, but support is disabled because the following processors are not incremental: androidx.room.RoomProcessor (NON_INCREMENTAL).
123. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions { arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }
124. Room
125. android { ... defaultConfig { ... javaCompileOptions { annotationProcessorOptions { arguments += [ "room.schemaLocation":"$projectDir/schemas".toString(), "room.incremental":"true"] } } } }
126. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss!
127. "room.schemaLocation":"$projectDir/schemas".toString(), Remote Cache miss! https://developers.soundcloud.com/blog/gradle-remote-build-cache-misses
128. Move Room to it's own Room https://github.com/gradle/android-cache-fix-gradle-plugin Module
130. Disabling unused AGP Features
131. Cost of Generating BuildConfig All 186 are UP-TO-DATE.
132. Unused so Proguard will remove it later
133. BuildConfig adds java sources to pure kotlin modules
134. Which BuildConfig?
135. Which version code?
136. Which version code?
137. VERSION_CODE is 1 by default in library modules
138. Aidl what?
139. Also adds task configuration time.
141. android.defaults.buildfeatures.buildconfig=false android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false gradle.properties New in AGP 4.0.0 https://developer.android.com/studio/releases/gradle-plugin#buildFeatures
142. Disable for all projects
143. Enable when needed
144. android.buildFeatures { buildConfig = true } app/build.gradle
145. Code Multipliers
146. AndroidX Jetifier
147. Migrates support-library- dependent libraries to rely on Android X packages.
148. android.enableJetifier=true
150. android.enableJetifier=false
151. Dependency Substitution https://issuetracker.google.com/issues/156449751
152. Stuck on the Jetifier?
153. plugins { id "com.github.plnice.canidropjetifier" version "0.5" } Can I drop Jetifier? ./gradlew -Pandroid.enableJetifier=false canIDropJetifier
154. Jetify At Home ./jetifier-standalone -i <source-library> -o <output-library> https://developer.android.com/studio/command-line/jetifier
155. Jetify At Home Upload to nexus
156. Flavors
157. Flavor Explosion
159. Flavor Explosion When you add a build dimension multiplying build speed in a surprising surprise.
160. Flavor Explosion BuildType * BuildFlavor = BuildVariants
161. Flavor Explosion BuildType * BuildFlavor = BuildVariants BuildVariant are code multipliers
162. Flavor Explosion 2 BuildTypes * 3 BuildFlavors = 6 Variants 6x code 6x tasks 6x resources 6x cache utilization
163. def prop = project.findProperty("enabledBuildVariant") ?: "freeDebug" android.variantFilter { variant -> if (variant.name != prop) { variant.ignore = true } } ./gradlew assemblePaidDebug -PenableBuildVariant=paidDebug
164. Flavor Explosion • Mostly a configuration time win. • Permanently disable flavors you don't need.
165. How to cut build times in half
166. How to cut build times in half.
167. apply plugin: 'com.android.library'
168. assembleDebug assembleRelease
169. 2x code 2x resources 2x tasks 2x tests !!!!!
170. How often does the debug and release code differ for library modules?
171. ./gradlew test runs all tests twice
172. ./gradlew testDebug java module tests don't run
173. Recompile everything when switching to release
174. CI takes twice as long
175. if (engBuild) { } Disable unused flavors in library modules
176. if (engBuild) { android.variantFilter { variant -> } } Disable unused flavors in library modules
177. if (engBuild) { android.variantFilter { variant -> if (variant.name == 'release') { variant.ignore = true } } } Disable unused flavors in library modules
178. if (engBuild) { android.variantFilter { variant -> if (variant.name == 'release') { variant.ignore = true } } } Disable unused flavors in library modules
179. if (engBuild) { android.variantFilter { variant -> if (variant.name == 'release') { variant.ignore = true } } } Disable unused flavors in library modules
180. if (engBuild) { android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore = true } } } Disable unused flavors in library modules
181. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore = true } } Disable unused flavors in library modules android.debug.matchingFallbacks = ['release'] in application module
182. android.variantFilter { variant -> if (variant.name == 'debug') { variant.ignore = true } } Disable unused flavors in library modules
183. Build and test times cut in half
184. Switching to release build doesn't recompile everything
185. Disable stuff! #protip
186. Disable stuff! #protip measure!
187. ./gradlew help Measure configuration time
188. Disable debug variant Disable debug variant and AGP features Disable AGP features Standard build 0.0 200.0 400.0 600.0 800.0 Avg Configuration Time (ms) Average of 10 builds help using Gradle Profiler
189. 783ms to 688ms ~= 13% faster
190. 13% configuration time improvement that scales
191. Regular build assembleDebug
192. assembleDebug Debug disabled, AGP features disabled
193. Disable More Things https://github.com/android/gradle-recipes
194. engBuild Ideas
195. engBuild Ideas •Higher minSdk •Disables Multidex •Disable coreLibraryDesugaring •Strip other languages •Strip large images •Strip out native libraries •Try new versions of AGP https://medium.com/@runningcode/testing-new-versions-of-the-android-gradle-plugin-ea80df978316
196. Always Measure!
197. Upcoming Performance Enhancements
198. File System Watching
199. Fast input snapshotting
200. File System Watching https://blog.gradle.org/introducing-file-system-watching org.gradle.unsafe.watch-fs=true
201. Configuration Caching
202. Configuration Caching
203. Configuration Caching The build scripts of all projects which are part of the build are executed.
204. ext.engBuild = (project.findProperty('engBuild') ?: 'true') .toBoolean()
205. ext.engBuild = providers.gradleProperty("engBuild") .forUseAtConfigurationTime() .orElse("true") .map { it.toBoolean() }
206. if (!engBuild) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }
207. if (!engBuild.get()) { apply plugin: 'com.google.firebase.crashlytics' apply plugin: 'com.google.firebase.firebase-perf' }
208. Improving Android Build Performance Devoxx Belgium 2018 Cédric Champeau https://www.youtube.com/watch?v=StahHZ-L83k https://speakerdeck.com/melix/improving-android-build-performance
209. Build Bigger, Better: Gradle for Large Projects Google I/O 2019 Aurimas Liutikas and Xavier Ducrohet https://www.youtube.com/watch?v=sQC9-Rj2yLI
210. The Secrets of the Build Scan Plugin and the Internals of Gradle Virtual Android Makers Paris 2020 Me https://www.youtube.com/watch?v=lgaqS0pmUzk
211. Measuring remote build cache performance https://medium.com/@runningcode https://github.com/runningcode/gradle-doctor
212. Thank you! Need advice? osacky.com https://gradle.com/enterprise/trial/ https://speakerdeck.com/runningcode/beyond-modularization-scaling-your-android-build-with-gradle