Deep Dive into the Android Gradle Plugin (Droidcon UK 2018)

Android developers use it every day, but who knows what actually goes on under the hood of the Android Gradle Plugin?

In this talk, you'll learn about AGP internals and answer questions like:

* What's D8 and R8?
* What is mergeResources and why does it take so long?
* Why are there so many tasks?
* How can I measure and speed up my builds?

You'll come away with a better understanding of the toolchain which will make you a better equipped Android developer!

Video: http://uk.droidcon.com/skillscasts/12718-deep-dive-into-the-android-gradle-plugin

Transcript

1. Deep Dive into the Android Gradle Plugin John Rodriguez
4. Build phases
5. • Initialization • Configuration • Execution Build phases rootProject.name = 'cash' include ':analytics' include ':app' include ':db' include ':keypad' include ':presenters' include ':protos' include ':screens' include ':viewmodels'
6. • Initialization • Configuration • Execution Build phases Settings rootProject.name = 'cash' include ':analytics' include ':app' include ':db' include ':keypad' include ':presenters' include ':protos' include ':screens' include ':viewmodels' project project project project project project project project project
7. • Initialization • Configuration • Execution Build phases project('cash') project(':analytics') project(':db') project(':app') project(':keypad') project(':presenters') project(':protos') project(':screens') project(':viewmodels')
8. • Initialization • Configuration • Execution Build phases project('cash') project(':analytics') project(':db') project(':app') project(':keypad') project(':presenters') project(':protos') project(':screens') project(':viewmodels') > Configure project : > Configure project :analytics > Configure project :app > Configure project :db > Configure project :keypad > Configure project :presenters > Configure project :protos > Configure project :screens > Configure project :viewmodels
9. • Initialization • Configuration • Execution Build phases project('cash') project(':analytics') project(':db') project(':app') project(':keypad') project(':presenters') project(':protos') project(':screens') project(':viewmodels') > Configure project : > Configure project :db > Configure project :protos ./gradlew :db:assembleDebug —configure-on-demand
10. Configuration buildscript { repositories { google() jcenter() }d dependencies { classpath 'com.android.tools.build:gradle:3.2.1' }f }e
11. Configuration buildscript { repositories { google() jcenter() }d dependencies { classpath 'com.android.tools.build:gradle:3.2.1' classpath 'org.jetbrains.kotlin:kotlin-gradle-plugin:1.2.71' classpath 'net.ltgt.gradle:gradle-errorprone-plugin:0.0.16' classpath 'com.squareup.sqldelight:android-gradle-plugin:1.0.0' }f }e
12. Configuration apply plugin: 'com.android.application' apply plugin: 'com.android.library'
13. class GreetingPlugin implements Plugin<Project> { @Override void apply(Project project) { }f }e
14. class GreetingPlugin implements Plugin<Project> { @Override void apply(Project project) { project.tasks.register('Greeting') { it.doLast { println 'Hello from the Greeting Plugin' }b }a }f }e
15. class GreetingPlugin implements Plugin<Project> { @Override void apply(Project project) { project.tasks.register('Greeting') { it.doLast { println 'Hello from the Greeting Plugin' }b }a }f }e project.apply plugin: GreetingPlugin
16. class GreetingPlugin implements Plugin<Project> { @Override void apply(Project project) { project.tasks.register('Greeting') { it.doLast { println 'Hello from the Greeting Plugin' }b }a }f }e apply plugin: GreetingPlugin
17. class GreetingPluginExtension { String message String greeter }a class GreetingPlugin implements Plugin<Project> { @Override void apply(Project project) { project.tasks.register('Greeting') { it.doLast { println 'Hello from the Greeting Plugin' }b }a }f }e apply plugin: GreetingPlugin
18. class GreetingPluginExtension {1 String message String greeter }a class GreetingPlugin implements Plugin<Project> {2 @Override void apply(Project project) {3 def extension = project.extensions.create('greeting', GreetingPluginExtension) project.tasks.register('Greeting') {4 it.doLast {0 println 'Hello from the Greeting Plugin' }b }a }f }e apply plugin: GreetingPlugin
19. class GreetingPluginExtension {1 String message String greeter }a class GreetingPlugin implements Plugin<Project> {2 @Override void apply(Project project) {3 def extension = project.extensions.create('greeting', GreetingPluginExtension) project.tasks.register('Greeting') {4 it.doLast {0 println "${extension.message} from ${extension.greeter}" }b }a }f }e apply plugin: GreetingPlugin
20. apply plugin: GreetingPlugin greeting { message 'Hi' greeter 'Gradle' } $ ./gradlew Greeting > Task :Greeting Hi from Gradle
21. interface Project extends ExtensionAware, PluginAware { } interface ExtensionAware { ExtensionContainer getExtensions(); } interface PluginAware { void apply(…); PluginManager getPluginManager(); }
25. class SomeTask extends DefaultTask { @OutputDirectory File getOutputDir() {…} @InputFiles List<File> getInputFiles() {…} @TaskAction void doStuff() {…}
29. • Initialization • Configuration • Execution Build phases
30. • Initialization • Configuration • Execution Build phases ./gradlew :db:assembleDebug
31. Notifications project.tasks.whenTaskAdded {} project.beforeEvaluate {} project.afterEvaluate {} gradle.projectsEvaluated {} gradle.taskGraph.whenReady {} gradle.taskGraph.beforeTask {} gradle.taskGraph.afterTask {}
32. Debugging Builds
33. $ ./gradlew task
34. $ ./gradlew -Dorg.gradle.debug=true task
35. $ ./gradlew --no-daemon --no-build-cache —rerun-tasks / -Dorg.gradle.debug=true task
36. $ ./gradlew --no-daemon --no-build-cache —rerun-tasks / -Dorg.gradle.debug=true task To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/4.10.2/ userguide/gradle_daemon.html. > Starting Daemon
38. dependencies { compileOnly ‘com.android.tools.build:gradle:3.2.1' }
39. ?
40. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') gradle.taskGraph.allTasks.each { task -> task.taskDependencies.getDependencies(task).each { dep -> // add task -> dep to graph }a }b 'dot -Tpng -O project.dot'.execute(…) }c
42. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') gradle.taskGraph.allTasks.each { task -> task.taskDependencies.getDependencies(task).each { dep -> // add task -> dep to graph }a }b 'dot -Tpng -O project.dot'.execute(…) }c
43. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') gradle.taskGraph.allTasks.each { task -> task.taskDependencies.getDependencies(task).each {…}a }b 'dot -Tpng -O project.dot'.execute(…) }c
44. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') def startTasks = gradle.startParameter.getTaskNames().join(" ") println "command: ./gradlew " + startTasks gradle.taskGraph.allTasks.each { task -> println "task: " + task.name + ", class: " + task.class.name task.taskDependencies.getDependencies(task).each {…}a }b 'dot -Tpng -O project.dot'.execute(…) }c
45. command: ./gradlew app:assembleDebug compileJava -> JavaCompile processResources -> ProcessResources classes -> DefaultTask jar -> Jar checkDebugClasspath -> AppClasspathCheckTask preBuild -> DefaultTask preDebugBuild -> AppPreBuildTask compileDebugAidl -> AidlCompile compileDebugRenderscript -> RenderscriptCompile checkDebugManifest -> CheckManifest generateDebugBuildConfig -> GenerateBuildConfig prepareLintJar -> PrepareLintJar mainApkListPersistenceDebug -> MainApkListPersistence generateDebugResValues -> GenerateResValues generateDebugResources -> DefaultTask mergeDebugResources -> MergeResources …e
46. command: ./gradlew app:assembleDebug compileJava -> JavaCompile processResources -> ProcessResources classes -> DefaultTask jar -> Jar checkDebugClasspath -> AppClasspathCheckTask preBuild -> DefaultTask preDebugBuild -> AppPreBuildTask compileDebugAidl -> AidlCompile compileDebugRenderscript -> RenderscriptCompile checkDebugManifest -> CheckManifest generateDebugBuildConfig -> GenerateBuildConfig prepareLintJar -> PrepareLintJar mainApkListPersistenceDebug -> MainApkListPersistence generateDebugResValues -> GenerateResValues generateDebugResources -> DefaultTask mergeDebugResources -> MergeResources …e
47. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') def startTasks = gradle.startParameter.getTaskNames().join(" ") println "command: ./gradlew " + startTasks gradle.taskGraph.allTasks.each { task -> println "task: " + task.name + ", class: " + task.class.name task.taskDependencies.getDependencies(task).each {…}a }b 'dot -Tpng -O project.dot'.execute(…) }c
48. gradle.taskGraph.whenReady { def dot = new File(rootProject.buildDir, 'project.dot') def startTasks = gradle.startParameter.getTaskNames().join(" ") println "command: ./gradlew " + startTasks gradle.taskGraph.allTasks.each { task -> println "task: " + task.name + ", class: " + task.class.name println " inputs: " task.inputs.each { it.files.each { println " " + it } } println " outputs: " task.outputs.each { it.files.each { println " " + it } } task.taskDependencies.getDependencies(task).each {…}a }b 'dot -Tpng -O project.dot'.execute(…) }c
49. gradle.taskGraph
50. gradle.taskGraph.afterTask { task -> println " inputs: " task.inputs.files.each { println " " + it } println " outputs: " task.outputs.files.each { println " " + it } }
51. > Task :app:checkDebugClasspath inputs: …/gradle-3.2.1.jar …/builder-3.2.1.jar …/tracker-26.2.1.jar …/shared-26.2.1.jar …/crash-26.2.1.jar … outputs: …/app/build/intermediates/checkDebugClasspath/debug
55. buildTypes { debug { applicationIdSuffix '.debug' minifyEnabled false proguardFile file('proguard-cash.pro') proguardFile file('proguard-cash-debug.pro') } release { minifyEnabled true shrinkResources true proguardFile file('proguard-cash.pro') } }
56. productFlavors { flavorDimensions 'environment' internal { dimension 'environment' applicationId 'com.squareup.cash.beta' } production { dimension 'environment' applicationId 'com.squareup.cash' } } debug release
57. production internal release debug
58. production internal release debug internal Release
59. production internal release debug internal Release Release
60. production internal release debug internalDebug internalRelease productionDebug productionRelease release
61. Configurations
62. dependencies { implementation project(‘:api') api project(‘:db’) }a
63. dependencies { implementation project(‘:api') api project(‘:db’) kapt project(‘:dagger’) testImplementation deps.mockito }a
64. Images from https://jeroenmols.com/blog/2017/06/14/androidstudio3/
65. Images from https://jeroenmols.com/blog/2017/06/14/androidstudio3/
66. Images from https://jeroenmols.com/blog/2017/06/14/androidstudio3/
67. :app dependencies { implementation project(':api') }a
68. :app dependencies { implementation project(':api') }a :api dependencies { implementation project(':protos') implementation deps.retrofit implementation deps.rx2 }
69. :app dependencies { implementation project(':api') } :api dependencies { implementation project(':protos') implementation deps.retrofit implementation deps.rx2 }
70. :app dependencies { implementation project(':api') }a :api dependencies { implementation project(':protos') implementation deps.retrofit implementation deps.rx2 }a
71. :app dependencies { implementation project(':api') } :api dependencies { implementation project(':protos') implementation deps.retrofit implementation deps.rx2 } :protos dependencies { api deps.wire.runtime }
72. :app dependencies { implementation project(':api') } :api dependencies { api project(':protos') implementation deps.retrofit implementation deps.rx2 } :protos dependencies { api deps.wire.runtime }
73. :app dependencies { implementation project(':api') } :api dependencies { api project(':protos') implementation deps.retrofit implementation deps.rx2 } :protos dependencies { api deps.wire.runtime }
74. :app dependencies { implementation project(':api') }a :api dependencies { api project(':protos') implementation deps.retrofit api deps.rx2 }b :protos dependencies { api deps.wire.runtime }c
75. :app dependencies { implementation project(':api') } :api dependencies { api project(':protos') api deps.retrofit api deps.rx2 } :protos dependencies { api deps.wire.runtime }
80. class AppClasspathCheckTask extends ClasspathComparisionTask { @TaskAction void run() { compareClasspaths(); } }e
81. class AppClasspathCheckTask extends ClasspathComparisionTask { @Override void onDifferentVersionsFound(…)a{ String message = String.format( "Conflict with dependency '%1$s:%2$s' in project '%3$s'. Resolved versions for runtime classpath (%4$s) and compile classpath (%5$s) differ. This can lead to runtime crashes. To resolve this issue …\n”, …); reporter.reportWarning(EvalIssueReporter.Type.GENERIC, message); } @TaskAction void run() { compareClasspaths(); } }e
82. class AppClasspathCheckTask extends ClasspathComparisionTask { a{…} }e
83. class AppClasspathCheckTask extends ClasspathComparisionTask { a{…} static class ConfigAction TaskConfigAction<AppClasspathCheckTask> { String getName() { return variantScope.getTaskName("check", "Classpath"); } @Override void execute(@NonNull AppClasspathCheckTask task) { task.setVariantName(variantScope.getFullVariantName()); task.runtimeClasspath = variantScope.getArtifactCollection(RUNTIME_CLASSPATH, …); task.compileClasspath = variantScope.getArtifactCollection(COMPILE_CLASSPATH, …); } } }e
86. :moduleA dependencies { api libs.retrofit implementation libs.okhttp } :moduleB dependencies { implementation libs.retrofit } :app dependencies { implementation project(':module-a') implementation project(':module-b') }
87. $ ./gradlew app:dependencies debugCompileClasspath: debug +--- project :module-a | \--- com.squareup.retrofit2:retrofit:2.3.0 | \--- com.squareup.okhttp3:okhttp:3.8.0 | \--- com.squareup.okio:okio:1.13.0 \--- project :module-b … debugRuntimeClasspath: debug +--- project :module-a | +--- com.squareup.retrofit2:retrofit:2.3.0 | | \--- com.squareup.okhttp3:okhttp:3.8.0 -> 3.9.0 | | \--- com.squareup.okio:okio:1.13.0 | \--- com.squareup.okhttp3:okhttp:3.9.0 (*) \--- project :module-b \--- com.squareup.retrofit2:retrofit:2.3.0 (*)
92. class JavaPreCompileTask extends AndroidBuilderTask { @OutputFile File getProcessorListFile() { return processorListFile; } @TaskAction void preCompile() { Set<String> classNames = Sets.newHashSet(); classNames.addAll( convertArtifactsToNames( collectAnnotationProcessors(annotationProcessorConfiguration) )); Gson gson = new GsonBuilder().create(); try (FileWriter writer = new FileWriter(processorListFile)) { gson.toJson(classNames, writer); } } /** * Returns a List of packages in the configuration believed to
93. } } /** * Returns a List of packages in the configuration believed to * contain an annotation processor. * * We assume a package has an annotation processor if it contains the * META-INF/services/javax.annotation.processing.Processor file. */ List<…> collectAnnotationProcessors(ArtifactCollection config) {}
96. > Task :app:mergeDebugResources inputs: ~/.gradle/caches/…/aapt2-3.2.1-4818971-osx.jar/… …/app/build/generated/res/resValues/debug …/app/build/generated/res/rs/debug …/app/src/main/res …/app/src/debug/res outputs: …/app/build/intermediates/blame/res/debug …/app/build/generated/res/pngs/debug …/app/build/intermediates/incremental/mergeDebugResources …/app/build/intermediates/res/merged/debug
99. > Task :app:processDebugResources inputs: ~/.gradle/caches/…/aapt2-3.2.1-4818971-osx.jar/… …/app/build/intermediates/apk_list/…/apk-list.gson …/app/build/intermediates/res/merged/debug …/app/build/intermediates/split_list/…/split-list.gson …/app/build/intermediates/merged_manifests/…/merged outputs: …/app/build/intermediates/incremental/processDebugResources …/app/build/intermediates/processed_res/…/out …/app/build/generated/not_namespaced_r_class_sources/…/r …/app/build/intermediates/res/symbol-table…/…/package-aware-r.txt …/app/build/intermediates/symbols/debug/R.txt
100. Resource processing • Before in aapt: take all resources and outputs one binary • Now in aapt2: compile + link • converts individual resources into binary flat files • merge at the end • uses Gradle incremental check feature for compile avoidance • Improving mergeReleaseResources for libraries • avoids full merge of resources for all the transitive dependencies
101. ArtifactTransform API Request • aar -> android-manifest Internally • aar -> exploded-aar • exploded-aar -> android-manifest • Runs once, cached globally • Downloaded, transformed on demand
102. android.libraryVariants.all { variant -> def runtimeConfiguration = variant.getRuntimeConfiguration() runtimeConfiguration.getIncoming() .artifactView(new Action<ArtifactView.ViewConfiguration>() { void execute(ArtifactView.ViewConfiguration config2) { config2.attributes(new Action<AttributeContainer>() { void execute(AttributeContainer c) { c.attribute(Attribute.of("artifactType", String), “android-assets“) } }) } }) .artifacts.artifacts.each { result -> // This file is an asset def file = result.file }
103. android.libraryVariants.all { variant -> def runtimeConfiguration = variant.getRuntimeConfiguration() runtimeConfiguration.getIncoming() .artifactView(new Action<ArtifactView.ViewConfiguration>() { void execute(ArtifactView.ViewConfiguration config2) { config2.attributes(new Action<AttributeContainer>() { void execute(AttributeContainer c) { c.attribute(Attribute.of("artifactType", String), “android-assets“) } }) } }) .artifacts.artifacts.each { result -> // This file is an asset def file = result.file }
104. ArtifactTransform API “give me all the assets” Before: view = configuration.getFiles() // check if AAR (could have JARS), then find assets… After: view = configuration.getIncoming().artifactView(config -> {...}) Downstream tasks can set up their inputs as a lazy FileCollection • allowing resolution on demand during task execution, as opposed to configuration. FileCollection will only contain the assets and tasks generating them • helpful for variant-awareness by not generating extra tasks
105. • the process of transforming .class bytecode into .dex bytecode for Android
 • D8 - new dex compiler, replaces DX
 • DX takes a class,zip,jar,apk and converts to a single dex
 • When comparing with the current DX compiler, D8 compiles faster and outputs smaller .dex files, while having the same or better app runtime performance.
 • To test in 3.0, android.enableD8=true
 • Set to default in 3.1.
 Dexing
106. DX $ $ANDROID_HOME/build-tools/LATEST/dx --dex —output=out input.jar D8 Debug mode build: $ java -jar build/libs/d8.jar --output out input.jar Release mode build: $ java -jar build/libs/d8.jar --release --output out input.jar Dexing
107. • the process of transforming .class bytecode into .dex bytecode for Android
 • D8 - new dex compiler, replaces DX
 • DX takes a class,zip,jar,apk and converts to a single dex
 • When comparing with the current DX compiler, D8 compiles faster and outputs smaller .dex files, while having the same or better app runtime performance.
 • To test in 3.0, android.enableD8=true
 • Set to default in 3.1.
 Desugaring
108. • Migration from PSI to UAST as of AGP 3.0 • Jetbrains + Google • Kotlin UAST support as of AGP 3.1 • Quick fixes • Lots more detail here: • Lint to the Finish Line: https://www.youtube.com/watch?v=wFe0WZm_xm8 • Kotlin Static Analysis with Android Lint: https://www.youtube.com/watch?v=p8yX5-lPS6o Lint
109. Improving your Builds
110. enum class BooleanOption( override val propertyName: String, override val defaultValue: Boolean = false, override val status: Option.Status = Option.Status.EXPERIMENTAL, override val additionalInfo: String = "" ) : Option<Boolean> { ENABLE_AAPT2("android.enableAapt2", true, DeprecationReporter.DeprecationTarget.AAPT), ENABLE_BUILD_CACHE("android.enableBuildCache", true), ENABLE_PROFILE_JSON("android.enableProfileJson", false), // Used by Studio as workaround for b/71054106, b/75955471 ENABLE_SDK_DOWNLOAD("android.builder.sdkDownload", true, status = Option.Status.STABLE), ENABLE_TEST_SHARDING("android.androidTest.shardBetweenDevices"), ENABLE_DEX_ARCHIVE( "android.useDexArchive", true, aDeprecationReporter.DeprecationTarget.LEGACY_DEXER ),a
111. VERSION_CHECK_OVERRIDE_PROPERTY("android.overrideVersionCheck"), OVERRIDE_PATH_CHECK_PROPERTY("android.overridePathCheck"), ENABLE_DESUGAR( "android.enableDesugar", true, DeprecationReporter.DeprecationTarget.DESUGAR_TOOL), ENABLE_INCREMENTAL_DESUGARING( “android.enableIncrementalDesugaring", true, DeprecationReporter.DeprecationTarget.INCREMENTAL_DESUGARING), ENABLE_GRADLE_WORKERS("android.enableGradleWorkers", false), ENABLE_AAPT2_WORKER_ACTIONS( “android.enableAapt2WorkerActions", true), ENABLE_CORE_LAMBDA_STUBS( "android.enableCoreLambdaStubs", true, DeprecationReporter.DeprecationTarget.CORE_LAMBDA_STUBS), ENABLE_D8("android.enableD8", true, DeprecationReporter.DeprecationTarget.LEGACY_DEXER),
112. AndroidProject.PROPERTY_REFRESH_EXTERNAL_NATIVE_MODEL, status = Option.Status.STABLE), IDE_GENERATE_SOURCES_ONLY( AndroidProject.PROPERTY_GENERATE_SOURCES_ONLY, status = Option.Status.STABLE), ENABLE_SEPARATE_APK_RESOURCES("android.enableSeparateApkRes", true), ENABLE_EXPERIMENTAL_FEATURE_DATABINDING( "android.enableExperimentalFeatureDatabinding", false), ENABLE_SEPARATE_R_CLASS_COMPILATION( "android.enableSeparateRClassCompilation"), ENABLE_JETIFIER("android.enableJetifier", false, status = Option.Status.STABLE), USE_ANDROID_X("android.useAndroidX", false, status = Option.Status.STABLE), ENABLE_UNIT_TEST_BINARY_RESOURCES( "android.enableUnitTestBinaryResources", false), DISABLE_EARLY_MANIFEST_PARSING(
113. ~/.gradle/gradle.properties org.gradle.caching=true
114. ~/.gradle/gradle.properties org.gradle.caching=true …/build.gradle kapt { useBuildCache true }
115. ~/.gradle/gradle.properties org.gradle.caching=true …/build.gradle kapt { useBuildCache !isCi }
116. variantFilter { }a
117. variantFilter { variant -> if (variant.flavors[0].name == 'production' && variant.buildType.name == 'debug') { variant.ignore = true } }a
118. $ ./gradlew app:assembleDebug --profile
130. http://bit.ly/sdk-search-graph
131. Deep Dive into the Android Gradle Plugin @jrodbx