What is the Android Cache Fix plugin and why do I need to solve my own cache misses.

This talk covers why cache misses happen, how to diagnose them, and why Gradle can't always automatically fix them for you. We'll also dive under the hood of the Android Cache Fix plugin to figure out some of the dirtiest cache misses in the Android ecosystem and why they still exist.

Give at Droidcon Berlin October 20, 2021

Transcript

1. Android Cache Fix Plugin Nelson Osacky and why do I need to solve my own cache misses?
2. Me • Previously Android Engineer • Large projects • SoundCloud • Square • Small startups • Gradle Plugin Maintainer • Fladle - Easily Scale Instrumentation Tests on Firebase https://github.com/runningcode/fladle • Gradle Doctor - Actionable Insights for your build https://github.com/runningcode/gradle-doctor Solutions Engineer
3. I see a lot of builds from 10 person teams to 1000+ Android, Gradle and Maven
4. • Optimize and understand builds • Increase build stability • Preach Developer Productivity
5. • Improve Gradle ecosystem • Fix issues in open source plugin affecting customers
6. • Meet with Google and Jetbrains to help prioritize issues impacting customers
8. Android Development Feature Development Tech Debt (There are other things too)
9. Tech Debt Refactoring Improving builds
10. Slow builds are Tech Debt And it always pays off
11. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day
12. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day not including lost focus https://gradle.com/roi-calculator
13. Fast Builds Matter
14. Cost of Builds 60s waste * 50 builds / day * 50 devs 
 = 42 hours lost / day hire 5 new people without paying them! no recruiting https://gradle.com/roi-calculator
15. Slow builds are Tech Debt And it always pays off And is easy to justify working on it
17. What is build caching? https://docs.gradle.org/current/userguide/build_cache.html
18. JavaCompile task Source Files Compiler Args Dependencies Class files any task can be made cacheable
19. Cache key Source Files Compiler Args Dependencies
20. Cache key Source Files Compiler Args Dependencies Build cache Class files
22. What is the Android Cache Fix plugin?
23. Solves cache misses/issues in the Android Gradle Plugin
24. What is a cache miss? not "broken" cache
25. "Broken" Cache • Cache miss -> Expected a hit but didn't • Cache hit when inputs changed • Missing outputs from cache • etc
26. How to discover a cache miss?
27. Run two identical builds and compare ./gradlew assembleDebug ./gradlew assembleDebug
28. Task input comparison
29. How does the Android Cache Fix plugin work?
30. Task input comparison
31. Path sensitivity Which part of the file path is considered a part of the key
32. CompileLibraryResourcesTask Source Files Compiler Args Dependencies
33. CompileLibraryResourcesTask build/intermediates/packaged_res/debug Compiler Args Dependencies
34. CompileLibraryResourcesTask build/intermediates/packaged_res/debug
35. CompileLibraryResourcesTask /Users/alice/build/intermediates/packaged_res/debug CompileLibraryResourcesTask /Users/jenkins/build/intermediates/packaged_res/debug
36. CompileLibraryResourcesTask build/intermediates/packaged_res/debug
37. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() {
 }
38. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:InputFile s abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
39. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:PathSensitive(PathSensitivity.ABSOLUTE ) @get:InputFile s abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
40. @CacheableTas k abstract class CompileLibraryResourcesTask : NewIncrementalTask() { @get:PathSensitive(PathSensitivity.RELATIVE ) @get:InputFile s abstract val mergedLibraryResourcesDir: DirectoryProperty
 }
41. How does the cache fix plugin workaround this? https://issuetracker.google.com/issues/155218379
42. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
43. @AndroidIssue( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
44. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
45. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaroun d }
46. @AndroidIssue ( introducedIn = "4.0.0" , fixedIn = ['7.0.0-alpha09'] , link = "https://issuetracker.google.com/issues/155218379" ) class CompileLibraryResourcesWorkaround implements Workaround { // the workaround }
47. Three-card Monte https://www.youtube.com/watch?v=QJv44-Ghj_Y aka Find the Lady, das Kümmelblättchen, Jeu du Bonneteau, Gioco delle tre carte, Bul Karayı Al Parayı
48. Con artists enticing people on Potsdamer Platz, Berlin, to play, and lose money in the game in 2018. https://en.wikipedia.org/wiki/Three-card_Monte
51. Gradle Monte 🎩
52. Gradle Monte 🎩 don't try this at home
53. 1.Add new workaround property with RELATIVE path sensitivity to existing task 2.Set workaround property with value of original property 3.Set original property to dummy (empty) value 4.Gradle performs input snapshotting on RELATIVE workaround property and dummy original property 5.After input snapshotting, set original property back its original value 6.Task executes on original value of original property Gradle Monte 🎩
54. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > }
55. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) }
56. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty() // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) }
57. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) } } }
58. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Dummy file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } }
59. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
60. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
61. tasks.withType(CompileLibraryResourcesTask).configureEach { Task task - > DirectoryProperty originalPropertyValue = objects.directoryProperty( ) // Add a workaround input with the original property value and RELATIVE path sensitivit y task.inputs.dir(originalPropertyValue ) .withPathSensitivity(PathSensitivity.RELATIVE ) .withPropertyName("${propertyName}.workaround" ) .optional( ) gradle.taskGraph.beforeTask { if (it == task) { originalPropertyValue.set(task.getProperty(propertyName) ) def dummyProperty = objects.directoryProperty( ) // Non-existent file to give the DirectoryProperty a value . dummyProperty.set(project.file('/doesnt-exist') ) setPropertyValue(task, dummyProperty ) } } // Set the original task property back to its original valu e task.doFirst { setPropertyValue(task, originalPropertyValue ) } }
62. Please don't ever write that code
63. CompileLibraryResourcesTask is fixed in AGP 7
64. CompileLibraryResourcesTask is fixed in AGP 7*
65. * must enable android.experimental.enableSourceSetPathsMap and android.experimental.cacheCompileLibResources
66. Room cache miss
67. Room is an annotation processor
68. implementation("androidx.room:room-runtime:2.3.0" ) kapt("androidx.room:room-compiler:2.3.0")
69. How many use Room?
72. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } }
73. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) }
74. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) }
75. "$projectDir/schemas"
77. "$projectDir/schemas"
78. "User/no/home/workspace/myapp/schemas"
79. "User/no/home/workspace/myapp/schemas " "User/bob/home/workspace/myapp/schemas"
80. "User/no/home/workspace/myapp/schemas " "User/bob/home/workspace/myapp/schemas " "User/jenkins/home/myapp/schemas"
81. "$projectDir/schemas"
83. CommandLineArgumentProvider
84. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { }
85. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @Overrid e Iterable<String> asArguments() { return [] // TODO } }
86. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { } @Overrid e Iterable<String> asArguments() { return [] // TOD O } }
87. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return [] // TOD O } }
88. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["room.schemaLocation"] } }
89. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
90. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath" ] } b }a
91. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath" ] } b }a
92. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
93. How to use it?
94. annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) }
95. annotationProcessorOptions { compilerArgumentProviders.add(RoomSchemaLocationArgumentProvider("$projectDir/schemas") ) }
96. And now you get cache hits
97. And now you get cache hits?
98. yes, but
101. schemas/1.json // exists and is committed to version contro l schemas/2.json // written when schema is updated
102. schemas/1.json // inpu t schemas/2.json // output
103. schemas // input and output
104. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
105. class RoomSchemaLocationArgumentProvider implements CommandLineArgumentProvider { @InputDirectory @OutputDirector y @get:PathSensitive(PathSensitivity.RELATIVE) final Directory schemaLocationDi r RoomSchemaLocationArgumentProvider(Directory schemaLocationDir) { this.schemaLocationDir = schemaLocationDi r } @Overrid e Iterable<String> asArguments() { return ["-Aroom.schemaLocation=schemaLocationDir.get().asFile.absolutePath"] } }
106. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' has conflicting type annotations declared: @OutputDirectory, @InputDirectory . Reason: The different annotations have different semantics and Gradle cannot determine which one to pick . Possible solution: Choose between one of the conflicting annotations. - In plugin 'kotlin-android' type 'org.jetbrains.kotlin.gradle.internal.KaptWithoutKotlincTask' property 'annotationProcessorOptionProviders$kotlin_gradle_plugin.$0.$0.schemaLocationDir' is annotated with @PathSensitive but that is not allowed for 'OutputDirectory' properties . Reason: This modifier is used in conjunction with a property of type 'OutputDirectory' but this doesn't have semantics . Possible solution: Remove the '@PathSensitive' annotation . Please refer to https://docs.gradle.org/7.2/userguide/ validation_problems.html#incompatible_annotations for more details about this problem.
107. Input and output cannot be overlapping
108. ?
111. Task specific output directory
112. android { defaultConfig { javaCompileOptions { annotationProcessorOptions { arguments += mapOf ( "room.schemaLocation" to "$projectDir/schemas", ) } } } } kaptDebug and kaptRelease write to the same folder
115. •Absolute path sensitivity •Overlapping inputs and outputs •Non exclusive output directory •Kapt removes contents of output directories
117. • Annotation Processors are not tools to write arbitrary fi les on the fi lesystem. • This is what Gradle tasks and Gradle plugins are for.
118. This needs to be a Gradle plugin
119. This really needs to be a Gradle plugin
120. Until then, the Android Cache Fix plugin will exist forever. 😭
121. • Absolute path sensitivity • Overlapping inputs and outputs • Non exclusive output directory • Kapt removes contents of output directories RoomSchemaLocationWorkaround fi xes all of these. Use the Android Cache Fix plugin. Shout out to Gary!
122. https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 What is Google doing about this?
123. Room now has KSP support because performance matters. This doesn't solve the cache miss 🥲
124. What looks better in a promotion case? • Use new technology to rewrite Room to improve build speeds • Fix a 41 star bug filed by community with a Gradle plugin
125. https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 What is Google doing about this? please star ⭐ #fixroom
127. What else does the Android Cache Fix plugin do?
130. What is caching?
131. JavaCompile task Source Files Compiler Args Dependencies Class files
132. Copy task Source Files Source Files
133. Project Directory Up-to-date checking is the file already there?
134. Gradle build cache is there a zip entry with the same build cache key? ~/.gradle/caches Gradle Home
135. Copy task Source Files Source Files
136. Cache key Source Files Build cache Source Files Cache Retrieval
137. Project Directory zip unzip ~/.gradle/caches Build Cache Copy task Source Files Source Files Re-execution Cache store and fetch Happens along every execution
138. Sync, Copy, Zip and Jar should not be cached https://docs.gradle.org/current/userguide/build_cache_concepts.html#non_cacheable_tasks
140. BundleLibraryClassesJar is a Jar task
141. Disabling caching
142. outputs.doNotCacheIf {btrueb}b
143. tasks.withType(BundleLibraryClassesJar).configureEacha{a outputs.doNotCacheIf ("I/O bound tasks do not benefit from caching") {btrueb}b }
145. Thanks to the Android Gradle plugin team at Google
146. https://github.com/gradle/android-cache-fix-gradle-plugin Android Cache Fix plugin plugins { id "org.gradle.android.cache-fix" version "2.4.4" apply false } subprojects { plugins.withType(com.android.build.gradle.api.AndroidBasePlugin) { project.apply plugin: "org.gradle.android.cache-fix" } }
147. Gradle Enterprise Android Features • Find your slowest tests in the test dashboard • Learn which tests are flaky with flaky test insights • Android Instrumentation Test results in Build Scans! Check it out! https://gradle.com/enterprise-customers/oss-projects/
148. https://ge.gradle.org/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=Europe/Berlin&tests.sortField=MEAN_DURATION&tests.unstableOnly=true https://bit.ly/3DUBTu0
150. Tweet, Like, Subscribe, Upvote, Star https://github.com/gradle/android-cache-fix-gradle-plugin https://issuetracker.google.com/issues/132245929 https://issuetracker.google.com/issues/139438151 please star ⭐ #fixroom
151. More Resources Gradle Training https://gradle.com/training/ https://gradle.com/training/introduction-to-gradle https://gradle.com/training/build-cache-deep-dive Gradle Community Slack https://gradle.com/slack-invite
152. Thank you! Gradle Enterprise Trial osacky.com https://gradle.com/enterprise/trial/ https://go.gradle.com/solutions-engineer Gradle Solutions is hiring #fixroom @nellyspageli Find us at our booth here at Droidcon