Ultimate Iteration Speeds with Gradle Configuration Cache

Talk by Aurimas Liutikas (Google) at Droidcon SF 2024

Transcript

1. Ultimate Iteration Speeds with Gradle Configuration Cache Aurimas Liutikas / Google @aurimas@androiddev.social
2. Who’s Aurimas? 12 years at Google 8.5 years on AndroidX Work closely with Android Studio, Gradle, and Jetbrains
3. Idea to Cool Product
4. Not a Single Step Process Only trivial projects can be done in a single step
5. Iteration repetition of a [...] procedure applied to the result of a previous application, [...] successively closer approximations to the solution [...]. - Oxford Languages
6. Faster Iteration (Feedback) Loop Countless research - it leads to faster end-to-end and/or higher quality software
7. Faster Iteration (Feedback) Loop Countless research - it leads to faster end-to-end and/or higher quality software xkcd.com/303/
8. “Useful Work” Work preparation Useful work Work finalization
9. “Useful Work” Work preparation Useful work Work finalization Ideally overhead is 0s
10. I thought this was about Gradle?
11. 1. Initialization Phase 2. Configuration Phase 3. Execution Phase Gradle Build Lifecycle
15. Useful work
16. Gradle Configuration Cache The configuration cache is a feature that significantly improves build performance by caching the result of the configuration phase and reusing this for subsequent builds. org.gradle.configuration-cache=true
17. Without Configuration Cache Feature Initialization phase Configuration phase Execution phase
18. Configuration Cache Miss (e.g. First Run) Initialization phase Configuration phase Configuration serialization / deserialization Execution phase
19. Configuration Cache Hit Configuration deserialization Execution phase
20. But do I really need this?
21. Motivating Example 1: AndroidX Project Modify …/src/androidUnitTest/…/MutatorMutexTest.kt ./gradlew :compose:foundation:foundation:testDebugUnitTest --tests "androidx.compose.foundation.MutatorMutexTest" Repeat!
25. 32x less overhead
26. Motivating Example 2: Parallel Tasks Within a Project By default Gradle only runs one task at a time
28. Motivating Example 2: Parallel Tasks Within a Project Execute tasks belonging to different subprojects in parallel with org.gradle.parallel=true Gradle worker API allows parallel within project tasks, but non-trivial to migrate to org.gradle.configuration-cache=true all tasks parallel by default
30. How does configuration caching work?
31. Example Task abstract class MyTask : DefaultTask() { @get:Input abstract val inputString: Property<String> @get:InputFile abstract val inputFile: RegularFileProperty @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun doIt() { outputFile.get().asFile.writeText( inputFile.get().asFile.readText() + inputString.get()) } }
32. Registering The Task tasks.register<MyTask>("myTask") { inputString.set("hello") inputFile.set(layout.projectDirectory.file("input.txt")) outputFile.set(layout.buildDirectory.file("output.txt")) }
33. ./gradlew myTask Creates .gradle/configuration-cache/**/ buildfingerprint.bin entry.bin projectfingerprint.bin work.bin Magic 󰩃! It just works Adoption
34. gcc2speedscope github.com/gradle/gcc2speedscope ./gradlew assemble -d |gcc2ss -- >> speedscope.json
36. @get:Input abstract val inputString: Property<String> @get:InputFile abstract val inputFile: RegularFileProperty
37. @get:OutputFile abstract val outputFile: RegularFileProperty
38. Task metadata such as task name
39. Task action data such as Java Class and method
40. Oh nice, we done!
41. Oh nice, we done! Or are we?!
42. - Enable warn mode org.gradle.configuration-cache=true org.gradle.configuration-cache.problems=warn - Upgrade Gradle Plugins that have issues - Fix your own build logic Likely Adoption Path
43. What do I need to fix?
44. Custom Types as Inputs class CustomType(val text: String, val number: Int) abstract class MyTask : DefaultTask() { @get:Input abstract val inputCustomType: Property<CustomType> } tasks.register<MyTask>("myTask") { inputCustomType.set(CustomType("someText", 100)) }
45. Custom Types as Inputs $ ./gradlew myTask > Task :myTask FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':myTask'. > Cannot fingerprint input property 'inputCustomType': value 'Build_gradle$CustomType@d00c052' cannot be serialized.
46. Custom Types as Inputs class CustomType( val text: String, val number: Int ) : Serializable
47. val number: Int val text: String
48. file.exists() in Configuration Phase tasks.register<MyTask>("myTask") { val primary = File(projectDir, "primary.txt") if (primary.exists()) { inputFile.set(primary) } else { inputFile.set(File(projectDir, "secondary.txt")) } }
49. If primary.txt Gets Created $ ./gradlew myTask Calculating task graph as configuration cache cannot be reused because the file system entry 'primary.txt' has been created.
50. build/reports/configuration-cache/**/configuration-cache-report.html
52. Where Did We See This https://issuetracker.google.com/issues/291331139 package.xml in SDK https://issuetracker.google.com/issues/289098893 AndroidManifest.xml https://issuetracker.google.com/issues/278767328 analytics.settings https://issuetracker.google.com/issues/295395616 lint common.jar in fatjar task https://youtrack.jetbrains.com/issue/KT-61154/ konan native jar
53. Alternatives - Set up proper task dependencies if input created by a different task - Add both files as @InputFile, mark one with @Optional - Perform exists(), isDirectory() during task execution
54. Reading File Contents During Configuration tasks.register<MyTask>("myTask") { val primary = File(projectDir, "input.txt") inputString.set(primary.readText()) }
55. If input.txt Contents Change $ ./gradlew myTask Calculating task graph as configuration cache cannot be reused because file 'input.txt' has changed.
57. Alternatives providers.fileContents( layout.projectDirectory.file("input.txt") ).asText or declare as @InputFile and read during execution
58. Init Scripts $ ./gradlew --init-script init.gradle myTask Calculating task graph as configuration cache cannot be reused because init script 'init.gradle' has been added.
59. Where Did We See This https://youtrack.jetbrains.com/issue/IDEA-232317 Intellij init scripts for test runs
60. Reading Environment Variables tasks.register<MyTask>("myTask") { inputString.set(System.getenv("FOO").toString()) }
61. Environment Variable Change $ FOO=bar ./gradlew myTask Calculating task graph as configuration cache cannot be reused because environment variable 'FOO' has changed.
63. Alternatives inputString.set( providers.environmentVariable("FOO") )
64. Reading System Properties tasks.register<MyTask>("myTask") { inputString.set(System.getProperty("FOO").toString()) }
65. System Property Change $ ./gradlew myTask3 -DFOO=bar Calculating task graph as configuration cache cannot be reused because system property 'FOO' has changed.
67. Alternatives inputString.set( providers.systemProperty("FOO") )
68. Where Did We See This https://github.com/spdx/spdx-gradle-plugin/issues/94 SPDX Gradle plugin reading all system properties
69. Gradle Properties $ ./gradlew myTask -Pfoo=bar Calculating task graph as configuration cache cannot be reused because the set of Gradle properties has changed.
70. Alternatives - Environment variables (until Gradle fixes https://github.com/gradle/gradle/issues/20969) - providers.gradleProperty("foo")
71. Contents of FileTree Queried During Configuration abstract class MyTask : DefaultTask() { @get:InputFiles abstract val inputFiles: ListProperty<File> ... } tasks.register<MyTask>("myTask") { inputFiles.set(fileTree("mydir").files.sortedBy { it.name }) ... }
72. New File Added to mydir $ ./gradlew myTask Calculating task graph as configuration cache cannot be reused because an input to build file 'build.gradle.kts' has changed.
74. Breakpoint in Gradle
77. Reproduce with Debugger Attached
78. Another Breakpoint in Gradle
79. Step Through to Find myDir
80. Find The Place Where mydir Becomes CC Input
81. Where Did We See This? $ ./gradlew room:room-benchmark:assembleAndroidTest Calculating task graph as configuration cache cannot be reused because an input to plugin 'com.android.internal.library' has changed. https://issuetracker.google.com/issues/285320724
82. Alternatives / Notes - Traverse files during task execution or use FileCollection with filters - Better reporting from Gradle https://github.com/gradle/gradle/issues/25469
83. Oh my, that sure is a lot of work
84. - Prioritized most common tasks - It took many months to go from WARN to ERROR - Reported and helped fix dozens of bugs in Gradle, Android Gradle Plugin, Kotlin Gradle Plugin, Protobuf Gradle Plugin, SPDX Gradle Plugin - Fixed dozens of issues within our own build logic - Upgrades to plugins still can come with regressions (e.g. KGP 2.0.0) → test, report and nudge AndroidX Adoption of Configuration Cache
86. Preventing Regressions - Allow enumerating configuration cache inputs to prevent new additions https://github.com/gradle/gradle/issues/26019 - Allow confirming configuration cache was reused https://github.com/gradle/gradle/issues/27964
87. Takeaways - Use progressive adoption, especially for compile/test iteration loops - Once you get it working, you will not want to go back - If you enable CC in CI, store configuration cache reports for debugging - Bring down configuration cache report entry count to a minimum - Once CC is enabled you’ll notice task performance more (👋 CompileKotlin) - It will not help Gradle Sync in Android Studio / Intellij
88. Any questions?