Gradle: The Build System That Loves To Hate You

Talk by Aurimas Liutikas (Google) at DroidCon London 2024

Transcript

1. 🐘 Gradle: The Build System That Loves To Hate You The Survival Tips Aurimas Liutikas / AndroidX @ Google @aurimas@androiddev.social
2. 🐘 Who’s Aurimas? 12 years at Google 8.5 years on AndroidX Work closely with Android Studio, Gradle, and Jetbrains
3. 🐘 Is Gradle a bad build tool?
4. 🐘 When it works, it is pretty amazing 🚀
5. 🐘 androidx one of the largest OSS Gradle projects
6. 🐘
7. 🐘
8. 🐘
9. 🐘 local assembleDebug (90 days)
10. 🐘 The end? 🎉
11. 🐘 As project grows, you need a build engineer(s) 󰞳
12. 🐘 Sadly, the path to becoming a Gradle build engineer is treacherous
13. 🐘 Let’s look at some examples
14. 🐘 Let’s create a task
15. 🐘 tasks.create("myTask") { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println( "Spent ${end - start} ms" ) }
16. 🐘 $ ./gradlew lib:myTask > Configure project :lib Spent 5000 ms BUILD SUCCESSFUL in 6s
17. 🐘 $ ./gradlew lib:jar > Configure project :lib Spent 5000 ms BUILD SUCCESSFUL in 6s
18. 🐘 tasks.create("myTask") { // task configuration }
19. 🐘
20. 🐘 class MyBetterTask extends DefaultTask { @TaskAction void doTheWork() { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println("Spent ${end - start} ms") } } tasks.create("myBetterTask", MyBetterTask)
21. 🐘 class MyBetterTask extends DefaultTask { @TaskAction void doTheWork() { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println("Spent ${end - start} ms") } } tasks.create("myBetterTask", MyBetterTask)
22. 🐘 class MyBetterTask extends DefaultTask { @TaskAction void doTheWork() { long start = System.currentTimeMillis() Thread.sleep(5000) long end = System.currentTimeMillis() println("Spent ${end - start} ms") } } tasks.create("myBetterTask", MyBetterTask)
23. 🐘 We have one more issue
24. 🐘 tasks.create("myBetterTask", MyBetterTask) { // some expensive configuration }
25. 🐘 tasks.register("myBetterTask", MyBetterTask) { // some expensive configuration // only runs if task will be executed }
26. 🐘 Lazy configuration / tasks
27. 🐘 - Gradle should deprecate TaskContainer.create and friends https://github.com/gradle/gradle/issues/17705 (open since 2021) Lessons
28. 🐘 - Gradle should deprecate TaskContainer.create and friends https://github.com/gradle/gradle/issues/17705 (open since 2021) - Build engineers need to understand configuration vs execution phase - avoid doing expensive work in configuration phase Lessons
29. 🐘 - Gradle should deprecate TaskContainer.create and friends https://github.com/gradle/gradle/issues/17705 (open since 2021) - Build engineers need to understand configuration vs execution phase - avoid doing expensive work in configuration phase - Gradle plugin authors should use androidx.lint:lint-gradle checks Lessons
30. 🐘 Let’s configure some tasks
31. 🐘 tasks.withType(MyBetterTask) { configureTask() }
32. 🐘 tasks.withType(MyBetterTask) { configureTask() } ❌
33. 🐘 tasks.withType(MyBetterTask) .configureEach { configureTask() }
34. 🐘 tasks.create tasks.getByPath tasks.findByPath tasks.findByName tasks.withType tasks.getByName tasks.all tasks.matching Eager APIs to Avoid tasks.whenTaskAdded tasks.whenObjectAdded tasks.findAll tasks.iterator, thus all of the kotlin-stdlib collection extension functions, e.g. tasks.any { }
35. 🐘 tasks.create tasks.getByPath tasks.findByPath tasks.findByName tasks.withType tasks.getByName tasks.all tasks.matching Eager APIs to Avoid tasks.whenTaskAdded tasks.whenObjectAdded tasks.findAll tasks.iterator, thus all of the kotlin-stdlib collection extension functions, e.g. tasks.any { }
36. 🐘
37. 🐘 - Gradle should deprecate all TaskContainer, TaskCollection, DomainObjectCollection methods that are not register, named, or configureEach Lessons
38. 🐘 - Gradle should deprecate all TaskContainer, TaskCollection, DomainObjectCollection methods that are not register, named, or configureEach - Build engineers should pay attention to eagerly created tasks tasks.register("eagerCanary") { throw Exception("Eagerly configured tasks!") } Lessons
39. 🐘 - Gradle should deprecate all TaskContainer, TaskCollection, DomainObjectCollection methods that are not register, named, or configureEach - Build engineers should pay attention to eagerly created tasks tasks.register("eagerCanary") { throw Exception("Eagerly configured tasks!") } - Gradle plugin authors should use androidx.lint:lint-gradle checks Lessons
40. 🐘 Let’s wire two tasks together
41. 🐘 abstract class FooWriter : DefaultTask() { @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText("Hello") } }
42. 🐘 abstract class FooReader : DefaultTask() { @get:InputFile abstract val inputFile : RegularFileProperty @TaskAction fun doTheWork() { println(inputFile.get().asFile.readText()) } }
43. 🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register<FooWriter>("fooWriter") { outputFile.set(foo) } tasks.register<FooReader>("fooReader") { inputFile.set(foo) }
44. 🐘 $ ./gradlew fooReader > Task fooReader FAILED FAILURE: Build failed with an exception. * What went wrong: A problem was found with the configuration of task 'fooReader' (type 'FooReader'). - Type 'FooReader' property 'inputFile' specifies file 'foo.txt' which doesn't exist.
45. 🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register<FooWriter>("fooWriter") { outputFile.set(foo) } tasks.register<FooReader>("fooReader") { inputFile.set(foo) dependsOn(writer) }
46. 🐘 val foo = layout.buildDirectory.file("foo.txt") val writer = tasks.register<FooWriter>("fooWriter") { outputFile.set(foo) } tasks.register<FooReader>("betterFooReader") { inputFile.set(writer.flatMap { it.outputFile }) }
47. 🐘 - Gradle should deprecate Task.dependsOn Lessons
48. 🐘 - Gradle should deprecate Task.dependsOn - Build engineers should use Provider<Task>.flatmap to establish dependencies Lessons
49. 🐘 Let’s run the build again
50. 🐘 $ ./gradlew lib2:betterFooReader > Task :lib2:betterFooReader Hello BUILD SUCCESSFUL in 1s 2 actionable tasks: 1 executed, 1 up-to-date
51. 🐘 Gradle will always rerun tasks if they don’t declare any outputs
52. 🐘 - Gradle should force task authors to mark tasks as @AlwaysRerunTask or declare an output Lessons
53. 🐘 - Gradle should force task authors to mark tasks as @AlwaysRerunTask or declare an output - Build engineers should always add outputs to tasks tasks.register<FooReader>("betterFooReader") { inputFile.set(writer.flatMap { it.outputFile }) cacheEvenIfNoOutputs() } Lessons
54. 🐘 fun Task.cacheEvenIfNoOutputs() { this.outputs.file(getPlaceholderOutput()) } private fun Task.getPlaceholderOutput(): Provider<RegularFile> { return project.layout.buildDirectory.file( "placeholder/" + this.name.replace(":", "-") ) }
55. 🐘 Let’s investigate what does deleting build/ do to the build
56. 🐘 abstract class FooEnhancer : DefaultTask() { @get:InputFile abstract val inputFile : RegularFileProperty @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText( inputFile.get().asFile.readText() + "!" ) } }
57. 🐘 $ ./gradlew fooEnhancer BUILD SUCCESSFUL in 965ms 2 actionable tasks: 2 executed
58. 🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer Caching disabled for task ':lib2:fooEnhancer' because: Build cache is disabled Caching has not been enabled for the task Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed.
59. 🐘 UP-TO-DATE (default on) Task’s inputs and outputs did not change. FROM-CACHE (default off) Task’s outputs could be found from a previous execution.
60. 🐘 org.gradle.caching=true
61. 🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer Caching disabled for task ':lib2:fooEnhancer' because: Caching has not been enabled for the task Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed.
62. 🐘 @CacheableTask abstract class FooEnhancer : DefaultTask() { @get:[InputFile PathSensitive(PathSensitivity.NONE)] abstract val inputFile : RegularFileProperty @get:OutputFile abstract val outputFile : RegularFileProperty @TaskAction fun doTheWork() { outputFile.get().asFile.writeText(inputFile.get().asFile.readText() + "!") } }
63. 🐘 $ ./gradlew fooEnhancer --info > Task :lib2:fooEnhancer FROM-CACHE Build cache key for task ':lib2:fooEnhancer' is eb57efc9cb2dd2b6587672bac94200b4 Task ':lib2:fooEnhancer' is not up-to-date because: Output property 'outputFile' file build/enhancedFoo.txt has been removed. Loaded cache entry for task ':lib2:fooEnhancer' with cache key eb57efc9cb2dd2b6587672bac94200b4
64. 🐘 - Gradle should enable build caching by default and force task authors to mark tasks as @DisableCachingByDefault or @CacheableTask Lessons
65. 🐘 - Gradle should enable build caching by default and force task authors to mark tasks as @DisableCachingByDefault or @CacheableTask - Build engineers should enable build cache and always mark their tasks @CacheableTask, unless it is an I/O operation such as copy. Lessons
66. 🐘 - Gradle should enable build caching by default and force task authors to mark tasks as @DisableCachingByDefault or @CacheableTask - Build engineers should enable build cache and always mark their tasks @CacheableTask, unless it is an I/O operation such as copy. - Gradle Plugin authors should enable: tasks.withType<ValidatePlugins>().configureEach { enableStricterValidation.set(true) } Lessons
67. 🐘 Let’s share artifacts between projects
68. 🐘 // projectA tasks.register("zip", Zip) { it.from(layout.projectDirectory.file("build.gradle.kts")) destinationDirectory.set(layout.buildDirectory.dir("zip")) archiveFileName.set("my.zip") } // projectB val zipTask = project(":projectA").tasks.named<Zip>("zip") tasks.register<ZipConsumer>("zipConsumer") { inputFile.set(zipTask.flatMap { it.archiveFile }) }
69. 🐘 // projectA tasks.register("zip", Zip) { it.from(layout.projectDirectory.file("build.gradle")) destinationDirectory.set(layout.buildDirectory.dir("zip")) archiveFileName.set("my.zip") } // projectB val zipTask = project(":projectA").tasks.named<Zip>("zip") tasks.register<ZipConsumer>("zipConsumer") { inputFile.set(zipTask.flatMap { it.archiveFile }) }
70. 🐘 $ ./gradlew projectB:zipConsumer --dry-run
71. 🐘 $ ./gradlew projectB:zipConsumer --dry-run :projectA:zip SKIPPED :projectB:zipConsumer SKIPPED
72. 🐘 // projectA val zipTask = tasks.register<Zip>("zip") { … } configurations.register("zipPublish") { isCanBeConsumed = true isCanBeResolved = false attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip"))} } artifacts { add("zipPublish", zipTask) }
73. 🐘 // projectB val zipConfiguration = configurations.create("zipConsumer") { attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip")) } } zipConfiguration.dependencies.add(dependencies.create(project(":projectA"))) val zipFile : FileCollection = zipConfiguration.incoming.artifactView { }.files tasks.register<ZipConsumer>("zipConsumer") { inputFile.from(zipFile) }
74. 🐘 // projectB val zipConfiguration = configurations.create("zipConsumer") { attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip")) } } zipConfiguration.dependencies.add(dependencies.create(project(":projectA"))) val zipFile : FileCollection = zipConfiguration.incoming.artifactView { }.files tasks.register<ZipConsumer>("zipConsumer") { inputFile.from(zipFile) }
75. 🐘 // projectB val zipConfiguration = configurations.create("zipConsumer") { attributes { attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named("special-zip")) } } zipConfiguration.dependencies.add(dependencies.create(project(":projectA"))) val zipFile : FileCollection = zipConfiguration.incoming.artifactView { }.files tasks.register<ZipConsumer>("zipConsumer") { inputFile.from(zipFile) }
76. 🐘 - By default Gradle should flag cross-project task access and make cross project artifact sharing easier Lessons
77. 🐘 - By default Gradle should flag cross-project task access and make cross project artifact sharing easier - Build engineers should test their builds with org.gradle.unsafe.isolated-projects=true to catch such violations Lessons
78. 🐘 Let’s reuse expensive work between projects
79. 🐘 abstract class MyBuildService : BuildService<MyBuildServiceParameters> { val expensiveValue: String by lazy { Thread.sleep(5000) "hello" } } abstract class MyBuildServiceParameters : BuildServiceParameters
80. 🐘 class MyPlugin : Plugin<Project> { override fun apply(target: Project) { val serviceProvider = target.gradle.sharedServices.registerIfAbsent( "myService", MyBuildService::class.java) target.tasks.register("myTask", MyTask::class.java) { it.myService.set(serviceProvider) it.usesService(serviceProvider) } } }
81. 🐘 // projectA plugins { id("myPlugin") }
82. 🐘 $ ./gradlew myTask BUILD SUCCESSFUL in 7s
83. 🐘 // projectB plugins { id("myPlugin") alias(libs.plugins.ktfmt) }
84. 🐘 $ ./gradlew myTask FAILURE: Build failed with an exception. * What went wrong: A problem occurred configuring project ':projectB. > Could not create task ':projectB:myTask'. > Cannot set the value of task ':projectB:myTask' property 'myService' of type org.example.MyBuildService using a provider of type org.example.MyBuildService.
85. 🐘 // projectA plugins { id("myPlugin") } // projectB plugins { id("myPlugin") alias(libs.plugins.ktfmt) } • myPlugin + its dependencies • myPlugin + its dependencies • ktmft + its dependencies build classloaders
86. 🐘 // root build.gradle.kts plugins { alias(libs.plugins.ktfmt) apply false id("myPlugin") apply false }
87. 🐘 $ ./gradlew myTask BUILD SUCCESSFUL in 7s
88. 🐘 - By default Gradle should forbid build classpath difference between projects and if they don’t, catch this failure with a better message https://github.com/gradle/gradle/issues/17559 Lessons
89. 🐘 - By default Gradle should forbid build classpath difference between projects and if they don’t, catch this failure with a better message https://github.com/gradle/gradle/issues/17559 - Build engineers should add all their plugins in root build.gradle.kts while setting apply false Lessons
90. 🐘 and there are many more examples 😭
91. 🐘 Takeaways - If you work at Gradle - deprecate APIs that have better alternatives - enable helpful features by default (build cache & configuration cache) - enforce best build practices by default - If you own a build - enable build cache, configuration cache, and test with project isolation - use well known Gradle builds as inspiration - https://github.com/android/gradle-recipes - https://github.com/slackhq/foundry - https://github.com/androidx/androidx - If you own a Gradle plugin - use androidx.lint:lint-gradle on it and enable stricter validation
92. 🐘 Takeaways - If you work at Gradle - deprecate APIs that have better alternatives - enable helpful features by default (build cache & configuration cache) - enforce best build practices by default - If you own a build - enable build cache, configuration cache, and test with project isolation - use well known Gradle builds as inspiration - https://github.com/android/gradle-recipes - https://github.com/slackhq/foundry - https://github.com/androidx/androidx - If you own a Gradle plugin - use androidx.lint:lint-gradle on it and enable stricter validation
93. 🐘 Takeaways - If you work at Gradle - deprecate APIs that have better alternatives - enable helpful features by default (build cache & configuration cache) - enforce best build practices by default - If you own a build - enable build cache, configuration cache, and test with project isolation - use well known Gradle builds as inspiration - https://github.com/android/gradle-recipes - https://github.com/slackhq/foundry - https://github.com/androidx/androidx - If you own a Gradle plugin - use androidx.lint:lint-gradle on it and enable stricter validation
94. 🐘 Thanks! Any questions?
95. 🐘 Takeaways - If you work at Gradle - deprecate APIs that have better alternatives - enable helpful features by default (build cache & configuration cache) - enforce best build practices by default - If you own a build - enable build cache, configuration cache, and test with project isolation - use well known Gradle builds as inspiration - https://github.com/android/gradle-recipes - https://github.com/slackhq/foundry - https://github.com/androidx/androidx - If you own a Gradle plugin - use androidx.lint:lint-gradle on it and enable stricter validation