Remote Build Cache Misses and how to solve them by annoying your colleagues.

This is a talk about remote build cache misses and how we solved them at SoundCloud.
This was presented and Berlin Gradle Night on April 8, 2019.
Here is a recording of the presentation:
https://youtu.be/2frfDMJwvf4?t=384

2. Gradle Remote Cache Misses and how to solve them by annoying your colleagues Nelson Osacky
4. Gradle Task
5. A Task represents a single atomic piece of work for a build.
6. • Copying files • Compiling Classes • Running Tests
7. A task has inputs and ouputs.
8. JavaCompile • Inputs = source files • Outputs = class files
9. Gradle Task Work Input Input Output Output
10. JavaCompile Task Compile Input1=Audio.java Input2=Music.java Output=Audio.class Output=Music.class
11. Build Cache
12. Build Cache • Mechanism to reuse outputs from other builds • Allows builds to refetch outputs when it is determined that the inputs haven’t changed • Can be local (filesystem) or remote (http)
13. A build cache key uniquely defines the task’s outputs based on its inputs.
14. • The task type and its classpath • The names of the output properties • The names and values of properties annotated as described in the section called "Custom task types" • The names and values of properties added by the DSL via TaskInputs • The classpath of the Gradle distribution, buildSrc and plugins • The content of the build script when it affects execution of the task
15. LruCache<BuildCacheKey, BuildOutput>()
16. Local Cache If you have built it before, skip building it again. Remote Cache If anyone else has built it before, skip building it again.
17. Local Push Pull Pull Remote Cache CI
18. Local CI In order to re-use task outputs we need to have the same build cache key across environments.
19. Seed build cache task on CI. ./gradlew assembleDevDebug
21. Build Scans
23. Gradle Enterprise
24. Gradle Enterprise • Aggregates Build Scans • Task Input Comparison
25. Identify what changed between two executions of a task that prevented the output from being reused from a build cache.
26. Local CI
27. Gradle treats whitespace as build cache miss.
29. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
31. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
32. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true" variant.resValue "bool", "verbose_logging", "false" variant.buildConfigField "int", "TEST_RETRY_COUNT", "${getRetryCount()}" } }
33. private static int getRetryCount() { if (isCI) { return 1 } else { return 0 } }
34. Retry mechanism against flaky tests.
35. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 0; }
36. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 1; }
37. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true" variant.resValue "bool", "verbose_logging", "false" variant.buildConfigField "int", "TEST_RETRY_COUNT", "${getRetryCount()}" } }
38. afterEvaluate { android.libraryVariants.all { variant -> variant.resValue "bool", "analytics_enabled", "true" variant.resValue "bool", "verbose_logging", "false" } }
39. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; public static final int TEST_RETRY_COUNT = 1; }
40. public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("true"); public static final String APPLICATION_ID = "com.soundcloud.android"; public static final String BUILD_TYPE = "debug"; public static final int VERSION_CODE = -1; }
41. open class ActivityTest<T : Activity> protected constructor(activityClass: Class<T>) { private val isRunningOnTestLab = Settings.System.getString(contentResolver, "firebase.test.lab") == "true" @JvmField val retryRule = RetryRule(if (isRunningOnTestLab) 1 else 0) }
43. Local src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java src/main/java/com/soundcloud/audio/ CI src/main/java/com/soundcloud/Player.java src/main/java/com/soundcloud/Artwork.java
44. Empty directories are considered part of a task’s input properties
45. tasks.withType(SourceTask).configureEach {at -> t.doFirst { t.source.visit { FileVisitDetails d -> if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }
46. tasks.withType(SourceTask).configureEach {at ->
47. tasks.withType(SourceTask).configureEach {at -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) } } } }
48. t.doFirst {a t.source.visit {aFileVisitDetails d ->a
49. tasks.withType(SourceTask).configureEach { t -> t.doFirst {a t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }
50. t.source.visit {aFileVisitDetails d ->a if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d
51. if (d.file.directory && d.file.listFiles().size() == 0) {b throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c
52. tasks.withType(SourceTask).configureEach { t -> t.doFirst { t.source.visit { FileVisitDetails d -> if (d.file.directory && d.file.listFiles().size() == 0) { throw new IllegalStateException( "Found an empty source directory. This causes build cache misses. Please remove it manually. rmdir " + d.file.absolutePath) }c }d } }
56. buildSrc
57. A special module where code added to buildscript classpath instead of runtime classpath.
58. !"" build.gradle !"" buildSrc # !"" build.gradle # $"" src # !"" main # $"" java # $"" com # $"" enterprise # !"" Deploy.java # $"" DeploymentPlugin.java $"" settings.gradle
61. • All classes that are visible to the task class are considered part of the implementation. • Usually a transient cache miss when adding or removing a plugin • Also a transient miss happens when modifying buildSrc
62. Our buildSrc was removed. Or was it?
65. if (file("buildSrc").exists()) { throw new IllegalStateException("Please run the following command in order to solve your build cache issues: rm -r buildSrc.") }
66. InAppBillingService.aidl
67. android.sourceSets.main { aidl.srcDirs java.srcDirs } build.gradle
71. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach { directory -> directory.traverse(type: FILES) { file -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
72. tasks.named("compileDebugAidl").configure {a doLast {b
73. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile -> file.setText((file as String[]).findAll { !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
74. outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile ->
75. tasks.named("compileDebugAidl").configure {a doLast {b outputs.files.forEach {adirectory -> directory.traverse(type: FILES) {bfile -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
76. file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8')
77. tasks.named("compileDebugAidl").configure { doLast { outputs.files.forEach { directory -> directory.traverse(type: FILES) { file -> file.setText((file as String[]).findAll {d !it.contains('Original file:') }.join(‚\n'), 'utf-8') } } } }
78. Fixed future release of AGP https://issuetracker.google.com/issues/121251997
80. android.sourceSets.main { aidl.srcDirs java.srcDirs }
81. https://github.com/gradle/gradle/issues/8559
82. Third Party Plugins
84. • PathSensitivity.ABSOLUTE • Consider the full paths of files and directories. • PathSensitivity.RELATIVE • Use the location of the file relative to a hierarchy.
85. • If a Task declares a file property without PathSensivity annotation, the default is PathSensivity.ABSOLUTE.
88. protobuf-gradle- plugin
90. Results
91. 10 Dec - 14 Dec
92. 1 Apr - 5 Apr
93. 25% faster builds on avg (49.55s - 36.84s) / 49.55s = 1000+ builds / week ~12s / build
94. Questions?