Share
Embed
Improve Build Times in Less Time
Build times are never as fast as we want but there is a lot you can do to improve them today without being a Gradle domain expert or buying everyone new computers! In this talk we'll cover a number of different tools, patterns, and common pitfalls to avoid for improving your build times quickly and take your (team's) developer productivity to the next level.
Zac Sweers
April 26, 2022
Transcript
183 slides
-
Zac Sweers – Slack – @ZacSweers Improve Build Times in
Less Time https://speakerdeck.com/zacsweers/improve-build-times-in-less-time -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Tasks @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract
val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs @CacheableTask abstract class PrintMessageTask :
DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable @CacheableTask abstract class
PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable • Task action
@CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable • Task action
• Cacheability @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable • Task action
• Cacheability @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable • Task action
• Cacheability • Con fi gured at con fi guration-time • Should be lazy! project.tasks.register<PrintMessageTask>("printMessage") { inputMessage.set("Hello World!") outputFile.set( project.layout.buildDirectory. fi le( "outputs/logs/message.txt")) } -
Tasks • Inputs and outputs • Parallelizable • Task action
• Cacheability • Con fi gured at con fi guration-time • Should be lazy! @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Tasks • Inputs and outputs • Parallelizable • Task action
• Cacheability • Con fi gured at con fi guration-time • Should be lazy! • Some common types (SourceTask, CompileTask, etc) @CacheableTask abstract class PrintMessageTask : DefaultTask() { @get:Input abstract val inputMessage: Property<String> @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun run() { val message = inputMessage.get() println(message) outputFile.asFile.get().writeText(message) } } -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Projects • Projects contain a set of tasks • Also
contain things like source sets • Can have subprojects • Every directory in between is technically a project • Can have plugins applied to them -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Plugins • Reusable implementations of gradle logic • Well known
examples • Android Gradle Plugin (AGP) • Kotlin Gradle Plugin (KGP) • Many 1st-party plugins • Ours too! github.com/slackhq/slack-gradle-plugin plugins { kotlin("jvm") id("com.android.library") `java-platform` } -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Conf i gurations • Essentially named fi le collections •
“implementation”, “api”, etc dependencies { implementation("com.squareup.moshi:moshi:1.13.0") testImplementation("junit:junit:4.13") } -
Conf i gurations • Essentially named fi le collections •
“implementation”, “api”, etc dependencies { implementation("com.squareup.moshi:moshi:1.13.0") testImplementation("junit:junit:4.13") } -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Conf i guration • Don’t confuse it with con fi
gurations • The time spent from “go” to tasks actually executing -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Caching/Incremental Builds • Multiple kinds • build dir • Build
cache • Local/Remote • Con fi guration cache • GRADLE_USER_HOME • Not Studio caches https://www.liutikas.net/2022/04/19/Caches-Everywhere.html -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Incremental Compilation (IC) • Multiple kinds • Java – Gradle
– JavaCompile • Kotlin – JB/Google – KotlinCompile* • kaptGenerateStubs*Kotlin • kapt*Kotlin • compile*Kotlin • compile*JavaWithKotlinC -
Incremental Compilation (IC) – Java • Compilation avoidance • “Can
we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b -
Incremental Compilation (IC) – Java • Compilation avoidance • “Can
we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b -
Incremental Compilation (IC) – Java • Compilation avoidance • “Can
we not run?” • ABI-based • Incremental Compilation • “Can we only recompile a subset?” https://blog.gradle.org/incremental-compiler-avoidance :project-a :project-b -
Incremental Compilation (IC) – Kotlin • Incremental only for now,
no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky -
Incremental Compilation (IC) – Kotlin • Incremental only for now,
no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b -
Incremental Compilation (IC) – Kotlin • Incremental only for now,
no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b <IC data> produces consumes -
Incremental Compilation (IC) – Kotlin • Incremental only for now,
no avoidance • Coming soon though! • Depends on producer-side IC data • This makes it tricky :project-a :project-b <IC data> produces consumes -
• Multiple kinds • Java – Gradle – JavaCompile •
Kotlin – JB/Google – KotlinCompile* • kaptGenerateStubs*Kotlin • kapt*Kotlin • compile*Kotlin • compile*JavaWithKotlinC Incremental Compilation (IC) – Kotlin -
Incremental Compilation (IC) – Kotlin kotlinc Frontend Backend AnalysisHandler IR
Lowering https://medium.com/google-developer-experts/crash-course-on-the- kotlin-compiler-1-frontend-parsing-phase-9898490d922b Parse/PSI psi2ir -
Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering
Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC -
Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering
Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC -
Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering
Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC -
Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering
Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC -
Incremental Compilation (IC) – Kotlin kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC Frontend
Backend AnalysisHandler IR Lowering Parse/PSI psi2ir -
Incremental Compilation (IC) – Kotlin Frontend Backend AnalysisHandler IR Lowering
Parse/PSI psi2ir kaptGenerateStubs*Kotlin kapt*Kotlin compile*Kotlin compile*JavaWithKotlinC -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Annotation Processing • Non - incremental • Two types of
incremental • ISOLATING (Dagger, Moshi, etc) • AGGREGATING (Glide) javac/kapt KSP -
Gradle Refresher • Tasks • Projects • Plugins • Conf
i gurations • Conf i guration • Caching/Incremental Builds • Incremental Compilation (IC) • Daemons • Annotation Processing -
Things That Are Hurting Your Build Things That Will Improve
Your Build Things That Will Keep Your Build Fast -
Kapt • Minimum 3 compilation tasks • 4 if you
generate/have Java sources (hi, Dagger!) • Brittle to classpath changes • Disables new FIR • Keeps appearing in later problems as a multiplier -
kotlinc Frontend Stub gen .java f i les Kapt (javac)
Kapt .java/.kt f i les -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les javac compileJava More .class f i les -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les javac compileJava More .class f i les -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les javac compileJava More .class f i les -
kotlinc (Frontend only) KSP .java/.kt f i les javac
javaCompile More .class f i les kotlinc (Frontend and Backend) compileKotlin .class f i les -
kotlinc (Frontend only) KSP .kt f i les kotlinc
(Frontend and Backend) compileKotlin .class f i les -
kotlinc (Frontend and Backend) compileKotlin .class f i les
all - open Kotlinx - serialization parcelize moshi etc. https://blog.bnorm.dev/writing-your-second-compiler- plugin-part-1 -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les javac compileJava More .class f i les -
kotlinc (Frontend only) Stub gen .java f i les
Kapt (javac) Kapt .java/.kt f i les kotlinc (Frontend and Backend) compileKotlin .class f i les javac compileJava More .class f i les https://github.com/square/anvil -
kotlinc (Frontend and Backend) compileKotlin .class f i les
Anvil runs here https://github.com/square/anvil -
buildSrc • Regularly invalidates buildscript classpath • It’s like occasionally
having - - rerun - tasks • What’s the alternative? • Version catalogs for deps • Separate repo + artifactory for plugins -
buildSrc • What about live testing? • Use included builds
if ("slack.plugin.internal.localPath" in extra.properties) { val path = (extra["slack.plugin.internal.localPath"] as String) includeBuild(path) { dependencySubstitution { substitute(module("slack.internal.gradle:slack-plugin-internal")) .using(project(":slack-plugin-internal")) } } } -
“What do you think you’d produce if you were hammered?”
– Chet, certi fi ed Gradle expert™ -
Gradle API Surface Area con fi gurations.all { } extensions.get(...)
tasks.getByName(...) var property: String con fi gurations.con fi gureEach { // Lazy! } extensions.con fi gure(...) { // Lazy! } tasks.named(...) { // Lazy! } val property: Property<String> http://autonomousapps.com/blog/rules-for- gradle-plugin-authors.html -
Incremental Compilation Issues • Pure Java projects break IC on
every project that depends on them (directly or transitively) • Java - only, resources - only, etc • Workaround: add a Kotlin f i le • KT-30980 KT-38622 • Code gen tools generating non - deterministic outputs • Resource ABI changes break IC (KT-40772) • Cleaning https://www.zacsweers.dev/ optimizing-your-kotlin-build/ -
Identifying Caching Issues ./gradlew :app:assembleDebug Build 1 Build 2 🗂
Di ff erent directory -
Literally bugs • Kotlin build cache entries often break incremental
compilation. KT-34862 • Gradle AbstractCompile tasks (i.e. KotlinCompile) are sensitive to classpath jar ordering. gradle/gradle#15626 • Error - prone freeCompilerArgs use. KT-41985 https://www.zacsweers.dev/ optimizing-your-kotlin-build/ -
Old tools • JDK • We are on 17, moving
to 18 • Kotlin • Gradle • Gradle plugins • Compiler plugins • Literally anything on your buildscript classpath -
Tune GCs org.gradle.jvmargs=... -XX:+UseG1GC \ -XX:+UnlockExperimentalVMOptions \ -XX:G1NewSizePercent=67 \ -XX:G1MaxNewSizePercent=67
https://github.com/slackhq/slack-gradle-plugin/blob/main/slack- plugin/src/main/kotlin/slack/gradle/tasks/BootstrapTask.kt -
Remote Build caching buildCache { local { isEnabled = true
} remote<HttpBuildCache> { isEnabled = !gradle.startParameter.isOf fl ine setUrl(“…”) // only write to cache from build server isPush = isCi credentials { username = ... password = ... } } } -
android - cache - f i x - gradle -
plugin https://github.com/gradle/android-cache- fi x-gradle-plugin -
Non - transitive R classes • 5.5MB or 8.5% of
Slack APK size • 14% incremental build time improvement w/ layout change • import slack.l10n.R as L10nR -
Disable Android Features # Build features that are disabled by
default in all projects android.defaults.buildfeatures.aidl=false android.defaults.buildfeatures.buildcon fi g=false android.defaults.buildfeatures.renderscript=false android.defaults.buildfeatures.resvalues=false android.defaults.buildfeatures.shaders=false -
Make A Platform Plugin import slack.gradle.Platforms plugins { id("slack.base") `java-platform`
} val catalog = extensions. fi ndByType<VersionCatalogsExtension>() ?. fi nd("libs") ?: error("Could not fi nd libs catalog!") Platforms.applyFromCatalog(project, catalog.get()) https://github.com/slackhq/slack-gradle-plugin/blob/main/slack- plugin/src/main/kotlin/slack/gradle/Platforms.kt -
Other Things • Remove jetif i er (use `checkJetif i
er` task in AGP 7.1) • api/impl projects • api/impl conf i gurations don’t avoid compilations, just control visibility • -
Gradle Enterprise • Remote build cache • Build scans •
Compare builds • Trends/insight/visualization • Export API • Critical path • Tags/custom values • Debugging (everyone has a scan ready) -
Telemetry • Local vs CI (GE tag) • System arch
(rolling out M1 Macs) • Thermals • https: / / github.com/slackhq/slack - gradle - plugin/blob/main/slack - plugin/src/main/kotlin/slack/gradle/util/ThermalsWatcher.kt • VCS state • Env info (OS, version, etc) -
modules - graph - assert https://github.com/jraska/modules-graph-assert moduleGraphAssert { maxHeight =
4 allowed = [':.* -> :core', ':feature.* -> :lib.*'] // regex to match module names restricted = [':feature-[a-z]* -X> :forbidden-to-depend-on'] // regex to match module names con fi gurations = ['api', 'implementation'] // Dependency confs to look. } -
dependency - analysis - android - gradle - plugin https://github.com/slackhq/slack-gradle-plugin/blob/main/slack-plugin/src/main/
kotlin/slack/dependencyrake/DependencyRake.kt /** * Task that consumes the generated advice report json generated by `AdviceTask` * and applies its advice to the project build fi le. This is usually not run * directly, but rather added as a fi nalizer to the `AdviceTask` it reads from. */ class RakeDependencies(...) : AbstractPostProcessingTask() { // ... } -
https://github.com/slackhq/slack-gradle-plugin Convention Plugins plugins { id("com.android.library") kotlin("android") id("com.squareup.anvil") } android
{ namespace = "slack.libraries.emoji.api" compileSdk = 31 defaultCon fi g { minSdk = 26 } buildTypes { debug { matchingFallbacks.add("release") } } compileOptions { sourceCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_1_8 } } androidComponents { beforeVariants { builder -> if (builder.buildType == "debug") { builder.enable = false } else { builder.enableAndroidTest = hasProperty("slack.libraryWithAndroidTest") } } } anvil { generateFactories = true } dependencies { compileOnly(libs.dagger.dagger) implementation(projects.libraries.emoji.data) implementation(projects.libraries.foundation.slackCommons.android) implementation(projects.libraries.imageLoading) implementation(projects.libraries.model) implementation(projects.libraries.telemetry.telemetryPublic) implementation(libs.rxJava.java3) } -
https://github.com/slackhq/slack-gradle-plugin Convention Plugins plugins { id("slack.base") id("com.android.library") kotlin("android") } slack
{ features { dagger() } } dependencies { implementation(projects.libraries.emoji.data) implementation(projects.libraries.foundation.slackCommons.android) implementation(projects.libraries.imageLoading) implementation(projects.libraries.model) implementation(projects.libraries.telemetry.telemetryPublic) implementation(libs.rxJava.java3) } -
Future New Kotlin IC • Consumer - side IC based
on classpath snapshot • Unstable preview available in 1.6.20, targeting 1.7 • Solves many producer - side issues (Java projects, caching, etc) • True compilation avoidance in some cases ( ! ! ) kotlin.incremental.useClasspathSnapshot=true https://youtrack.jetbrains.com/issue/KT-45777 -
Future K2 Compiler • New Kotlin compiler frontend (aka FIR)
• Promises 2-4x build speed improvements • Preview coming in 1.7 https://blog.jetbrains.com/kotlin/2021/10/the-road-to-the-k2-compiler/ -
Future Gradle Project Isolation • Conf i guration cache pt.
2 • True project isolation, will allow parallel conf i guration and tooling model creation (aka Studio sync) • WIP. TL;DR don’t use allprojects/subprojects https://gradle.github.io/con fi guration-cache/#project_isolation systemProp.org.gradle.unsafe.isolated-projects=true -
How can I help? • Report bugs! • Open source
your build stuff • Test/prepare for new features • Use modern APIs • Make your own convention plugin • File feature requests * -
What should I do today? • Audit deprecated/eager APIs •
Update your tools (JDK, Gradle, Kotlin, etc) • Swap KSP for kapt where possible • Single - variant libraries • Disable android features by default • Fix java - only projects breaking IC • Dependency analysis to remove unused deps -
What should I do tomorrow? • Learn some Gradle •
Reduce/remove Kapt • Create gradle - prof i ler scenarios • Modularization • Gradle Enterprise • Apple Silicon • Incorporate new tools prep in planning • OSS your stuff -
People to follow for more! • Tony Robalik – @AutonomousApps
– https: / / dev.to/autonomousapps • Aurimas Liutikas – @_aurimas – https: / / w w w .liutikas.net/blog - posts • Iñaki Villar – @inyaki_mwc • Ryan Harter – @rharter – http: / / ryanharter.com/ • Xavier Ducrohet – @droidxav – https: / / developer.android.com/reference/tools/gradle - api • “Xav says read the docs and stop touching internal APIs” • Ivan Gavrilovic – @gavra0 • Cesar Puerta – @CesarDielo • Nelson Osacky – @nellyspageli – https: / / osacky.com/ • Jendrik Johannes – @jeoj – youtube.com/c/onepieceSoftware – github.com/jjohannes