Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Becoming and Staying a Productive Developer wit...

Becoming and Staying a Productive Developer with Build Scans, Build Validation Scripts and Gradle

We will start by discussing some of the top Gradle issues that affect the productivity of Android developers. We will cover build cache misses and configuration cache issues as well as how to debug and solve them with various tools such as with the free Build Scan service, Android Studio debugger and also with Develocity.

We will then demonstrate combining the powerful Build Validation Scripts with automation and notifications to systematically surface new build cache misses and to keep them from reappearing once they have been fixed.

We will close by diving into the just released Artifact Transform insights in the free Build Scan tool. This is one of the most commonly requested features by Android developers. We will also close off by showing how you can use the new Tests API in Develocity to automate your flaky test workflows.

Nelson Osacky

October 26, 2023
Tweet

More Decks by Nelson Osacky

Other Decks in Programming

Transcript

  1. Becoming and Staying a Productive Developer with Build Scans, Build

    Validation Scripts, and Gradle Nelson Osacky, Lead Solutions Engineer Etienne Studer, SVP of Engineering
  2. Etienne • SVP of Engineering, Develocity • Leading the Develocity

    Engineering team • Creator and former lead of the Gradle Solu ti ons team
  3. Nelson • Lead Solu ti ons Engineer, Develocity • Leading

    the Gradle Solu ti ons team • Previously Android Engineer (SoundCloud, Square) • Gradle Plugin Maintainer • Fladle — Easily Scale Instrumenta ti on Tests on Firebase • Gradle Doctor — Ac ti onable Insights for your build
  4. • Helping teams of 50-10k engineers • Upgrading and optimizing

    builds • Migrating builds • Scaling builds Gradle Solutions team
  5. • Improving the Gradle ecosystem • Fixing issues in OSS

    plugins • Aligning with Google and JetBrains • … Gradle Solutions team
  6. DPE is a new software development practice used by leading

    software development organizations to maximize developer productivity and happiness.
  7. Costs of Inefficiency 60s waste * 50 builds / day

    * 50 devs 
 = 42 hours lost / day h tt ps:/ /gradle.com/roi-calculator
  8. What are artifact transforms? • Dependency management concept • Changes

    dependency from incorrect format to requested format via Ar ti fact Transforms • Example: classes jar to dex fi le • Users do not directly request transforms • Gradle schedules Transforms between units of work • Transforms can depend on tasks or transforms
  9. Artifact transforms are now visible in Build Scan! 🥳 Requires

    Develocity 2023.1+, Develocity Gradle Plugin 3.13+, Gradle Build Tool 8.1+ 
 Only visualizes project ar ti fact transforma ti ons in Develocity 2023.3
  10. Why are transforms important in Build Scans? • Understand transform

    dependencies • Understand gaps in ti meline • Understand ar ti fact transform up-to-date, caching, etc • View transform logs
  11. Artifact transforms upcoming work • Ar ti fact Transform Execu

    ti ons insights • Upcoming in Develocity 2023.4, Gradle Build Tool 8.4
  12. What is a cache miss? •Expect a cache hit but

    cache key is di ff erent •Task outcome is the same •Only inputs that a ff ect outcome should be tracked
  13. Build Configuration Differences kotlin.incremental=true kotlin.incremental=true CI Local Incremental has no

    effect on outcome of task h tt ps:/ /youtrack.jetbrains.com/issue/KT-40974/isIncrementalCompila ti onEnabled-should-not-be-marked-as-Input
  14. •LiveLiterals$AnalyticsEvent.kt contains fully qualified path •Fix is to disable LiveLiterals

    composeOptions { useLiveLiterals = false } LiveLiterals https://issuetracker.google.com/issues/233716574
  15. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
  16. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
  17. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
  18. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") } extensions.configure<KspExtension> { arg(RoomSchemaArgProvider(File(projectDir, "schemas"))) } h tt ps:/ /developer.android.com/training/data-storage/room/migra ti ng-db-versions#test
  19. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") }
  20. KSP Cache misses build.gradle.kts class RoomSchemaArgProvider( @get:InputDirectory @get:PathSensitive(PathSensitivity.RELATIVE) val schemaDir:

    File, ) : CommandLineArgumentProvider { override fun asArguments() = listOf("room.schemaLocation=${schemaDir.path}") }
  21. KSP Cache misses KspSubplugin.kt commandLineArgumentProviders.get().forEach { it.asArguments().forEach { argument ->

    options += InternalSubpluginOption("apoption", argument) } } Fixed in KSP 1.08
  22. The case of java.net.URL and .hashCode() Calling URL.hashCode() vs. not

    calling URL.hashCode() changes the serialized value of URL
  23. The case of java.net.URL and .hashCode() URL -> URI @get:Input

    abstract val url: Property<URL> @get:Input abstract val url: Property<URI> https://www.apollographql.com/blog/mobile/kotlin/how-apollo-kotlin-leverages-gradle-enterprise-to-rev-up-build-times/#1-java-net-url-serializes-di ff erently-depending-if-hashcode-was-called ge.apollographql.com
  24. Room Gradle Plugin plugins { id("androidx.room") version "2.6.0" } room

    { schemaDirectory("$projectDir/schemas/") } Use Android Cache Fix Plugin 3.0 and Room Gradle Plugin 2.6.0 Thanks Google!
  25. How Build Caching works Build Caching avoids doing the same

    work already done before by reusing the work unit output stored in a local or remote build cache.
  26. Volatile inputs of work units cause unnecessary execution of work

    units. Volatility in the inputs of work units
  27. Verification if a build is fully cacheable When a build

    is executed twice under the same conditions, all of the second build’s cache-compatible work units should take their output from the build cache.
  28. Automation of the verification process Use the Build Validation Scripts

    to make the process 
 reliable, repeatable, automated, and productive. https:/ /github.com/gradle/gradle-enterprise-build-validation-scripts
  29. Investigation of build cache misses Use Develocity Build Scan Comparison

    to investigate 
 changes in task inputs that caused a build cache miss. https:/ /ge.solutions-team.gradle.com/c/gxypocyl2jwgq/efifh6cmix5xe/task-inputs
  30. Build caching regressions are costly Build changes that make a

    build no longer fully cacheable must be detected and notifications issued so the regression can be promptly investigated and fixed.
  31. Automation of the regression detection The Build Validation Scripts should

    be run on a continuous base in CI and cause the CI run to fail if a build is 
 no longer fully cacheable. ./03-validate-local-build-caching-different-locations.sh 
 -r https:/ /github.com/spring-projects/spring-framework —fail-if-not-fully-cacheable
  32. Cross-machine regression detection The verification process can be extended by

    orchestrating the execution of two builds of the same version on different machines and submitting them to the Build Validation Scripts to verify full cacheability. ./04-validate-remote-build-caching-ci-ci.sh 
 -1 https:/ /ge.solutions-team.gradle.com/s/gxypocyl2jwgq 
 -2 https:/ /ge.solutions-team.gradle.com/s/efifh6cmix5xe —fail-if-not-fully-cacheable
  33. Develocity API h tt ps:/ /docs.gradle.com/enterprise/api-manual •REST based •Build models

    •OpenAPI standard •Allows the genera ti on of models and clients in many languages or technology
  34. /api/builds/{id}/gradle-build-cache-performance Develocity API •Build time •Serial and parallel execution time

    •Task duration, outcome, avoidance savings and cached artifact size •Local and remote build cache performance •Task fingerprinting summary
  35. • Rank teammates by build time • Task wall of

    shame • Find longest non-cacheable tasks across all projects • Custom dashboards and alerts
  36. Retrieve test suites and test case details, filtering by: -

    test container (e.g test suite, etc) - test outcome (success, failed, flaky etc) - tags - custom values - much more and a combination of all! Develocity Test API BETA
  37. Develocity Test API $ curl -H "Authorization: Bearer <token>" \

    https://ge.solutions-team.gradle.com/api/tests/containers \ 
 ?container=* \ 
 &testOutcomes=flaky \ 
 &startTimeMax=2023-09-30T10:00:00Z \ 
 &startTimeMin=2023-08-30T10:00:00Z \ BETA
  38. Develocity Test API { "content": [ { "name": "com.example.Test", "workUnits":

    [], "outcomeDistribution": { "passed": 1, "failed": 2, "skipped": 3, "flaky": 4, "notSelected": 5, "total": 15 }, "buildScanIdsByOutcome": {} }, { "name": "com.example.MavenTest", "workUnits": [], "outcomeDistribution": {}, "buildScanIdsByOutcome": {} } ] } $ curl -H "Authorization: Bearer <token>" \ https://ge.solutions-team.gradle.com/api/tests/containers \ ?container=* \ &testOutcomes=flaky \ &startTimeMax=2023-09-30T10:00:00Z \ &startTimeMin=2023-08-30T10:00:00Z \ BETA
  39. Develocity Test API $ curl -H "Authorization: Bearer <token>" \

    https://ge.solutions-team.gradle.com/api/tests/cases \ ?container=com.example.Test \ &testOutcomes=flaky \ &startTimeMin=2023-08-30T10:00:00Z \ &startTimeMax=2023-09-30T10:00:00Z BETA
  40. Develocity Test API { "content": [ { "name": "com.example.Test", "workUnits":

    [], "outcomeDistribution": { "passed": 1, "failed": 2, "skipped": 3, "flaky": 4, "notSelected": 5, "total": 15 }, "buildScanIdsByOutcome": {} }, ] } $ curl -H "Authorization: Bearer <token>" \ https://ge.solutions-team.gradle.com/api/tests/cases \ ?container=com.example.Test \ &testOutcomes=flaky \ &startTimeMin=2023-08-30T10:00:00Z \ &startTimeMax=2023-09-30T10:00:00Z BETA
  41. Develocity Test API What is a fl aky test according

    to Develocity? • Non-determinis ti c outcome • Fails and succeeds within the execu ti on of a single Gradle task W BETA
  42. Reach out to us after the talk for access to

    Develocity Test API BETA Importing Instrumentation Test Results to Develocity BETA
  43. More Resources Gradle Training https:/ /gradle.com/training Gradle Community Slack https:/

    /gradle.com/slack-invite https://gradle.com/enterprise-customers/oss-projects/ ge.apollographql.com Develocity OSS Projects ge.androidx.dev