Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Netflix + Gradle, A Journey in Developer Produc...

Netflix + Gradle, A Journey in Developer Productivity

A case study on how Netflix and Gradle have been working together to optimize Netflix Android builds, find bottlenecks, and improve developer productivity. Emmanuel Boudrant from Netflix and Rooz Mohazzabi from Gradle will share wins and findings from their journey.

Emmanuel Boudrant

September 02, 2022
Tweet

Other Decks in Programming

Transcript

  1. Agenda • How Netflix does builds ◦ Gradle / AGP

    / Android Studio / CI/CD ◦ Eng Mode ◦ Modularisation ◦ Gradle profiler • The Developer Productivity Journey ◦ The first Build Scan™ ◦ Build caching, who’s enabled it and who hasn’t? ◦ Identifying build performance bottlenecks ◦ Enabling local and remote build caching • Build Performance Regressions: One year later ◦ Android SDK and JDK version ◦ CI vs Local cache-miss ◦ Code generation ◦ Dependencies management • Wrap Up
  2. How Netflix Does Builds Build Environment & Tools Kapt with

    Dagger/Hilt, Epoxy, Auto Value App Bundles Jetpack Compose Android Studio / AGP 7.2.1 / Kotlin 1.7.10 Gradle project with ~300 Modules and 10+ variants / flavors
  3. SCM : Bitbucket/stash, CI : Jenkins, CD : Spinnaker Pull

    request with: • Static analysis with ktlint, detekt, lint (custom lint) • Unit testing and on-device testing • 4.6K CI builds per week (220K per year) • 2.3K local builds per week (110K per year) How Netflix Does Builds Workflow / CI / CD
  4. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  5. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  6. A Day In the Life of an Android Developer at

    Netflix Commit to Publish • Make change in a feature branch • Open a pull request • Wait for automatic code review • Wait for peers to review • Merge to main • Included in next release train (gated if necessary)
  7. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug
  8. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 11s 3512 actionable tasks: 0 executed, 3512 up-to-date
  9. ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 7m 11s 3512 actionable tasks:

    3512 executed, 0 up-to-date ./gradlew :NetflixMainApplication:assembleDebug BUILD SUCCESSFUL in 2m 1s 3512 actionable tasks: 562 executed, 2950 up-to-date
  10. public final class BuildConfig { public static final boolean DEBUG

    = Boolean.parseBoolean("true"); public static final String LIBRARY_PACKAGE_NAME = "com.netflix… public static final String BUILD_TYPE = "debug"; -/ Field from default config. public static final String BUILD_DATE = "02082022-091312"; }
  11. def buildDate = if (engBuild()) "N-A" else getBuildDate() android {

    defaultConfig { buildConfigField "String", "BUILD_DATE", "\"$buildDate\"" } }
  12. First Results of Remote Build Cache for Netflix CI CI

    build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. EngBuild assembleDebug build time: ~2 min versus assembleDebug build time of ~7 min
  13. Netflix Android Developer Productivity Wins Why Modularize? - More usability

    - Speed up incremental builds - Improved caching and parallelism
  14. Modularization - First modules ✏ ⚙ ⚙ ⚙ ⚙ ✏

    code change ⚙ need recompilation
  15. Modularization - Api / Impl ⚙ ✏ ⚙ ✏ code

    change ⚙ need recompilation
  16. Modularization • Dependency Injection using api and impl modules •

    Multiple DI framework available : Dagger, Hilt, Anvil, in house… • Better architecture, testable … faster incremental build time
  17. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection

    (default: ui:) [1-.2] Module name? (default: ): droidcon Modularization - Tooling
  18. ./gradlew newModule Module type-: 1: library: 2: ui: Enter selection

    (default: ui:) [1-.2] Module name? (default: ): droidcon BUILD SUCCESSFUL in 1m 35s Modularization - Tooling
  19. How Netflix Does Builds Best Practices Eng Build mode •

    Enabled by default • Optimized for build • No app bundle • Less plugins Api & impl modules • Dependency injection • Easier testing • Compilation avoidance • Tooling
  20. Tools to profile your build Keeping track of our build

    performances, measuring the impact of changes…
  21. Benchmark the build • Gradle-profiler • Define build scenarios •

    Run it • Pro Tip: Config time analysis github.com/gradle/gradle-profiler
  22. Grab the lowest hanging DPE fruit: The Build Cache Faster

    build and test feedback cycles Improved DevProd and developer experience Higher quality software
  23. Build Cache and Binary Repositories • A build cache is

    very different and complementary to the concept of a binary repository (sometimes referred to as a dependency cache or artifact repository). • Whereas a binary repository is for caching binaries that represent dependencies from one source repository to another, a build cache caches build actions, like Gradle tasks or Maven goals. A build cache makes building a single source repository faster. • A modern and productive development environment should include both binary repositories and build caches because they are complementary. • Because they share the goal of making the build process more efficient and because certain features overlap, it is not uncommon to assume that both address the same issues.
  24. How it works: Local & Remote Caching Local Cache Remote

    Cache Build Action (Gradle Task, Maven Goal) PUT PUT READ READ Child Node Child Node Child Node • When the build action inputs have not changed, the output can be reused from a previous run. • Build caching can be applied to build actions like compilation, testing, source code generation, Javadoc, checkstyle, and PMD. • Build caching works out of the box for many standard Gradle tasks and Maven goals. • For Maven goals or Gradle tasks that are not pre-configured to be cacheable, there is a cache configuration DSL available. That way custom build actions are also cacheable. • The typical use case is to let CI builds push to a distributed remote build cache, and allow other CI builds as well as developer builds to pull from it. • A local build cache is also available to reuse outputs produced earlier on the same machine.
  25. How to Setup the remote build cache • Free (via

    docker image) • Gradle Enterprise • Use build scans for analysis docs.gradle.com/build-cache-node/
  26. Show me all cacheable tasks that were executed Aggregate build

    scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
  27. Build Scan -> Timeline view -> Critical Path Aggregate build

    scan data from build scans Build caching: who’s using it, who’s not? Debugging cache misses The lowest hanging fruit for speeding CI builds: Remote Build Cache
  28. Build Bottlenecks discovered & Resolved Problem Solution Many cache misses

    locally due to empty directories remaining in source tree Gradle Doctor Plugin: Removing local directories led to more cache hits. Issues with core AGP tasks were causing cache misses Gradle Cache Fix Plugin: Applying the android-cache-fix-gradle-plugin fixed many of these issues. The Gradle team feeds these improvements back to the AGP team to integrate AGP was generating PNG files for SVG inputs, but the outputs were not reproducible resulting in downstream cache misses that were exposed by GE Since the PNG files were not required for debug, this file generation was removed. assembleDebug execution time went from 3:41 to 0:27. The generateBuildConfig task was configured with a timestamp input that resulted in `BuildConfig.java` being different on each build invocation causing caused downstream changes in compiled outputs, and meant that local developers were not able to use these from the remote cache. Moving this volatility to a leaf module resulted in cache hits for all other modules (but still got a cache miss for that particular leaf module).
  29. First Results of Remote Build Cache for Netflix CI CI

    build/test time has improved by 46%. Average CI build time is down from 14.6 min to 8 min. The remote cache is saving 6.6 min per CI build on aggregate for all CI builds. • 24 min of build execution time saved per CI build • Serial Execution Factor: 3.5 • Result: 6.6 Min wait time saved per CI build (24.5/3.5)
  30. Impact of remote caching on PR build times • Average

    PR check build time is down from 39 min to 25 min • Best case ~11 min 25 sec. • The remote cache is saving on average 14 min per PR build. • The Netflix Android team ran 251 PR check builds last week.
  31. Cache misses are back • Different JDK Versions: Identify and

    fix major differences in JDK between CI/local that were causing cache misses on all compile jobs • Setting Android Properties: Identify that cache misses were occurring for CompileLibraryResourcesTask which were fixed by enabling certain Android properties • Older Versions of Dagger: Identify a cacheability issue caused by an older version of Dagger and verify that the upgraded version fixed it • Non deterministic codegen: Identify code generations with differences between CI/local
  32. Use Input Normalization allprojects { normalization { runtimeClasspath { metaInf

    { ignoreAttribute("Implementation-Version") ignoreAttribute("Created-By") } } } }
  33. After we fixed all the cache misses… PR builds at

    record speed: best case 5 min Screen shot 5 min
  34. Managing dependencies • Dependencies centralized : Dependencies.kt in buildSrc/ •

    buildSrc/ is part of build script • Changes in buildSrc/ break incremental compilation
  35. Gradle Version Catalog • All dependencies centralized in a file

    : gradle/libs.versions.toml • Available in Gradle 7.4
  36. Gradle Version Catalog • All dependencies centralized in a file

    : gradle/libs.versions.toml • Available in Gradle 7.4 • Not breaking incremental compilation
  37. Gradle Version Catalog dependencies { implementation libs.my.dep implementation libs.bundles.androidx implementation

    libs.bundles.compose } my-dep = { module = "group:artifact", version = "version" }
  38. Gradle Dependency locking • From Netflix Nebula, available in Gradle

    • Lock all dependencies on a file • Fail the build when lockfiles are not up-to-date
  39. Performance Best Practices ⬢ Upgrade to the latest of all

    tools and plugins: Gradle, AGP, Kotlin ⬢ Tools: Build Scans, Gradle Profiler ⬢ Build Essentials: Daemon, Parallel, Configuration on demand ⬢ Modularize ⬢ L1: Optimize your incremental build cache ⬢ L2: Local Cache ⬢ L3: File Watching, Configuration cache ⬢ Final Boss: Remote Cache ⬢ Enable caching, parallel, file system watching, configuration cache ⬢ Identify and fix performance issues like bottlenecks, cache misses, configuration time slow downs ⬢ Use convention plugins to organize build logic ⬢ Cache Fix plugin ⬢ Gradle Doctor Plugin