Advanced Dependency Management with Gradle

Gradle's dependency management is declarative, highly flexible and can cope with demanding requirements posed by enterprise projects. Among its key functionalities are transitive dependency resolution, full compatibility with existing infrastructure like Maven and Ivy repositories, as well as precise dependency resolution control, reporting and diagnostics. I am sure you are already using those features for your own projects but you can go further. In this demo-driven session, we will focus on less prominent, but extremely powerful dependency management capabilities you can use in your build today.

In the course of this talk, we'll touch on topics such as:

* Location, structure and performance features of the Gradle cache
* Configuring dependency caching rules
* Enforcing artifact versions
* Making dependency version recommendations
* Replacing legacy artifacts
* Modifying dependency metadata
* Defining fine-grained metadata rules
* Resolving binary and metadata artifacts with the help of the artifact query API
* Typical enterprise dependency management use cases and how to solve them

Transcript

1. Advanced Dependency Management with Gradle Benjamin Muschko, Gradle Inc.
2. Custom requirements in complex builds Dependency  management  requires  conscious   decisions  and  trade-­‐offs… Transitive  dependencies Caching  strategies Broken  module  versions Accessing  res.  artifacts Enterprise  requirements Modifying  metadata
3. Deep model and API as enabler Dependency  management  runtime  behavior   can  be  fine-­‐tuned…   • Dependency  resolve  rules   • Component  metadata  rules   • Component  selection  rules   • Artifact  Query  API
4. Gradle’s dependency cache Enabler  for  bandwidth  efficiency,   artifact  integrity  and  dependency   resolution  performance
5. Opaque cache I  updated  the  Gradle  version  and  now  all  dependencies  are     downloaded  again.  What  happened  to  the  cache?
6. Opaque cache I  updated  the  Gradle  version  and  now  all  dependencies  are     downloaded  again.  What  happened  to  the  cache? The  cache  structure  is  versioned  and  might  change  across   Gradle  versions. Allows  for  cache  optimizations  and  reorganizations
7. Cache structure Resolved  artifacts Metadata
8. Performance features Lazy  download  of  binary  artifacts Checksum-­‐based  download Minimize  number  of  HTTP  calls TTL  for  dynamic/changing  modules
9. Other features Store  metadata  on  artifact’s  origin Concurrency-­‐safe Optimize  disk  usage
10. Time To Live (TTL) for cached dependencies My  build  consumes  a  SNAPSHOT  dependency.  I  know  that  it   changed  10  mins  ago  but  Gradle  doesn’t  resolve  it  properly.
11. Time To Live (TTL) for cached dependencies My  build  consumes  a  SNAPSHOT  dependency.  I  know  that  it   changed  10  mins  ago  but  Gradle  doesn’t  resolve  it  properly. Gradle  caching  kicks  in.  Changing  and  dynamic  versions  are   not  checked  on  the  remote  repository  for  24  hours.
12. Time To Live (TTL) for cached dependencies My  build  consumes  a  SNAPSHOT  dependency.  I  know  that  it   changed  10  mins  ago  but  Gradle  doesn’t  resolve  it  properly. Gradle  caching  kicks  in.  Changing  and  dynamic  versions  are   not  checked  on  the  remote  repository  for  24  hours. How  can  I  change  the  default  behavior?
13. Fine-tuning dependency caching TTL  for  changing  and  dynamic  modules  can   be  customized 
 configurations.all {
 resolutionStrategy.cacheDynamicVersionsFor 10, 'minutes'
 resolutionStrategy.cacheChangingModulesFor 4, 'hours'
 }
14. DEMO Changing  the  TTL  for  changing  dependencies
15. Resolving a particular Maven SNAPSHOT version All  new  SNAPSHOT  versions  of  library  Y  from  today  are   broken.  Can  I  depend  on  a  SNAPSHOT  with  a  timestamp?
16. Resolving a particular Maven SNAPSHOT version All  new  SNAPSHOT  versions  of  library  Y  from  today  are   broken.  Can  I  depend  on  a  SNAPSHOT  with  a  timestamp? 
 dependencies {
 compile 'org.gradle.training:mylib:1.0-20150423.152626-8'
 }
 Gradle  2.4
17. DEMO Changing  the  TTL  for  dependencies  with   dynamic  version  identifier
18. Cache command line options Working  offline:     --offline Fresh  resolve  of  cache  dependencies:   --refresh-dependencies
19. Debugging resolution failures A  failed  resolution  doesn’t  give  you  the  cause… 
 $ gradle dependencies --configuration compile compile - Compile classpath for source set 'main'. +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.12 FAILED
20. Debugging resolution failures Logging  levels  are  your  friend… 
 $ gradle dependencies --configuration compile --info compile - Compile classpath for source set 'main'. Resource missing. [HTTP GET: https://repo1.maven.org/maven2/ javax/mail/mail/1.3.12/mail-1.3.12.pom] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/ javax/mail/mail/1.3.12/mail-1.3.12.jar] +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.12 FAILED
21. Resolution works, compilation fails Dependency  report  looks  good… 
 $ gradle dependencies --configuration compile compile - Compile classpath for source set 'main'. +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.1 \--- javax.activation:activation:1.0.2
22. Resolution  works,  compilation  fails Compilation  requires  compile  configuration  as   input. 
 $ gradle compileJava FAILURE: Build failed with an exception. * What went wrong: Could not resolve all dependencies for configuration ':compile'. > Could not find mail.jar (javax.mail:mail:1.3.1). Searched in the following locations: https://repo1.maven.org/maven2/javax/mail/mail/1.3.1/ mail-1.3.1.jar
23. Resolution works, compilation fails No  JAR  file  available
24. Using dependency resolve rules We  never  want  our  projects  to  use  the  version  X  of  library  Y.   How  do  we  automatically  pick  version  Z  for  consumers?
25. Using dependency resolve rules We  never  want  our  projects  to  use  the  version  X  of  library  Y.   How  do  we  automatically  pick  version  Z  for  consumers? We  want  to  standardize  on  “good”  versions  for  library  Y.   How  can  we  recommend  this  version  for  consumers?
26. Using dependency resolve rules We  never  want  our  projects  to  use  the  version  X  of  library  Y.   How  do  we  automatically  pick  version  Z  for  consumers? We  want  to  standardize  on  “good”  versions  for  library  Y.   How  can  we  recommend  this  version  for  consumers? Library  Y  got  new  coordinates.  Whenever  it  is  requested  with     old  and  new  coordinates,  how  do  we  force  the  new  one?
27. Enforcing an artifact version 
 // Apply rule to specific configuration
 configurations.all { 
 // Iterate over resolved dependencies
 resolutionStrategy.eachDependency { DependencyResolveDetails details -> 
 // Filter for a specific dependency
 if (details.requested.group == 'org.springframework') { 
 // Force a different version
 details.useVersion '3.2.13.RELEASE'
 }
 }
 }
28. DEMO Enforcing  an  artifact  version  of  Spring   framework
29. Recommending artifact versions • Registry  for  commonly-­‐used  artifact   versions   • Flexible  version  storage  data  structure   • Allow  for  user  to  look  up  default  version   • Similar  concept  as  Maven  POM’s   dependencyManagement  section
30. Repurposing the artifact version identifier Instead  of  a  concrete  version  use  the  self-­‐ defined  keyword  default 
 dependencies {
 compile 'org.springframework:spring-core:default'
 compile 'org.springframework:spring-web:default'
 }
31. DEMO Recommending  an  artifact  version  stored  in  a   simple  data  structure
32. Using other data structures What  happens  if  the  number  of  default   versions  exceeds  10  entries?   
 {
 "defaultVersions": [
 {
 "group": "org.springframework",
 "name": "spring-core",
 "version": "3.2.13.RELEASE"
 },
 ... ]
 }
33. DEMO Recommending  an  artifact  version  stored  in  a   JSON  file
34. Recommending artifact versions Want  a  more  fluent  DSL  that  ties  into   Gradle’s  extensibility  capabilities? 
 dependencies {
 compile defaultVersion('org.springframework', 'spring-core')
 compile defaultVersion('org.springframework', 'spring-web')
 }
35. DEMO Extending  Gradle’s  DSL  for  providing  a   default  artifact  version
36. Open Source plugins Nebula  Recommender  Plugin  (https:/ / github.com/nebula-­‐plugins/nebula-­‐ dependency-­‐recommender)   Spring  Dependency  Management  Plugin   (https:/ /github.com/spring-­‐gradle-­‐plugins/ dependency-­‐management-­‐plugin)
37. Using component metadata rules Customize  dependency  metadata  after   module  descriptor  has  been  downloaded   • Status  scheme   • Changing  status
38. Defining a custom status for published module 
 <?xml version="1.0" encoding="UTF-8"?>
 <ivy-module version="2.0">
 <info organisation="org.gradle.training" module="mylib" revision="1.0-1430314077370" status="snapshot" publication="20150429092757"/>
 ... </ivy-module>
39. Using custom status to resolve latest version 
 repositories {
 ivy {
 url "file://$projectDir/../repo"
 }
 }
 
 configurations {
 myConf
 }
 
 dependencies {
 myConf 'org.gradle.training:mylib:latest.snapshot'
 }
40. Using custom status to resolve latest version ComponentMetadataDetails  describes  a   resolved  component's  metadata 
 dependencies {
 components {
 all { ComponentMetadataDetails details -> 
 // modify descriptor metadata
 }
 }
 }
41. DEMO Assigning  changing  status  based  on  custom   status  scheme
42. Using component selection rules Reject  a  module  selection  based  on  custom   rules  e.g.  group,  name,  version 
 configurations.all.resolutionStrategy {
 componentSelection.all { ComponentSelection selection -> // Filter selection candidate
 if(selection.candidate.group == 'com.google.collections') { // Reject selection with message
 selection.reject("The dependency is deprecated.")
 }
 }
 }
43. DEMO Rejecting  legacy  module  Google  Collections   with  Guava  
44. Replacing legacy modules Module  coordinates  have  changed  but   consumers  still  have  access  to  new  and  legacy   artifacts 
 dependencies {
 modules {
 module('com.google.collections:google-collections') {
 replacedBy('com.google.guava:guava')
 }
 }
 }
45. DEMO Replacing  legacy  module  Google  Collections   with  Guava
46. Using the artifact query API Query  for  raw  artifacts  resolved  by  a   Configuration Sources   artifact Javadoc   artifact Metadata  artifact   (ivy.xml,  pom.xml)
47. Resolving selected component IDs 
 // Use specific configuration
 Configuration configuration = project.configurations.compile
 
 // Determine resolution result
 ResolutionResult result = configuration.incoming.resolutionResult
 
 // Get all dependencies (resolved and unresolved)
 Set<? extends DependencyResult> allDeps = result.allDependencies
 
 // Filter resolved dependencies and collect component IDs
 def componentIds = allDeps.collect { depResult ->
 if(dependencyResult instanceof ResolvedDependencyResult) {
 return it.selected.id
 }
 }
48. Creating and executing query 
 // Get dependency handler
 DependencyHandler handler = project.dependencies
 
 // Create artifact resolution query
 ArtifactResolutionQuery query = handler.createArtifactResolutionQuery()
 .forComponents(componentIds)
 .withArtifacts(JvmLibrary, SourcesArtifact, JavadocArtifact)
 
 // Execute artifact resolution query
 ArtifactResolutionResult result = query.execute()
49. Using resolved artifacts 
 // Iterate over resolved components
 result.resolvedComponents.each { component ->
 // Get sources artifacts
 Set<ComponentArtifactsResult> sourceArtifacts = component.getArtifacts(SourcesArtifact)
 
 sourceArtifacts.each { 
 println "Source artifact for ${component.id}: ${it.file}" 
 }
 
 // Get Javadoc artifacts
 Set<ComponentArtifactsResult> javadocArtifacts = component.getArtifacts(JavadocArtifact)
 
 ...
 }
50. Metadata artifact query types Component type: IvyModule Artifact type: IvyDescriptorArtifact Component type: MavenModule Artifact type: MavenPomArtifact ArtifactResolutionQuery withArtifacts(Class<? extends Component> componentType, Class<? extends Artifact>... artifactTypes) ArtifactResolutionQuery  API
51. DEMO Inspecting  metadata  of  a  resolved  POM   artifact
52. Building an offline repository Use  case:     Bundle  resolved  dependencies  and  ship  them   with  a  distribution   Benefits:   No  need  to  download  artifacts  from  repository   Being  able  to  work  offline  from  the  get-­‐go
53. DEMO Building  an  offline  Maven  repository
54. Solving other real-world Enterprise use cases Common  scenarios  do  not  have  a  one-­‐stop   solution  yet…
55. Locking dependency versions My  project  depends  on  published  modules  of  other  projects.   I  always  want  to  use  the  their  latest  version  during  develop-­‐   ment.  What  about  reproducibility  after  releasing  my  project?
56. Locking dependency versions My  project  depends  on  published  modules  of  other  projects.   I  always  want  to  use  the  their  latest  version  during  develop-­‐   ment.  What  about  reproducibility  after  releasing  my  project? Scenario:  Enforce  strong  integration  pressure Use  version  identifier  latest.release  during  development.   After  releasing  the  project,  pin  to  resolved  versions.
57. Nebula dependency lock plugin https:/ /github.com/nebula-­‐plugins/gradle-­‐dependency-­‐lock-­‐plugin        Binary   Repository          Gradle   build.gradle                    Nebula       dependencies.lock
58. Nebula dependency lock plugin build.gradle 
 dependencies {
 compile 'com.google.guava:guava:14.+'
 compile 'commons-lang:commons-lang:latest.release'
 }
59. Nebula dependency lock plugin build.gradle    dependencies.lock 
 dependencies {
 compile 'com.google.guava:guava:14.+'
 compile 'commons-lang:commons-lang:latest.release'
 } 
 {
 "com.google.guava:guava": { "locked": "14.0.1", "requested": "14.+" },
 "commons-lang:commons-lang": { "locked": "2.5", "requested": "latest.release" }
 }
60. Nebula dependency lock plugin build.gradle    dependencies.lock 
 dependencies {
 compile 'com.google.guava:guava:14.+'
 compile 'commons-lang:commons-lang:latest.release'
 } 
 {
 "com.google.guava:guava": { "locked": "14.0.1", "requested": "14.+" },
 "commons-lang:commons-lang": { "locked": "2.5", "requested": "latest.release" }
 }
61. Custom conflict resolution strategy My  team  is  transitioning  from  build  tool  X  to  Gradle.     Developers  are  used  to  a  different  version  conflict  resolution   strategy.  How  can  we  emulate  the  “known”  strategy?
62. Custom conflict resolution strategy My  team  is  transitioning  from  build  tool  X  to  Gradle.     Developers  are  used  to  a  different  version  conflict  resolution   strategy.  How  can  we  emulate  the  “known”  strategy? Gradle  doesn’t  support  configuring  a  custom  strategy.  It’s     always  latest  version  but  you  can  you  force  module  versions. Scenario:  Smoother  migration  between  tools
63. Example: Simplified breadth-first Forcing  top-­‐level  dependency  versions  (  the   ones  declared  in  the  build  script) 
 configurations.all { config ->
 resolutionStrategy {
 config.allDependencies.each { dep ->
 force "$dep.group:$dep.name:$dep.version"
 }
 } } Possible,  but  better  to  train  the  team  “the  Gradle  way”!
64. Switching between binary & project dependencies We  work  on  multiple  detached  projects  at  the  same  time.   Every  change  to  one  of  the  projects  needs  to  be  published.   To  simplify  my  work  I  want  to  use  project  dependencies.
65. Switching between binary & project dependencies We  work  on  multiple  detached  projects  at  the  same  time.   Every  change  to  one  of  the  projects  needs  to  be  published.   To  simplify  my  work  I  want  to  use  project  dependencies. Three  options  here:  Home-­‐grown  Gradle  code,     Prezi  Pride  or  dependency  substitution  rules  in  Grade  2.5. Scenario:  Flexible  dependency  definitions
66. Open Source approaches 
 dynamicDependencies {
 compile group: 'com.example', name: 'example', version: '1.0'
 } 
 dependencies {
 compile elastic('com.example:example:1.0', 'example')
 } Prezi  Pride  (https:/ /github.com/prezi/pride) Elastic  Deps     (https:/ /github.com/pniederw/elastic-­‐deps)
67. Dependency substitution rules Replace  a  project  dependency  with  an   external  dependency  and  vice  versa 
 configurations.all {
 resolutionStrategy {
 dependencySubstitution {
 withModule('com.example:my-module') {
 useTarget project(':project1') 
 }
 }
 }
 } Gradle  2.5
68. Thank  You! Please  ask  questions… https:/ /www.github.com/bmuschko @bmuschko