State of the art Gradle multi-module builds

Large, complex projects often consist of multiple modules based on particular functionality and logical boundaries. The benefits are obvious: better maintainability, less coupling and high cohesion of code. Gradle provides powerful support for configuring and executing modularized projects. In the course of this session, we will discuss the state of the art techniques for tackling highly-customized multi-project builds. Furthermore, this talk will give an outlook on the features to come.

With the help of demos, we'll focus on the following topics:

* Different approaches for structuring multi-project builds
* Adapting to legacy project structures
* Creating projects dynamically as part of the settings logic
* Externalizing Settings logic into a plugin
* Declaring project dependencies
* Switching between project and artifact dependencies
* Dependency substitution rules
* IDE support
* Optimizing multi-project builds for performance

Transcript

1. State of the Art Gradle Multi-Module Builds Benjamin Muschko, Gradle Inc.
2. Complex software consists of modules
3. Benefits of modularization Maintainability Reusability Separation  of  concerns
4. Enablers for implementing separation of concerns Minimize  coupling Maximize  cohesion
5. It’s all about your view of the world
6. Your view might involve legacy structures
7. Configurability is key
8. Multi-module or multi-project? A B C  Component  or  module  is  term   originating  from  software  engineering
9. Multi-module or multi-project? A B C  Component  or  module  is  term   originating  from  software  engineering A B C Project  is  a  term  referring  to   the  domain  object  in  Gradle
10. Projects can depend on each other A B C No  project  depends  on  the     output  of  another  project
11. Projects can depend on each other A B C A B C No  project  depends  on  the     output  of  another  project        Projects  define  one  or  many   dependencies  on  other  project(s)
12. Project structure - hierarchical or flat    Hierarchical:  deeply  nested  structure A B C D E
13. Project structure - hierarchical or flat A Flat:  structure  only  one  level  deep    Hierarchical:  deeply  nested  structure B C A B C D E
14. Project vs. binary dependencies A B C Different  VCS  repositories,   dependencies  are  published   artifacts  in  a  binary  repository VCS  repo VCS  repo VCS  repo
15. Project vs. binary dependencies A B C A B C Different  VCS  repositories,   dependencies  are  published   artifacts  in  a  binary  repository      Projects  usually  live  in            same  VCS  repository VCS  repo VCS  repo VCS  repo VCS  repo
16. Declaring project dependencies Method  Project#project(String)   provides  access  to  Project  reference 
 dependencies {
 compile project(':c')
 } build.gradle A C Creates  JAR  of  project  C  and  adds  it  to  the  compile  classpath  of  project  A.
17. Declaring module dependencies A  binary  dependency  is  resolved  by  its   coordinates  referring  to  a  published  artifact 
 dependencies {
 compile 'org.gradle:c:1.0'
 } build.gradle A C Binary  repository
18. Main role of the settings file Declares  projects  that  should  take  part  in  the   build 
 include 'a'
 include 'b:e'
 include 'c' settings.gradle 
 includeFlat 'a'
 includeFlat 'b'
 includeFlat 'c' settings.gradle                      Flat     project  structure        Hierarchical     project  structure
19. How many? Where to put it? Single  settings  file  per  multi-­‐project  build Root  project  name  is  derived  of  dir.  name  containing  settings  file. A B C settings.gradle A B C master settings.gradle
20. Resolving the settings file 2-­‐step  approach   1. Search  for  file  in  directory  named  master with  same  nesting  level  as  current  directory.   2. If  no  file  is  found  search  in  parent   directories  starting  from  current  directory.
21. Evaluation of logic in settings file Initialization            Phase Configuration              Phase Execution        Phase Gradle’s  Build  Lifecycle Executed  in  the  first  phase  of  Gradle’s  build  lifecycle. settings.gradle
22. Configuration injection Define  common  configuration  for  1..n   projects
23. Configuration injection Define  common  configuration  for  1..n   projects 
 subprojects {
 ...
 } All  subprojects  of  the  project     that  defines  this  logic
24. Configuration injection Define  common  configuration  for  1..n   projects 
 subprojects {
 ...
 } 
 allprojects {
 ...
 } All  subprojects  of  the  project     that  defines  this  logic All  subprojects  of  the  project     that  defines  this  logic  and  the   project  itself
25. Using configuration injection A B C D E
26. Using configuration injection A B C D E 
 subprojects {
 apply plugin: 'java'
 } C D E Java Java
27. Using configuration injection A B C D E 
 subprojects {
 apply plugin: 'java'
 } 
 allprojects {
 apply plugin: 'java'
 } C D E C D E Java Java Java Java Java
28. Filtering based on subproject configuration A B C D E 
 def webProjects() {
 subprojects.findAll { subproject -> subproject.plugins.hasPlugin('war') }
 }
 
 gradle.projectsEvaluated {
 configure(webProjects()) {
 ...
 }
 }
29. Filtering based on subproject configuration A B C D E 
 def webProjects() {
 subprojects.findAll { subproject -> subproject.plugins.hasPlugin('war') }
 }
 
 gradle.projectsEvaluated {
 configure(webProjects()) {
 ...
 }
 } Alternative:   evaluationDependsOnChildren()
30. Task execution based on name matching Find  and  execute  task(s)  with  requested  name A B C D E 
 ~/dev/myApp/c $ gradle war :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava :e:processResources UP-TO-DATE :e:classes :e:war BUILD SUCCESSFUL
31. Task execution based on name matching Find  and  execute  task(s)  with  requested  name A B C D E 
 ~/dev/myApp/c $ gradle war :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava :e:processResources UP-TO-DATE :e:classes :e:war BUILD SUCCESSFUL Project  E  depends  on  D
32. Project and task execution path notation Full  path  notation  allows  for  being  specific A B C D E 
 ~/dev/myApp/c $ gradle e:compileJava :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava BUILD SUCCESSFUL Project Task
33. Selecting project(s) A B C D E
34. Selecting project(s) A B C D E 
 project.rootProject A
35. Selecting project(s) A B C D E 
 project.rootProject A 
 project.project(':c').children D E
36. Selecting project(s) A B C D E 
 project.rootProject A 
 project.project(':c').children D E 
 project.project(':')    Top-­‐level     root  project
37. Case-study application • High-­‐level  information  about  available   Gradle  plugins   • Query  Bintray  API  to  retrieve  data   • Render  information  on  page
38. DEMO Showcasing  the  application  functionality
39. Single build file Only  recommended  for  builds  with  little  logic 
 project(':a') {
 ...
 }
 
 project(':b') {
 ...
 }
 
 project(':c') {
 ...
 } 
 include 'a'
 include 'b'
 include 'c' build.gradle settings.gradle Requires  explicit  use  of  project  method.
40. DEMO Using  a  single  build  file  to  structure  logic
41. Multiple build files Best  default  option  for  organizing  build  logic A B C build.gradle A  project  is  not  required  to  have  a  build.gradle  file. 
 include 'a'
 include 'b'
 include 'c' settings.gradle build.gradle build.gradle build.gradle
42. DEMO Using  a  multiple  build  files  to  structure  logic
43. Hybrid approach Avoid  using  this  option  as  it  might  cause   potential  maintenance  issues  down  the  road A B C build.gradle 
 project(':b') {
 ...
 } Root  project  build.gradle 
 include 'a'
 include 'b'
 include 'c' settings.gradle build.gradle build.gradle
44. DEMO Using  a  hybrid  approach  to  structure  logic
45. Partial builds of a subproject “build  this  project  but  none  of  its  project  dependencies”:   --no-rebuild, -a “build  &  test  this  project  and  its  project  dependencies”:   buildNeeded “build  &  test  this  project  and  projects  that  depend  on   current  project”:   buildDependents
46. DEMO Using  partial  build  options
47. Highly customized Settings configuration Typical  scenarios:   • Changing  default  project  names   • Changing  default  project  directories   • Changing  default  build  file  names   • Including  projects  through  external  file   definition
48. Changing project names I  inherited  a  legacy  project  structure.  Directory  names  are   not  descriptive  enough.  How  do  I  change  the  project  names?
49. Changing project names I  inherited  a  legacy  project  structure.  Directory  names  are   not  descriptive  enough.  How  do  I  change  the  project  names? 
 rootProject.name = 'myApp'
 include 'shared', 'web', 'client' 
 rootProject.children.each {
 it.name = 'myApp-' + it.name
 }
 settings.gradle
50. DEMO Assigning  custom  project  names
51. Changing build file names It  gets  pretty  confusing  in  my  IDE  if  I  work  on  multiple   build.gradle  files  in  parallel.  Can  I  give  them  custom  names?
52. Changing build file names It  gets  pretty  confusing  in  my  IDE  if  I  work  on  multiple   build.gradle  files  in  parallel.  Can  I  give  them  custom  names? 
 rootProject.name = 'myApp'
 include 'shared', 'web', 'client' 
 rootProject.children.each {
 it.buildFileName = it.name + '.gradle' - 'myApp' }
 settings.gradle
53. DEMO Assigning  custom  project  file  names
54. Including projects by external file definition My  list  of  projects  and  their  build  file  names  has  become  very     long.  Can  I  externalize  this  definition  into  another  file?
55. Including projects by external file definition My  list  of  projects  and  their  build  file  names  has  become  very     long.  Can  I  externalize  this  definition  into  another  file? 
 {
 "projects": {
 "shared": {
 "name": "myApp-shared",
 "buildFileName": "shared.gradle"
 },
 "web": {
 "name": "myApp-web",
 "buildFileName": "web.gradle"
 }
 }
 }
56. DEMO Deriving  project  inclusions  from  JSON  file
57. Externalizing Settings logic into a plugin As  soon  as  the  logic  becomes  more  complex,   write  a  plugin. 
 package org.gradle.training
 
 import org.gradle.api.Plugin
 import org.gradle.api.initialization.Settings
 
 class DynamicProjectsSettingsPlugin implements Plugin<Settings> {
 @Override
 void apply(Settings settings) {
 // work on Settings instance
 }
 }
58. DEMO Writing  a  Settings  plugin
59. Working on componentized projects A B C VCS  repo D VCS  repo E VCS  repo F VCS  repo Deliverable  artifact  comprises  of  all  components.
60. Declared module dependencies A B C VCS  repo D VCS  repo E VCS  repo F VCS  repo 
 dependencies {
 compile 'org.gradle:f:1.0'
 } 
 dependencies {
 compile 'org.gradle:d:2.3'
 }
61. Working on multiple components during development A B C VCS  repo D VCS  repo E VCS  repo F VCS  repo Requires  publishing  of  F  and  D.  What  about  E?
62. Replacing module with project dependencies A B C VCS  repo D VCS  repo E VCS  repo F VCS  repo 
 dependencies {
 compile project(':f')
 } 
 dependencies {
 compile project(':d')
 }
63. The problem Dynamically  switching  between  module  and   project  dependencies   • Different  scenarios  (e.g.  development   environment  vs.  CI).   • How  do  I  tell  Gradle  which  one  to  use?   • How  do  I  make  it  work  in  the  IDE?
64. Open Source approaches 
 dynamicDependencies {
 compile group: 'com.example', name: 'example', version: '1.0'
 } 
 dependencies {
 compile elastic('com.example:example:1.0', 'example')
 } Prezi  Pride  (https:/ /github.com/prezi/pride) Elastic  Deps     (https:/ /github.com/pniederw/elastic-­‐deps)
65. DEMO Exploring  a  simple  custom  solution
66. Dependency substitution rules Gradle’s  API  cousin  to  dependency  resolve   rules   • Replace  external  with  project  dependency   • Replace  project  with  external  dependency Gradle  2.5
67. Replace external with project dependency Scenario:  You  are  working  on  a  module  and   want  to  constantly  integrate  the  code  -­‐  “fast   turnaround”. 
 configurations.all {
 resolutionStrategy {
 dependencySubstitution {
 withModule('org.utils:api') {
 useTarget project(':api') 
 }
 }
 }
 }
68. Replace project with external dependency Scenario:  You  know  the  code  of  the  module   hasn’t  changed  -­‐  “build  avoidance”. 
 configurations.all {
 resolutionStrategy {
 dependencySubstitution {
 withProject(':api') {
 useTarget 'org.utils:api:1.3' }
 }
 }
 }
69. More down the road… Current  functionality  is  a  low-­‐level  API   intended  to  build  on  top. Low-­‐level            API DSL              IDE   capabilities
70. DEMO Using  dependency  substitution  rules  in   practice
71. IDE support No  support  for  IntelliJ  at  the  moment Eclipse  STS  available  through  custom           model  used  with  tooling  API Planned  support  for  Buildship
72. DEMO Showcasing  Eclipse  STS  support
73. Managing large multi-project builds Configuration  time  increases  with  #  projects
74. Upgrade to >= Gradle 2.4 Significant  performance  improvements  in   Gradle  internals Speeds  up  build  up  to  40%!
75. Performance-related command line options Building  projects  in  parallel:     --parallel, --parallel-threads Only  configure  relevant  projects:   --configure-on-demand Avoid  rebuilding  project  dependencies:   --no-rebuild, -a
76. Using the new configuration model Caching  and  reusing  the  project  model A B C D E    Cache Work  in  progress
77. Modeling projects as SourceSets Minimize  time  spend  in  configuration  phase   by  not  using  projects/project  dependencies A B C D E SourceSet  A SourceSet  B SourceSet  C SourceSet  D SourceSet  E compileClasspath Requires  a  lot  of  custom  code!
78. Thank  You! Please  ask  questions… https:/ /www.github.com/bmuschko @bmuschko