Share
Building a Continuous Delivery pipeline with Gradle and Jenkins
Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.
With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.
Benjamin Muschko
Transcript
75 slides
-
Building a Con,nuous Delivery pipeline with Gradle and Jenkins
Benjamin Muschko -
bmuschko bmuschko [email protected] Gradle in Ac,on
Java/Groovy developer Gradle contributor Open Source enthusiast Benjamin Muschko Presenta,on will be available on hFps://speakerdeck.com/bmuschko -
#1 Every commit can result in a release #2
Automate everything! #3 Automated tests are essen,al #4 Done means released -
Test Compile/Unit Tests Integra,on Tests Code Analysis
Package/Deploy UAT Prod Acceptance Tests -
The “revolu,onary” sample applica,on Internet To Do application Browser
Data store Writes Reads -
Project hierarchy todo model repository build.gradle settings.gradle web build.gradle
build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat -
Project hierarchy todo model repository build.gradle settings.gradle web build.gradle
build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat -
Project hierarchy todo model repository build.gradle settings.gradle web build.gradle
build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Define project-‐specific behavior -
Project hierarchy include 'model' include 'repository' include 'web' settings.gradle
Defines which projects are taking part in the build todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat -
Project hierarchy todo model repository build.gradle settings.gradle web build.gradle
build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Always use Wrapper to execute the build! -
Project hierarchy todo model repository build.gradle settings.gradle web build.gradle
build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Externalize concerns into script plugins and organize them in dedicated directory Examples: ü Versioning strategy ü Integra,on and func,onal test setup ü Deployment func,onality ü ... -
Project ar,facts root project model project repository project web
project JAR JAR WAR Deployable ar,fact -
Stages in build pipeline Acceptance stage Functional tests Publish
Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts that system works at a technical level -
Stages in build pipeline Acceptance stage Functional tests Publish
Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts that system works on a func,onal/non-‐func,onal level -
Stages in build pipeline Acceptance stage Functional tests Publish
Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Trigger manually Trigger manually -
Commit stage: Compile/unit tests Rapid feedback (< 5 mins)
Run on every VCS check-‐in Priority: fix broken build Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution -
Commit stage: Integra,on tests Long running tests Require
environment setup Hard to maintain Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution -
Separate tests in project layout todo model repository web
src integTest java main java test java Integra,on test Java sources Produc,on Java sources Unit test Java sources -
Separate tests with SourceSets sourceSets { integrationTest
{ java.srcDir file('src/integTest/java') resources.srcDir file('src/integTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task integrationTest(type: Test) { description = 'Runs the integration tests.' group = 'verification' testClassesDir = sourceSets.integrationTest.output.classesDir classpath = sourceSets.integrationTest.runtimeClasspath testResultsDir = file("$testResultsDir/integration") } Custom test results directory Set source and resources directory Set compile and run,me classpath gradlew integrationTest -
Database integra,on tests stop Database test start Database build
Schema integration Test check build ... -
Database integra,on tests apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn
startAndPrepareDatabase stopDatabase.dependsOn integrationTest task databaseIntegrationTest(dependsOn: stopDatabase) check.dependsOn databaseIntegrationTests stop Database test start Database build Schema integration Test check build ... Separate complex setup logic into script plugin Integrate tasks into build lifecycle -
Picking the “right” code coverage tool Cobertura Offline
bytecode instrumenta,on Source code instrumenta,on Offline bytecode instrumenta,on On-‐the-‐fly bytecode instrumenta,on -
Code coverage with JaCoCo buildscript { repositories
{ mavenCentral() } dependencies { classpath 'org.ajoberstar:gradle-jacoco:0.3.0' } } apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin jacoco { integrationTestTaskName = 'integrationTest' } Configures instrumenta,on for integra,on tests as well Add plugin to the buildscript’s classpath Gradle 1.6 includes this plugin as incuba,ng feature -
Genera,ng coverage reports test integration Test ... ... .exec
jacoco TestReport jacocoIntegration TestReport .exec .html .html -
Commit stage: Code analysis Publish Binaries Commit stage Compile
Unit Tests Integration Tests Code Analysis Assemble Distribution Perform code health check Fail build for low quality Record progress over ,me -
Sta,c code analysis tools Checkstyle FindBugs apply
plugin: 'pmd' pmd { ignoreFailures = true } tasks.withType(Pmd) { reports { xml.enabled = false html.enabled = true } } apply plugin: 'jdepend’ jdepend { toolVersion = '2.9.1' ignoreFailures = true } gradlew check -
Measure quality over ,me with Sonar Sonar database Gradle
Sonar Runner root model repo. web analyzes publishes reads / writes Gradle project -
Applying the Sonar Runner plugin apply plugin: 'sonar-runner'
sonarRunner { sonarProperties { property 'sonar.projectName', 'todo' property 'sonar.projectDescription', 'A task management application' } } subprojects { sonarRunner { sonarProperties { property 'sonar.sourceEncoding', 'UTF-8' } } } gradlew sonarRunner -
Commit stage: Assemble distribu,on Exclude env. configura,on Include
build informa,on Choose versioning strategy Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution -
Versioning strategy 1.0-‐SNAPSHOT 1.0 during development
when released …the Maven way Change version with Maven Release plugin -
Versioning strategy 1.0.134 1.0.134 during development
when released …the Con,nuous Delivery way 1.0.134 Jenkins build number Project version number -
Versioning strategy …implemented with Gradle allprojects { apply
from: "$rootDir/gradle/versioning.gradle" } todo model repository build.gradle settings.gradle web gradle versioning.gradle Contains version implementa,on -
Versioning strategy …implemented with Gradle ext.buildTimestamp = new
Date().format('yyyy-MM-dd HH:mm:ss') version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER) class ProjectVersion { Integer major Integer minor String build ProjectVersion(Integer major, Integer minor, String build) { this.major = major this.minor = minor this.build = build } @Override String toString() { String fullVersion = "$major.$minor" if(build) { fullVersion += ".$build" } fullVersion } } Jenkins Build Number Builds version String representa,on -
Packaging the deployable ar,fact project(':web') { apply plugin: 'war'
task createBuildInfoFile << { def buildInfoFile = new File("$buildDir/build-info.properties") Properties props = new Properties() props.setProperty('version', project.version.toString()) props.setProperty('timestamp', project.buildTimestamp) props.store(buildInfoFile.newWriter(), null) } war { dependsOn createBuildInfoFile baseName = 'todo' from(buildDir) { include 'build-info.properties' into('WEB-INF/classes') } } } Creates file containing build informa,on Include build info file Into WAR distribu,on gradlew assemble -
Commit stage: Publish binaries Version ar,fact(s) Use binary
repository Publish once, then reuse Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution -
Defining build configura,on binaryRepository { url = 'http://mycompany.bin.repo:8081/artifactory' username
= 'admin' password = 'password' name = 'libs-release-local' } environments { test { server { hostname = 'mycompany.test' port = 8099 context = 'todo' username = 'manager' password = 'manager' } } uat { server { hostname = 'mycompany.uat' port = 8199 context = 'todo' username = 'manager' password = 'manager' } } ... } Environment-‐specific configura,on Common configura,on Read creden,als from gradle.proper,es Read creden,als from gradle.proper,es Read creden,als from gradle.proper,es -
Reading build configura,on task loadConfiguration { def env =
project.hasProperty('env') ? project.getProperty('env') : 'test' logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy") def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL()) allprojects { ext.config = parsedConfig } } Initialization phase Conguration phase Execution phase gradlew –Penv=uat ... Assign configura,on to extra property -
Using the Maven Publishing plugin apply plugin: 'maven-publish' ext.fullRepoUrl
= "$config.binaryRepository.url/$config.binaryRepository.name” publishing { publications { webApp(MavenPublication) { from components.web } } repositories { maven { url fullRepoUrl credentials { username = config.binaryRepository.username password = config.binaryRepository.password } } } } Build repository URL from configura,on gradlew publish Assign publica,on name and component -
Acceptance stage: Retrieve binaries Request versioned ar,fact Store
in temp. directory Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries -
Downloading the deployable ar,fact 1.0.34 1.0.32 1.0.33
1.0.34 1.0.34 Test UAT Prod -
Task for downloading ar,fact ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") task
downloadBinaryArchive(type: com.manning.gia.BinaryDownload) { ext { repoPath = project.group.replaceAll('\\.', '/') repoBaseArtifactName = war.baseName repoVersion = project.version.toString() repoArtifact = war.archiveName binaryUrl = "${config.binaryRepository.url}/simple/ ${config.binaryRepository.name}/${repoPath}/ ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}" } sourceUrl = binaryUrl targetBinary = downloadedArtifact } Target loca,on for downloaded ar,fact gradlew downloadBinaryArchive Build exposed binary URL from configura,on -
Acceptance stage: Deploy binaries Deployment on request Make
it a reliable process Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Use process for all envs. -
Deploying to mul,ple environments Test UAT Prod
–Penv=prod –Penv=uat –Penv=test -
Deployment with the Cargo plugin cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote cargoUndeployRemote
{ onlyIf appContextStatus } cargo { containerId = 'tomcat7x' port = config.server.port deployable { file = downloadedArtifact context = config.server.context } remote { hostname = config.server.hostname username = config.server.username password = config.server.password } } Download ar,fact from binary repository and undeploy exis,ng one Only undeploy if URL context exists Use environment-‐ specific configura,on gradlew –Penv=uat cargoDeploy -
Acceptance stage: Func,onal tests Test all UI permuta,ons
Test important use cases Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Run against different envs. -
In-‐container func,onal tests functional JettyStop functional TestClasses functional JettyRun
functional Test check ... ... task functionalTest(type: Test) { ... } task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) { httpPort = functionalJettyHttpPort stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey contextPath = functionalJettyContextPath daemon = true } task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) { stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey } functionalJettyRun.dependsOn functionalTestClasses functionalTest.dependsOn functionalJettyRun functionalJettyStop.dependsOn functionalTest task inContainerFunctionalTest(dependsOn: functionalJettyStop) Func,onal test task Custom JeFy Run task Custom JeFy Stop task -
Execu,ng remote func,onal tests ext { functionalTestReportDir = file("$testReportDir/functional")
functionalTestResultsDir = file("$testResultsDir/functional") functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")] } task remoteFunctionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath testReportDir = functionalTestReportDir testResultsDir = functionalTestResultsDir systemProperties functionalCommonSystemProperties systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/" } gradlew –Penv=test remoteFunctionalTest Build URL from env. configura,on Reuse setup proper,es -
Going further: Capacity tes,ng buildscript { repositories { mavenCentral()
} dependencies { classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9" } } apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin ext.loadTestResources = "$projectDir/src/loadTest/resources" jmeterRun.configure { jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")] jmeterPropertyFile = file("$loadTestResources/jmeter.properties") jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"] logging.captureStandardError LogLevel.INFO } gradlew –Penv=uat jmeterRun -
Let’s bring Jenkins into play! Test UAT
Prod Deployment Acceptance Tests Publish Download Trigger Build Pull Source Code -
Model pipeline as series of jobs Triggered job when
change to SCM is detected -
Always use the Wrapper! Gradle Plugin Run clean
task to remove exis,ng ar,facts -
Use Jenkins environment variable Clearly iden,fy a build
through an expressive build name Build Name SeFer Plugin -
Next job to trigger if
build is stable Defines build number parameter provided to subsequent jobs Parameterized Trigger Plugin -
Archive all files Only archive if build
was successful Clone Workspace SCM Plugin -
Point to separated test results Fail build of quality
gate criteria are not met JaCoCo Plugin Point to JaCoCo files as well as source and class files -
Reuse ini,al workspace Reuse ini,al build number Clone
Workspace SCM Plugin Build Name SeFer Plugin -
Define the target environment Downstream job that requires manual
execu,on Build Pipeline Plugin -
> gradle qa! :askQuestions ! ! BUILD SUCCESSFUL! ! Total
time: 300 secs