Taming Jenkins

Continuous Integration is often the weak link in the move from infrastructure as a collection of pets to a well-organized cattle ranch. Jenkins, while extremely useful as an in-house CI factory, is more often than not a very opaque piece of legacy infrastructure, holding key company information hostage in a maze of XML files.

In this talk, we will look at recent changes in Jenkins that make it more compatible with the infrastructure as code movement, building a repeatable, scalable, and secure build infrastructure with simple configuration.

Transcript

1. TAMING JENKINS TAMING JENKINS BUILDING A CLOUD-NATIVE CI ENVIRONMENT BUILDING A CLOUD-NATIVE CI ENVIRONMENT 1
2. 2 . 1
3. HALLO HALLO : Three-line Bio CTO & Co-founder at Distributed systems and monitoring enthusiast Open-Source developer Clojure Libraries, OpenBSD, Riemann, Collectd, and more. @pyr Exoscale 3 . 1
4. TAMING JENKINS TAMING JENKINS Building a Cloud-Native CI environment 4 . 1
5. EXOSCALE EXOSCALE Infrastructure as a service Part of A1 Digital Zones in Frankfurt, Vienna, Zürich, Geneva 5 . 1
6. EXOSCALE EXOSCALE 6 . 1
7. EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 7 . 1
8. I THOUGHT THIS WAS A DEV I THOUGHT THIS WAS A DEV CONFERENCE! CONFERENCE! 8 . 1
9. WHAT'S IN A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER Datacenter operations So ware development 9 . 1
10. SOFTWARE AT EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network controller Internal SDN Customer management Metering system Billing Web portal 10 . 1
11. LANGUAGES AT EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure Python ClojureScript & JS 11 . 1
12. THE BUILD FACTORY THE BUILD FACTORY From code to artifact From artifact to deploy 12 . 1
13. BUILD FACTORY BUILD FACTORY 13 . 1
14. MAIN DESIGN IDEA MAIN DESIGN IDEA Security Reproducibility Observability Checkpoints Cloud-Native 14 . 1
15. SECURITY SECURITY No code download on production hosts Signed packages 15 . 1
16. REPRODUCIBILITY REPRODUCIBILITY Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 16 . 1
17. OBSERVABILITY OBSERVABILITY When did we last build this? What did the output look like? What commit did it correspond to? 17 . 1
18. CHECKPOINTS CHECKPOINTS CD is great for test and staging We are wary of unattended production deploys There should be a clear (but simple) trigger 18 . 1
19. CLOUD-NATIVE CLOUD-NATIVE Configuration as code Fast infrastructure set-up and teardown No decay over time It should be easy to grow or shrink the factory's throughput 19 . 1
20. BUILD FACTORY BUILD FACTORY 20 . 1
21. BUILD FACTORY BUILD FACTORY 21 . 1
22. BUILD FACTORY BUILD FACTORY 22 . 1
23. BUILD FACTORY BUILD FACTORY 23 . 1
24. BUILD FACTORY BUILD FACTORY 24 . 1
25. BUILD FACTORY BUILD FACTORY 25 . 1
26. BUILD FACTORY BUILD FACTORY 26 . 1
27. BUILD FACTORY BUILD FACTORY 27 . 1
28. BUILD FACTORY BUILD FACTORY 28 . 1
29. ARE WE THERE YET? ARE WE THERE YET? 29 . 1
30. MOST OF IT IS FINE MOST OF IT IS FINE But legacy lurks in every environment. 30 . 1
31. THE CHECKLIST THE CHECKLIST Security ✔ Reproducibility ✔ Observability ✔ Checkpoints ✔ Cloud-Native ❌ 31 . 1
32. OUR JENKINS OUR JENKINS 32 . 1
33. OUR JENKINS OUR JENKINS A single machine, builds close to configuration Jenkins user has sudo access Which plugins are installed? How is config restored? 33 . 1
34. HOW DID WE GET HERE? HOW DID WE GET HERE? Jenkins makes automation hard Password in logs No clear configuration file standard 34 . 1
35. HOW DID WE GET HERE? HOW DID WE GET HERE? <?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description>week in review</description> <keepDependencies>false</keepDependencies> <properties> <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github@1.17.0"> <projectUrl>https://github.com/exoscale/wir/</projectUrl> </com.coravy.hudson.plugins.github.GithubProjectProperty> <com.tikal.hudson.plugins.notification.HudsonNotificationProperty plugin="notifi <endpoints> <com.tikal.hudson.plugins.notification.Endpoint> <format>JSON</format> <url>http://notifier:8080/hubot/jenkins-notify</url> <event>all</event> <timeout>30000</timeout> <loglines>0</loglines> </com.tikal.hudson.plugins.notification.Endpoint> </endpoints> </com.tikal.hudson.plugins.notification.HudsonNotificationProperty> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.model.StringParameterDefinition> <name>branch</name> ... 35 . 1
36. XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS <com.tikal.hudson.plugins.notification.HudsonNotificationProperty plugin="notificati <endpoints> <com.tikal.hudson.plugins.notification.Endpoint> ... </endpoints> </com.tikal.hudson.plugins.notification.HudsonNotificationProperty> 36 . 1
37. XML-SERIALIZED OBJECTS XML-SERIALIZED OBJECTS Lengthy descriptions No standard text to config Too many combinations to template 37 . 1
38. NO STANDARD HOOK MECHANISM NO STANDARD HOOK MECHANISM All job configuration is in-situ No way to externally say: run IRC notifications for all these jobs Even fewer avenues for templating 38 . 1
39. FEW MITIGATION STRATEGIES FEW MITIGATION STRATEGIES Jobs are edited through the web Backups to git as primary recovery mechanism 39 . 1
40. DIRECT CONSEQUENCE DIRECT CONSEQUENCE Jenkins down means no deploy Time to recovery: ??? 40 . 1
41. BE GENTLE BE GENTLE This was all started in 2012 41 . 1
42. JENKINS AUTOMATION, CIRCA 2012 JENKINS AUTOMATION, CIRCA 2012 42 . 1
43. JENKINS IN 2017: A NEW HOPE JENKINS IN 2017: A NEW HOPE Pipeline Jobs Job DSL Init via groovy 43 . 1
44. OUR IMPLEMENTATION OUR IMPLEMENTATION The road to a cloud-native CI environment 44 . 1
45. PIPELINE JOBS PIPELINE JOBS node { stage('build') { checkout scm sh 'lein compile :all' sh 'lein uberjar' archiveArtifacts artifacts: '**/target/*.jar', fingerprint: true } stage('test') { sh 'lein test' } } 45 . 1
46. PIPELINE JOBS PIPELINE JOBS . ├── Dockerfile ├── Jenkinsfile ├── project.clj ├── src │ └── foo │ └── core.clj └── test └── foo └── core_test.clj 4 directories, 5 files 46 . 1
47. PIPELINE JOBS PIPELINE JOBS Keeps build information close to code No impedance mismatch between job and code Easier to work in branches 47 . 1
48. JOB DSL JOB DSL def githubPipelineJob(org, name, private_repo=true) { pipelineJob name { definition { cpsScm { scriptPath 'JenkinsFile' git { remote { github "$org/$name" if (private_repo) { credential 'github-token' } } } } } } } githubPipelineJob('exoscale', 'portal') githubPipelineJob('exoscale', 'api') ... 48 . 1
49. JOB DSL JOB DSL A single job generates all jobs Updates / Removals when job list changes All Jenkins jobs can be expressed 49 . 1
50. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Most plugins require some form of manual intervention Most are operated from the master 50 . 1
51. REGISTERING BUILD HOSTS REGISTERING BUILD HOSTS Reverse engineered API Private SSH key deployed on the master As a named credential Public SSH key deployed on host 51 . 1
52. THE LAST MISSING PIECE THE LAST MISSING PIECE Still need to bootstrap Jenkins Still need to provision additional configuration Plugins Perms Credentials 52 . 1
53. GROOVY INITIALIZATION SUPPORT GROOVY INITIALIZATION SUPPORT Code in $JENKINS_HOME/init.groovy.d/ gets picked up All Jenkins functionality can be accessed 53 . 1
54. LIVING THE DREAM LIVING THE DREAM executors: 0 url: "http://jenkins/" seedjob: git: "https://github.com/exoscale/jobs.git" name: "seed-job" credentials: "github-token" credentials: ssh-key: type: PRIVATE_KEY username: ubuntu description: "Private key for host SSH access" key: | ... github-token: type: USER_PASSWORD username: build-bot password: "..." description: "Github autogenerated token credentials for builds" perms: admin: password: '...' allowed: [ 'JENKINS_ADMINISTER' ] builder: password: '...' allowed: - 'JENKINS_READ'
55. - 'ITEM_BUILD' - 'ITEM_READ' - ... worker: password: '...' allowed: - 'JENKINS_READ' - 'COMPUTER_EXTENDED_READ' - 'COMPUTER_CONNECT' - ... 54 . 1
56. LIVING THE DREAM LIVING THE DREAM ant build-timeout email-ext extended-read-permission github-branch-source gradle job-dsl workflow-aggregator pipeline-github-lib ssh-slaves timestamper ws-cleanup blueocean matrix-auth authorize-project 55 . 1
57. GROOVY INITIAL SETUP GROOVY INITIAL SETUP def realm = new HudsonPrivateSecurityRealm(false) def strategy = new FullControlOnceLoggedInAuthorizationStrategy() realm.createAccount('admin', 'TemporaryLongPassword') jenkins.setSecurityRealm(realm) jenkins.setAuthorizationStrategy(strategy) jenkins.getDescriptor("jenkins.CLI").get().setEnabled(false) jenkins.save() 56 . 1
58. GROOVY PLUGIN PROVISIONING GROOVY PLUGIN PROVISIONING def path = System.getenv("JENKINS_PLUGIN_CONFIG") ?: "/etc/jenkins/plugins.list" plugins = new File(path).readLines() def pm = jenkins.getPluginManager() def uc = jenkins.getUpdateCenter() def install_list = plugins.findAll{ !pm.getPlugin(it)} install_list.each { def plugin = uc.getPlugin(it) if (plugin) { def ftr = plugin.deploy() ftr.get(1, TimeUnit.MINUTES) } } jenkins.save() if (install_list) { jenkins.restart() } 57 . 1
59. GROOVY YAML LOAD GROOVY YAML LOAD @Grab('org.yaml:snakeyaml:1.18') def path = System.getenv("JENKINS_YAML_CONFIG") ?: "/etc/jenkins/jenkins.yaml" config = new Yaml().load(new File(path).getText()) 58 . 1
60. GROOVY JOB DSL BOOTSTRAP GROOVY JOB DSL BOOTSTRAP dslScript = new ExecuteDslScripts(null) dslScript.setTargets("jobs.groovy") dslScript.setLookupStrategy(LookupStrategy.JENKINS_ROOT) remotes = GitSCM.createRepoList(config.seedjob.git, config.seedjob.credentials) branches = [new BranchSpec("*/master")] browser = new GithubWeb(config.seedjob.git) dslScm = new GitSCM(remotes, branches, false, [], browser, "git", []) job = new hudson.model.FreeStyleProject(jenkins, config.seedjob.name) job.scm = dslScm job.getPublishersList().add(dslScript); jenkins.add(job, config.seedjob.name) 59 . 1
61. ARE WE THERE YET? ARE WE THERE YET? 60 . 1
62. MUCH CLOSER MUCH CLOSER 61 . 1
63. LOOKING FORWARD LOOKING FORWARD Learning a bit more Groovy :-) Less dependencies on build hosts More artifact repositories Easier interaction with hosts 62 . 1
64. THANKS! THANKS! Questions? We are hiring! Try Exoscale out! on twitter and github @pyr 63 . 1
65. 64 . 1