Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Swiss JAM: State of Jenkins & Configuration as ...

Swiss JAM: State of Jenkins & Configuration as Code

A talk about recent changes and ongoing projects in the Jenkins community: "Jenkins: Shifting Gears" blogpost, Configuration-as-code plugin, Jenkins Evergreen, Jenkins X, and Cloud Native Jenkins.

There will be also a hands-on demo of the Configuration as Code plugin and Docker packaging.: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/tree/casc-plugin

Oleg Nenashev

September 11, 2018
Tweet

More Decks by Oleg Nenashev

Other Decks in Programming

Transcript

  1. > whoami @oleg_nenashev oleg-nenashev • Based in Neuchatel • Jenkins

    Architecture Team • CloudBees Core • CloudBees Jenkins Support 2
  2. > whoami -jenkins Hudson/Jenkins user since 2008 Jenkins contributor since

    2012 Now: • Maintainer: Core & plugins • Security Team member • Meetup organizer • Google Summer of Code org admin • Cloud Native SIG participant @oleg_nenashev oleg-nenashev 3
  3. • Building local community • Sharing experiences • Looking for

    contributors Swiss Jenkins Area Meetup 4 https://www.meetup.com/Swiss-Jenkins-Area-Meetup https://twitter.com/JenkinsMeetupCH
  4. Jenkins Area Meetups • 88 meetup groups • ~26000 participants

    • https://www.meetu p.com/pro/jenkins/ + Online Meetup https://jenkins.io/projects/jam/ 5
  5. 10 Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears”

    https://jenkins.io/blog/2018/08/31/shifting-gears/ “Jenkins: Shifting Gears” slides by Kohsuke https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view
  6. • Extensible • General Purpose • Community Shifting Gears: Our

    Amazing Success https://flic.kr/p/5FawSa 11 © 2018 CloudBees, Inc. All Rights Reserved
  7. 1. Most popular CI/CD tool in the world 2. It’s

    open source 3. Big community 4. Plugin ecosystem 5. … Who is Mr. Jenkins? https://jenkins.io 12
  8. • Service Instability • Brittle Configuration • Assembly Required •

    Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ 14 © 2018 CloudBees, Inc. All Rights Reserved
  9. • Service Instability • Brittle Configuration • Assembly Required •

    Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ Oleg: Jenkins is an automation framework, needs maintenance. Users need a solution 15 © 2018 CloudBees, Inc. All Rights Reserved
  10. • Service Instability • Brittle Configuration • Assembly Required •

    Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ Oleg: Developer Experiences == Contributor Experience. We need it 17 © 2018 CloudBees, Inc. All Rights Reserved
  11. • General purpose CI/CD engine that runs on Kubernetes that

    embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Shifting Gears: Cloud Native Jenkins 19 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
  12. • With all the good things of Jenkins we’ve been

    pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG • Follow the footsteps of Jenkins X and converge with it Shifting Gears: Cloud Native Jenkins 20 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
  13. • Groups focusing on particular areas • Regular public meetings

    • Public chats and mailing lists • Venue for Jenkins Enhancement Proposal discussions • Examples: • Platform SIG, GSoC SIG, Chinese Localization • Cloud Native SIG Special Interest Groups 22 https://jenkins.io/sigs/
  14. • https://jenkins.io/sigs/cloud-native/ • Focuses on making Jenkins cloud-native • Lead:

    Carlos Sanchez (Kubernetes Plugin maintainer) • Participants: • Individual Contributors • CloudBees, Google Cloud Platform, Microsoft Azure • My blog about the SIG: • https://jenkins.io/blog/2018/07/30/introducing-cloud-native-sig/ Cloud Native SIG 23
  15. • Closer than you might think, because we have many

    of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Shifting Gears: Cloud Native Jenkins MVP 24 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
  16. • Continue the incremental path of Jenkins 2, BUT with

    a jolt • We need to get out of the local optimum • Better speed of development, better stability • Renegotiate “contract” with users • Move away from “forever compatibility” • Release model change, ala Java SE • Continuous delivery of Jenkins • Must make sense to users • be largely compatible, but can’t afford to be perfectly compatible Shifting Gears: Jolt in Jenkins https://flic.kr/p/65YARD 25 © 2018 CloudBees, Inc. All Rights Reserved
  17. • Jenkins project already does incompatible changes • Security fixes

    and hardening • E.g. JEP-200: https://jenkins.io/blog/2018/03/15/jep-200-lts/ • Incompatible changes are fine if done properly • Now: upgrade guidelines and announcement blogposts • https://jenkins.io/doc/upgrade-guide/ Incompatible changes? FEAR NOT! 26
  18. • Continuous Delivery infrastructure and tooling • API Deprecation policy

    & Engine • Feature flags (early adoption, A/B testing) • Tools to warn Jenkins users about incoming changes • Tools to warn plugin maintainers about changes • Telemetry API for better usage stats Jolt in Jenkins. What do we need? (IMHO) 27 // See Jenkins Evergreen below for CD infra and telemetry
  19. • Evergreen • Configuration as Code • Continued evolution of

    Pipeline execution engine • Developer experience / onboarding improvements • Plugin spring cleaning • Table stakes service integration Shifting Gears: Viable Ideas CloudBees is backing up https://jenkins.io/blog/2018/08/31/shifting-gears/ 28 © 2018 CloudBees, Inc. All Rights Reserved
  20. • We have to move away from local optimum, we

    need to change • This time, money is where the mouth is • Two initiatives, in broad stroke • Lots of details to be filled out • Will happen in SIGs and JEPs, in coming days • Looking forward for others to join Shifting Gears: Conclusions https://jenkins.io/blog/2018/08/31/shifting-gears/ 29 © 2018 CloudBees, Inc. All Rights Reserved
  21. 34

  22. • Pluggable Storage • Cloud Native Jenkins itself • Replicated

    services, High Availability, zero downtime • Canary and rolling deployments • Integrations with Cloud Technologies, esp. Kubernetes • … • Configuration as Code Cloud Native SIG. Areas of interest 36
  23. Pluggable Storage for Jenkins • Discussions started at Jenkins World

    2016 • Main data types: • Artifacts • Logs • Configurations • Runs: metadata, test results, etc. • Fingerprints • … 37
  24. Artifacts • JEP-202 – External Artifact Storage • New API

    in the Jenkins core and Pipeline • Implementation: Artifact Manager for S3 • https://github.com/jenkinsci/artifact-manager-s3-plugin • Pipeline integration • Transparent support of archive/unarchive • Transparent support of stash/unstash 39
  25. Logs in Jenkins 40 Build logs System logs Other •

    Agent logs • SCM polling • Multibranch indexing • … java.util.logging
  26. Master Current state Agents Logs Build Console • All logs

    got through the master • Agents send logs to the master • Build console… comes from the disk as well 41
  27. • AWS CloudWatch Logs: https://github.com/jenkinsci/pipeline-log-fluentd- cloudwatch-plugin • Elasticsearch: https://github.com/jenkinsci/external- logging-elasticsearch-plugin

    • Old PoC: https://github.com/jenkinsci/custom-war- packager/tree/master/demo/external-logging- elasticsearch External Logs. Prototypes 43
  28. Configurations • Configurations are stored differently L • hudson.XmlFile covers

    95% of cases • Prior work: • API: https://github.com/jenkinsci/jenkins/pull/3393 • K8s resources: https://github.com/jstrachan/jenkins/tree/kubeify • JEP-213 – Current design (under review) • https://github.com/jenkinsci/jep/tree/master/jep/213 44
  29. Jenkins X. Value • Works out-of-the box • CLI for

    everything • Support of environments • Configuration management via GitOps • Extensible via Kubernetes 49
  30. Jenkins X. Quickstart • > brew install jx • >

    jx install • > jx create quickstart • (or jx create spring, jx import) • Creates a Git repo • Registers web hooks • Creates images for Kubernetes • Runs the first Pipeline 50
  31. Jenkins X. OOTB experience • Jenkins-master • Agents pool (Kubernetes

    plugin) • Nexus, chartmuseum, monocular • Environments: ▸ Local for Development ▸ Staging ▸ Integration with Production envs • IDE Integration 51
  32. 52

  33. More info about Jenkins X • Sep 12/13 – Workshops

    in Zurich • Cosmin Cojocar, Jenkins X: Continuous Delivery for Java services in Kubernetes ▸ https://workshoptage.ch/workshops/2018/jenkins-x-continuous-delivery-for-java- services-in-kubernetes/ • Paolo Carta, Matteo Baiguini. End-to-End Continuous Delivery on Kubernetes ▸ https://workshoptage.ch/workshops/2018/end-to-end-continuous-delivery-on- kubernetes/ • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes ▸ https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ 54
  34. • Automated Sane Defaults • Cloud Native: AWS, Docker •

    Automatically Upgrading Distribution • And rollback… • Connected: Telemetry & Feedback • Obvious Path To User Success • Opinionated • Clear out obstacles • Documentation Jenkins Evergreen basics 57
  35. 58

  36. Jenkins Evergreen. The four opens • Open Source • Open

    Design • Open Development • Open Community https://github.com/jenkins-infra/evergreen/#the-four-opens 59
  37. Jenkins Evergreen Architecture Evergreen Instance evergreen- client registration authentication update

    manifest update level = 1. Client downloads WAR, plugins, client update, etc. installs, triggers restart then health-checks Backend Services evergreen.jenkins.io 60
  38. CD for the core and plugins • Incrementals infrastructure (JEP-305)

    • Master, branches and pull requests go to Artifactory https://jenkins.io/blog/2018/05/15/incremental-deployment/ 61
  39. Incrementals - usage • Not available in the standard Update

    Center • Can be downloaded from Artifactory • Support in official Docker images (jenkins/jenkins) # plugins.txt workflow-aggregator:2.5 workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.19-rc289.d09828a05a74 configuration-as-code:experimental 62
  40. CI patches: essentialsTest() • Work-in-progress • New engine for plugin

    CI/CD in ci.jenkins.io • Pipeline Library • https://github.com/jenkins-infra/pipeline- library/blob/master/vars/essentialsTest.groovy • Integration tests during the build • Plugin Compat Tester • Acceptance Test Harness 63
  41. 65

  42. “Infrastructure as Code” in Jenkins API Clients Jenkins CLI and

    REST API python- jenkins jenkins- client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Jenkins- internal solutions Groovy Boot Hooks Scriptler Plugin JCasC Plugin 68
  43. State in 2016 – Stockholm JAM • No default solution

    recommended by the Jenkins Project • No active work on this front • System Config DSL, was abandoned • Configuration Management – outside Jenkins Project • Time lag in Features • Security fun • Compatibility (e.g. Jenkins 2) 69
  44. 71

  45. Groovy Hooks [1/2] • Standard Groovy engine • Triggered by

    Jenkins • Direct access to the runtime • Extensible hook engine • init, boot-failure – in the core https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 72
  46. Groovy Hooks [2/2] • Script locations: • WEB-INF/HOOK.groovy in jenkins.war

    • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy • Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 73
  47. Groovy Hook Limitations. General • Jenkins API is scattered and

    complex • Javadoc is not user-friendly • Public API is not enough sometimes • No export from UI Solution: • Configuration as Code Plugin 78
  48. Groovy Hook Limitations. Missing Features • No class support •

    No shared libraries support • Error propagation: Jenkins does not stop on script error • Restart is required to apply changes There are workarounds 79
  49. • Feb 2018 – first alpha • Aug 2018 –

    release candidates • Configuration via YAML • jenkinsci/configuration-as-code-plugin New plugin: Jenkins Configuration-as-Code 81
  50. jenkins: systemMessage: "JCasC Demo" numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL

    securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml 82
  51. • YAML specification • Configuration export to YAML • CLI

    and REST API for configuration management • Pre-flight validations and dry-run • Support of many plugins out-of-the-box Features 83
  52. Limitations (temporary?) • Not all plugins are supported • Forward/backward

    compatibility is not guaranteed • YAML - No dynamic logic / scripting • Startup Chicken&Egg for plugins • Race conditions for jobs • Handling of removed sections • Export from UI – many open issues 84
  53. Initialization with JCasC JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC

    PLUGINS_STARTED . . . • Runs after the plugins get loaded • Runs in parallel with job loading • Can trigger restart on-demand 85
  54. Comparison JCasC Plugin Groovy Hooks YAML – standard declarative definition

    Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for config management Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support 86
  55. Combining JCasC and Groovy hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks:

    init() JCasC PLUGINS_STARTED . . . • Groovy hooks run after JCasC • Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations • Use JCasC where possible 88
  56. Demo. Local Pipeline Development Env Intellij IDEA Filesystem SCM Plugin

    • Documentation, Syntax • Static analysis • Debug (only for hooks) Local Jenkins instance Source Code (local .git repos) • Configuration-as-Code • Same as production • Repos with libs • Jenkinsfile JENKINS_HOME (volume) 90
  57. JCasC. Takeaways • Select an engine depending on your needs

    and expertise • New to Jenkins? Use JCasC • Combine engines if needed 91
  58. What else? • Java 10+ Support • BlueOcean Extensibility •

    JEP-204 – BlueOcean Extensibility API • JEP-203 – BlueOcean Extensibility 93
  59. What else? • Java 10+ Support • BlueOcean Extensibility •

    Google Summer of Code • Code Coverage API Plugin • Remoting over Apache Kafka • Simple Pull Request Job (aka Pipeline as YAML) https://jenkins.io/projects/gsoc/ 94
  60. • Jenkins evolves. Key issues in the architecture and community

    are being addressed • There are new projects to try Takeaways != 95
  61. Takeaways • It’s a great time to contribute ;) •

    https://jenkins.io/participate 96
  62. 1. Join special interest groups (SIGs) 2. Let us know

    about your pain-points 3. Share your stories and experiences • Jenkins.io/blog and other blogs • Meetups Contribution != Code 97 Jenkins needs you!
  63. Links Shifting Gears: • https://jenkins.io/blog/2018/08/31/shifting-gears/ Projects: • https://jenkins.io/sigs/cloud-native/ • https://jenkins-x.io/

    • https://github.com/jenkins-infra/evergreen Configuration-as-Code Plugin • https://github.com/jenkinsci/configuration-as-code-plugin 98
  64. Jenkins Events. What’s next? • Sep 16-19 - DevOps World

    | Jenkins World US • https://www.cloudbees.com/devops-world/san-francisco • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes • https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ • Oct 19 – Day of Jenkins [as code], Copenhagen • Oct 22-25 – DevOps World | Jenkins World Nice • https://www.cloudbees.com/devops-world/nice • Late Oct – Lausanne or Neuchatel? 99 DevOps World | Jenkins World 20% Discount Code JWJAMMEMBER