Under the hood of serverless Jenkins. Jenkinsfile Runner

There were significant changes in the Jenkins project over last year. It is still one of the most popular automation servers in the world, but it is not just a “server” anymore. In 2019 Jenkins is a universal automation engine, which actively evolves towards the new Cloud Native architecture and which can be used in any CI/CD flows. It can be used not only as a server, but also as a “serverless” single-shot execution. And it is not only about Jenkins X, but also about more classic usages.

In my talk I will present Jenkinsfile Runner - an engine which allows running Jenkins Pipelines in a single-shot container. I will show how it is used in Jenkins X, GitHub Actions and other services. And, finally, I will show how to package your own Jenkinsfile Runner images and run it in any environments, CI systems and Cloud providers.

NOTE: This slidedeck will be periodically updated to reflect the latest version of the talk

Transcript

1. © 2019 CloudBees, Inc. All Rights Reserved. Under the hood of serverless Jenkins. Jenkinsfile Runner Oleg Nenashev, CloudBees @oleg_nenashev May 14, 2019 1
2. © 2019 CloudBees, Inc. All Rights Reserved. TL;DR “You can run Jenkins Pipelines as single-shot container in any environment” This talk: How to do it? 2 These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr
3. © 2019 CloudBees, Inc. All Rights Reserved. > whoami 3 @oleg_nenashev oleg-nenashev • Principal SW Engineer, CloudBees • Jenkins maintainer & ambassador • Swiss Jenkins Meetup organizer
4. © 2019 CloudBees, Inc. All Rights Reserved. About you 4 These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr
5. © 2019 CloudBees, Inc. All Rights Reserved. 1. It’s a framework => flexible 2. It’s open source => customizable 3. More than 1600 plugins 4. Big community 5. Commercial support is available Who is Mr. Jenkins?
6. © 2019 CloudBees, Inc. All Rights Reserved. 6 Jenkins in 2019 Jenkins Pipeline Config-as-Code Plugin Modern plugins Modern packaging Jenkins X
7. © 2019 CloudBees, Inc. All Rights Reserved. 7 Permanent agents On-demand agents from cloud providers Jenkins Master Web UI
8. © 2019 CloudBees, Inc. All Rights Reserved. Master as a SPoF Macroservice 8 Permanent agents On-demand agents from cloud providers Web UI Jenkins Master
9. © 2018 CloudBees, Inc. All Rights Reserved. 9 (┛ಠ_ಠ)┛彡┻━┻ What if Jenkins master was “on-demand”? 9
10. © 2018 CloudBees, Inc. All Rights Reserved. Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 10
11. © 2018 CloudBees, Inc. All Rights Reserved. Cloud Native Jenkins • Best service for each need • Pay per use • “Infinite” scaling • Easy to use • Fast to develop https://flic.kr/p/75ncBF 11
12. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner https://github.com/jenkinsci/jenkinsfile-runner Started by Kohsuke in Mar 2018 3 maintainers, 19 contributors Status: Beta (1.0-beta-8) 12
13. © 2019 CloudBees, Inc. All Rights Reserved. JFR TL;DR • Binary and base Docker image • Runs Pipeline and shuts down • Prints log output to STDOUT • Workspace mapping as a volume • No Web UI included https://github.com/jenkinsci/jenkinsfile-runner 13
14. © 2019 CloudBees, Inc. All Rights Reserved. Usage • Pass workspace with project/Jenkinsfile • Pass cache for local data (if needed) docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner 14
15. © 2019 CloudBees, Inc. All Rights Reserved. DEMO https://github.com/jenkinsci/jenkinsfile-runner/tree/master/demo/cwp use the 1.0-beta-8 tag 15
16. © 2019 CloudBees, Inc. All Rights Reserved. Demo. Our Jenkinsfile 16 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner
17. © 2019 CloudBees, Inc. All Rights Reserved. 17
18. © 2019 CloudBees, Inc. All Rights Reserved. Features Startup in few seconds • Performance tweaks Advanced options • Script sandbox can be disabled (--ns) • Pipeline parameters support Java 11 support ◦ Better garbage collector + optimization ◦ cgroups support => containers resource limits 18
19. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors 19 Vanilla: Docker/Binary in the official repo • https://github.com/jenkinsci/jenkinsfile-runner • https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner
20. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors Vanilla: Docker/Binary in the official repo • https://github.com/jenkinsci/jenkinsfile-runner • https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner Custom JFR packages • https://jenkins.io/blog/2018/10/16/custom-war-packager/ • Ready-to-fly Docker images with plugins and configs • Way to go in 95% of cases 20
21. © 2019 CloudBees, Inc. All Rights Reserved. Building custom Jenkinsfile Runner images Custom WAR Packager Jenkinsfile Runner Core and Plugins Configuration-as-code Packaging Ready-to-fly Docker image Base image https://jenkins.io/blog/2018/10/16/custom-war-packager/ 21
22. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager and Self-configuration 1. Custom WAR Packager is managed by a config YAML 2. Docker image: plugins.txt, Java flags, etc. 3. Groovy Init Scripts 4. AND: New Configuration-as-Code Plugin https://plugins.jenkins.io/configuration-as-code Allows configuring Jenkins from YAML 22 https://github.com/jenkinsci/custom-war-packager
23. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager Demos • https://github.com/jenkinsci/custom-war-packager#demo • https://github.com/jenkinsci/ci.jenkins.io-runner/ 23
24. © 2019 CloudBees, Inc. All Rights Reserved. Just building the stuff is not enough 24
25. © 2019 CloudBees, Inc. All Rights Reserved. 25
26. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework https://github.com/jenkinsci/jenkinsfile-runner-test-framework • Custom WAR Packager • Docker • shUnit2 26
27. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework Jenkinsfile Runner packages Jenkinsfiles Pipeline libraries 27 Tests
28. © 2019 CloudBees, Inc. All Rights Reserved. Examples Jenkins X, GitHub Actions, CloudBees CodeShip 28
29. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner. Usage examples Jenkins X, Serverless mode (before Jenkins X 2.0 with Tekton) • https://github.com/jenkins-x/jenkins-x-serverless ci.jenkins.io-runner • https://github.com/jenkinsci/ci.jenkins.io-runner “FaaS” packaging • GitHub Actions: https://github.com/jonico/jenkinsfile-runner-github-actions • Codeship: https://github.com/oleg-nenashev/codeship-jenkinsfile • AWS Lambda: https://github.com/carlossg/jenkinsfile-runner-lambda • Project Fn: https://github.com/carlossg/jenkinsfile-runner-fn 29
30. © 2019 CloudBees, Inc. All Rights Reserved. 30 https://jenkins-x.io
31. © 2019 CloudBees, Inc. All Rights Reserved. 31
32. © 2019 CloudBees, Inc. All Rights Reserved. Details about Jenkins X Paolo Carta, Matteo Baiguini, Continuous Deployment of Polyglot Microservices with Istio • https://devopsdays.org/events/2019-zurich/program/workshop-with- matteo-baiguini-and-paolo-carta/ Cosmin Cojocar, Jenkins X contributor • CloudBees Booth on Wednesday • Both Jenkinsfile runner and Tekton flows 32
33. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins X. Modes and engines 33 Static Masters Serverless build engines Jenkinsfile Runner Tekton (default in 2.0) Classic Jenkins
34. © 2018 CloudBees, Inc. All Rights Reserved. Jenkins X. Serverless Mode (before April 2019) 34 https://medium.com/@jdrawlings/ser verless-jenkins-with-jenkins-x-9134cb fe6870 Serverless Jenkins with Jenkins X
35. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins X Serverless Build flow Single-shot masters build flow 35
36. © 2019 CloudBees, Inc. All Rights Reserved. https://medium.com/@jdrawlings/serverless-jenki ns-with-jenkins-x-9134cbfe6870 Jenkinsfile Runner is Here 36 Jenkinsfile Runner
37. © 2018 CloudBees, Inc. All Rights Reserved. CommentOps - communication with users 37
38. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions … and Jenkinsfile Runner 38
39. © 2019 CloudBees, Inc. All Rights Reserved. https://github.com/features/actions 39
40. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions! Not designed to be CI, useful for short tasks/workflows https://github.com/jonico/jenkinsfile-runner-github-actions 40
41. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions is in Limited Beta … and I am still waiting for my limited Beta access to be approved 41
42. © 2019 CloudBees, Inc. All Rights Reserved. And CodeShip? 42
43. © 2019 CloudBees, Inc. All Rights Reserved. CloudBees CodeShip • Jenkins has no good SaaS • CodeShip Pro - CI SaaS with native Docker support • Free tiers for open-source projects • What if we run Jenkins there? 43 https://codeship.com/features/pro
44. © 2019 CloudBees, Inc. All Rights Reserved. Demo 1 44 codeship-steps.yml codeship-services.yml Jenkinsfile https://github.com/oleg-nenashev/codeship-jenkinsfile
45. © 2019 CloudBees, Inc. All Rights Reserved. 45
46. © 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip https://github.com/oleg-nenashev/mailer-plugin/tree/codeship-demo • Using ci.jenkins.io-runner ◦ https://github.com/jenkinsci/ci.jenkins.io-runner • Using the standard Jenkins Pipeline library ◦ https://github.com/jenkins-infra/pipeline-library • Using standard Jenkinsfile ◦ buildPlugin() 46
47. © 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip codeship-services.yml codeship-steps.yml 47 Jenkinsfile
48. © 2019 CloudBees, Inc. All Rights Reserved. 48
49. © 2019 CloudBees, Inc. All Rights Reserved. CodeShip. Build history 49
50. © 2019 CloudBees, Inc. All Rights Reserved. So... 50
51. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner runs everywhere 51 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner • Docker images • CLI Tool - on Java 8 or Java 11
52. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner is not a silver bullet! 52
53. © 2019 CloudBees, Inc. All Rights Reserved. Limitations Jenkinsfile Runner is not fully compatible • No Web UI, no way to extract Jenkins reports • No native support of external storage • No build throttling, cross-master communication, etc. Single container, single agent • Docker and Kubernetes plugins cannot be easily used to produce sidecar containers • https://github.com/jenkinsci/remoting/pull/286 53
54. © 2019 CloudBees, Inc. All Rights Reserved. Future work Waiting for user feedback Integrating patches • https://github.com/jenkinsci/jenkinsfile-runner/blob/master/CONTRIBUTING.md (?) Jenkinsfile Runner 1.0 (?) Sidecar containers Experiments with GraalVM and Quarkus 54
55. © 2019 CloudBees, Inc. All Rights Reserved. Takeaways • serverless Jenkins is real • you can run your Pipelines everywhere • there are tools for that! • use Jenkins X if applicable (K8s/GitOps) 55
56. © 2019 CloudBees, Inc. All Rights Reserved. Demos and Questions CloudBees Booth Demos... • Jenkins • Jenkinsfile Runner • Jenkins X - Wed • CodeShip & Co • Config-as-Code • Pipeline devel 56 We are here
57. © 2019 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 57
58. © 2019 CloudBees, Inc. All Rights Reserved. > whoami -cloudbees • CloudBees Jenkins Distribution • CloudBees Jenkins Support • Community (Java 11 support, Jenkinsfile Runner, Security) https://www.cloudbees.com 58
59. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins as you may knew it… in 2012 Complex configuration Mega-masters, limited scalability Painful upgrades Masters are always running . . . 59