Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jenkins Contributor Summit 2018: Cloud Native J...

Jenkins Contributor Summit 2018: Cloud Native Jenkins update

Cloud Native Jenkins update from the Jenkins contributor summit at DevOps World | Jenkins World (Sep 17, 2018). Cloud Native SIG, Pluggable Storage, Ephemeral Jenkins, and future plans.

Oleg Nenashev

September 17, 2018
Tweet

More Decks by Oleg Nenashev

Other Decks in Programming

Transcript

  1. Cloud Native Jenkins update Jenkins Contributor Summit Sep 17, 2018

    Oleg Nenashev Jesse Glick Cloud Native SIG members 1
  2. Cloud Native SIG ? // No official logo so far

    • SIG = Special Interest Group • Founded in July 2018 • https://jenkins.io/sigs/cloud-native/ • Goal: Improve Jenkins to run on Cloud environments as a "Cloud Native" application 2
  3. Cloud Native Jenkins SIG. Members 36 participants, 3 company members

    https://jenkins.io/sigs/cloud-native/ 4 . . .
  4. Pluggable storage • Jenkins stores everything in $JENKINS_HOME ◦ Well-known

    scalability / performance bottleneck • Many plugins and workarounds • Limited support by Jenkins architecture ◦ Custom steps are required ◦ s3() or artifactory() • No hacks, we change the architecture 7
  5. We want Pluggable Storage to be... • TRANSPARENT • No

    configuration on user side • Standard steps work out-of-the-box ▸ archiveArtifacts 'giant.war' ▸ junit 'test-*.xml' ▸ ... 8
  6. We want Pluggable Storage to be... • TRANSPARENT • SCALABLE

    • Handle high parallel load • Upload large data straight from agent • Multi-master setups 9
  7. We want Pluggable Storage to be... • TRANSPARENT • SCALABLE

    • EXTENSIBLE • We provide API and reference implementations • We encourage contributors to create more implementations 10
  8. Pluggable Storage: current state & priorities • Artifacts • Build

    logs • Configuration • Credentials • Builds • Jobs • Test results Defined at the JW 2016 Contributor Summit • Code coverage • Static analysis • System logs • Task logs • Fingerprints • ... Legend done & available work in progress not started 11
  9. External Artifact Manager Status: Available • JEP-202 - “External artifact

    storage” • API patches in Jenkins 2.118+ Implementations • Artifact Manager for S3 • http://plugins.jenkins.io/artifact-manager-s3 • Available in Evergreen 12
  10. Credentials Storage Status: Available API - Credentials Plugin 1.15+ (>2

    years) • CredentialsProvider extension point • No API for hudson.Secret, no plans Implementations: • Kubernetes Credentials Provider 13
  11. External Configurations Status: Work In Progress • Idea: Make XML

    file (and other types) storage destinations pluggable • JEP-213 - Configuration Storage API in the core Implementations: • SQL - by Alex Nordlund (deepy) • K8s Resources (same Kubeify by James Strachan) config.xml 14
  12. Test Results Status: Work in progress • API: Jenkins JUnit

    plugin extensions • junit-plugin/pull/110 • Upload test results from agents • Details on-demand • Transparent for plugin features Implementations • SQL (e.g. Aurora Serverless) • Document storage (e.g. DynamoDB) • TMD (e.g. TestLink) 15
  13. Jenkins master Build console External Build Logging Jenkins agent Execution

    Run Log Storage CLI Standard Logging Systems Status: Work In Progress
  14. External Build Logging: Implementations 1. Compatibility: Filesystem-based storage 2. Elasticsearch

    - generic implementation • jenkinsci/external-logging-elasticsearch-plugin 3. AWS CloudWatch Logs (perhaps via Fluentd) • jenkinsci/pipeline-log-fluentd-cloudwatch-plugin 17 https://youtu.be/9lTOtC9wA_I
  15. Pluggable Storage Future stories • Code Coverage storage (based on

    Code Coverage API) • Static Analysis: FindBugs, Codestyle, etc. (based on Warnings Plugin 5.0) More info: https://jenkins.io/sigs/cloud-native/pluggable-storage/ 18
  16. 20

  17. Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/

    Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 21
  18. • General purpose CI/CD engine that runs on Kubernetes that

    embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view Already in the SIG 23
  19. • With all the good things of Jenkins we’ve been

    pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG, led by Carlos Sanchez • Follow the footsteps of Jenkins X and converge with it Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 24
  20. • Closer than you might think, because we have many

    of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Cloud Native Jenkins MVP // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 25
  21. Oleg, Carlos and Jesse created a prototype (blog) • Single-shot

    masters • Configuration as code • Jenkinsfile Runner • Custom WAR Packager Jenkins X 2.x uses the toolchain as well Presentation & demo: Tuesday, 2:30 PM Cloud Native Jenkins MVP. Prototype https://jenkins.io/blog/2018/09/12/speaker-b log-a-cloud-native-jenkins/ 26
  22. • All activities in Jenkins X and Jenkins Evergreen •

    Jenkinsfile Runner by Nicolas and Kohsuke • Cloud Native networking in Jenkins a. Remoting over Kafka project by Pham Vu Tuan b. (?) Kubernetes-native solutions Other ongoing activities 27
  23. 1. Define plans towards Cloud Native Jenkins 2. Continue Pluggable

    storage work a. External Log Storage - GA b. Test Results c. JEP-213 - Configuration Storage for Jenkins 3. More alignment with Jenkins X & Evergreen 29
  24. Cloud Native Jenkins SIG at the summit • This presentation

    • 1.00PM-3:00PM - Cloud Native SIG Meeting (BoF table) ◦ All - Discuss Jenkins: Shifting Gears & Cloud Native SIG ◦ Pham Vu Tuan - Remoting over Kafka presentation ◦ Other topics (unconference) • 3.30PM - Ignite Talks ◦ Propose your own topic! 30
  25. 31 What can you do? 1. Join the Cloud Native

    SIG to shape the future of Jenkins! 2. Try existing implementations and provide feedback 3. Create new External Storage Implementations https://jenkins.io/sigs/cloud-native/