Share
Embed
Gothenburg Jenkins Meetup. Testing on Hardware with Jenkins Pipeline
This presentation is an overview of specific requirement for running tests on Hardware in Jenkins and of Jenkins Pipeline features available for this purpose. In particular, shared resource management and tool management is covered. The presentation also covers the speaker's Pipeline adopter experience.
Oleg Nenashev PRO
May 01, 2017
Transcript
70 slides
-
Running tests on hardware with Jenkins Pipeline Oleg Nenashev (@oleg_nenashev)
CloudBees, Inc. Days of Jenkins Göteborg, May 29, 2017 -
© 2017 CloudBees, Inc. All Rights Reserved. 2 #DayOfJenkins Göteborg
2017 About me @oleg_nenashev oleg-nenashev LibreCores project St. Petersburg Polytechnic University Jenkins meetups -
© 2017 CloudBees, Inc. All Rights Reserved. 3 #DayOfJenkins Göteborg
2017 Oleg’s “Hall of Shame”(c) • Jenkins Core • Remoting • Windows Service Wrapper • Plugins -
© 2017 CloudBees, Inc. All Rights Reserved. 5 #DayOfJenkins Göteborg
2017 ØJenkins Pipeline in Embedded • Managing workspaces • Managing shared resources • Tools Ø[TBD] - My Adopter experience Agenda -
© 2017 CloudBees, Inc. All Rights Reserved. 6 #DayOfJenkins Göteborg
2017 • Pipeline 101 • Pipeline Generic Features • Blue Ocean • Declarative Pipeline • Automation Infrastructure • Agents, Computing Grids, Virtualization • Added-value Pipeline features in CloudBees Jenkins Enterprise What is NOT in the talk? Robert’s talk Q&A Session Q&A -
© 2017 CloudBees, Inc. All Rights Reserved. 7 #DayOfJenkins Göteborg
2017 What is a typical Embedded Product? ` Hardware External IP Firmware / Embedded SW External SW Development tools / EDAs Demo software Demo hardware Developer kits “Main product” Documentation -
© 2017 CloudBees, Inc. All Rights Reserved. 8 #DayOfJenkins Göteborg
2017 Automotive… Source: http://www.popsautoelectric.com/auto-electric/ • System of systems • High level of integration • 3rd-party components • Many standards • High quality requirements -
© 2017 CloudBees, Inc. All Rights Reserved. 9 #DayOfJenkins Göteborg
2017 Embedded area. Welcome to the automation hell! • Continuous integration is must-have • Multiple technologies and tools • Legacy is everywhere • Unstable hardware & networks • … -
© 2017 CloudBees, Inc. All Rights Reserved. 10 #DayOfJenkins Göteborg
2017 1. The most popular CI/CD automation server in the world 2. It’s a framework => flexible 3. It’s open source => customizable 4. Commercial support is available 5. Useful plugins Why Jenkins? -
© 2017 CloudBees, Inc. All Rights Reserved. 11 #DayOfJenkins Göteborg
2017 Система Docs Plugins Configs Infra- structure Samples Demos How do we build CI Systems? Infra- structure Infra- structure • Agents (FKA ”slaves”) • Tools • Hardware peripherals • Computing grids -
© 2017 CloudBees, Inc. All Rights Reserved. 13 #DayOfJenkins Göteborg
2017 Where do build/tests run? Software Hardware Simulators, EDA tools FPGAs, ASIC prototypes HW Accelerators, etc. x -
© 2017 CloudBees, Inc. All Rights Reserved. 14 #DayOfJenkins Göteborg
2017 Tests are reproducible Easy to debug issues No need in expensive hardware (+) Single-shot VMs • No environment pollution, etc. Simulators. PROS “We run Hello World in two hours.” -
© 2017 CloudBees, Inc. All Rights Reserved. 15 #DayOfJenkins Göteborg
2017 Models are just… models • Not accurate for all cases • May contain bugs Long execution time • Slow spot-checks • Computing grids for massive runs Simulators. CONS “We run Hello World in two hours.” -
© 2017 CloudBees, Inc. All Rights Reserved. 16 #DayOfJenkins Göteborg
2017 And hardware… -
© 2017 CloudBees, Inc. All Rights Reserved. 17 #DayOfJenkins Göteborg
2017 Example. Jenkins Node with an Attached Board 17 VM or bare-metal Power Reset when it hangs Deploy bit-files, Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins agent Drivers Tools Peripherals Attached peripherals Runtime -
© 2017 CloudBees, Inc. All Rights Reserved. 18 #DayOfJenkins Göteborg
2017 Minimize amount of hardware tools Utilize hardware as much as possible Share hardware between projects • Also use it outside Jenkins... Reuse the existing hardware 18 Issue #1. Hardware is Expensive! M – x000$ L –x0000$ M - http://www.xilinx.com/univ/xupv5-lx110t.htm L - HAPS-64, www.synopsys.com -
© 2017 CloudBees, Inc. All Rights Reserved. 19 #DayOfJenkins Göteborg
2017 System monitoring • Discover hanging runs • Discover deadlocks Automatic recovery from Infra issues • Check if it is an infra issues • Rerun tests if possible • Common use-case: confirm HW failures with simulations Environment management • Environment cleanup • Workspace cleanup 19 Issue #2. Hardware is Unstable -
© 2017 CloudBees, Inc. All Rights Reserved. 21 #DayOfJenkins Göteborg
2017 What do we need? Resource management • Shared resources: hardware, licenses • Access management: Permissions, etc. Tools • Integration with tools • Shared tool infrastructures, preinstalled tools • Tool versioning Reusable solutions • Templates • Pipeline libraries -
© 2017 CloudBees, Inc. All Rights Reserved. 22 #DayOfJenkins Göteborg
2017 Approach: Using “classic” Job types: • Freestyle, Matrix, JobDSL, … Classic Approaches Keywords: • Script steps to run the logic • Custom Tools Plugin to manage tools • Throttle Concurrent Builds and Lockable Resources to dispatch access to hardware • Naginator Plugin to restart builds on-demand • Parameterized Trigger / Copy artifacts to parallelize tests and to collect results JUC2015/London https://jenkins.io/solutions/embedded/ -
© 2017 CloudBees, Inc. All Rights Reserved. 23 #DayOfJenkins Göteborg
2017 Classic Approaches Spaghetti Automation • Barely traceable dependencies • Complex build flow • Complex configurations • Duplication • Expensive maintenance -
© 2017 CloudBees, Inc. All Rights Reserved. 24 #DayOfJenkins Göteborg
2017 ü Store build definitions with the project ü Track changes together with the project ü Tests together with the project Configuration as Code -
© 2017 CloudBees, Inc. All Rights Reserved. 25 #DayOfJenkins Göteborg
2017 Configuration as Code in Jenkins Job Configuration Jenkins system configuration -
© 2017 CloudBees, Inc. All Rights Reserved. 26 #DayOfJenkins Göteborg
2017 TODO: Oil Pipelines Image Pipelines? -
© 2017 CloudBees, Inc. All Rights Reserved. 27 #DayOfJenkins Göteborg
2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description - defined in SCM (Jenkinsfile) - defined in Job configuration // Run on a node having the “my_fpga” label node("linux && ml509") { git url:"http://github.com/myorg/myproject.git" sh "make all" } http://bit.ly/pipeline-tutorial -
© 2017 CloudBees, Inc. All Rights Reserved. 28 #DayOfJenkins Göteborg
2017 Main features Configuration-as-Code Independent from node/workspace Parallelization Robustness -
© 2017 CloudBees, Inc. All Rights Reserved. 29 #DayOfJenkins Göteborg
2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description 2. Item type Pipeline Job Multi-Branch Pipeline Organization Folder • GitHub • BitBucket* • … -
© 2017 CloudBees, Inc. All Rights Reserved. 30 #DayOfJenkins Göteborg
2017 Jenkins Pipeline is a… 1. Groovy DSL for automatic flow description 2. Item type 3. Pipeline as Code Ecosystem -
© 2017 CloudBees, Inc. All Rights Reserved. 31 #DayOfJenkins Göteborg
2017 Branch- Source, BlueOcean, ... 89 - explicit support 42 - Pipeline- specific Documentation Samples Development Tools Jenkins Pipeline Ecosystem Plugins: -
© 2017 CloudBees, Inc. All Rights Reserved. 33 #DayOfJenkins Göteborg
2017 2016 http://bit.ly/nenashev-automotive-day-2016 -
© 2017 CloudBees, Inc. All Rights Reserved. 34 #DayOfJenkins Göteborg
2017 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016 -
© 2017 CloudBees, Inc. All Rights Reserved. 35 #DayOfJenkins Göteborg
2017 • Management of resources. • Integration with Throttle Concurrent Builds plugin • Use-case: limiting the license utilization in automation infrastructures (JENKINS-31801) • Better support of tools. • Common build flow – dozens of tools • Integration with Custom Tools Plugin seems to be the best option, especially in the case of multiple tool versions (JENKINS-30680) • Pipeline package manager (JENKINS-34186) • Lack of plugins for EDA tools in Jenkins, code duplications • A common approach is to keep the shared "functions" in libraries • Pipeline Global Library and Pipeline Remote Loader can be used 2016 – Pipeline improvement areas From my talk at CloudBees | Jenkins Automotive and Embedded Day 2016 -
© 2017 CloudBees, Inc. All Rights Reserved. 36 #DayOfJenkins Göteborg
2017 Resource Management • Lockable Resources Plugin • Since 2016 https://plugins.jenkins.io/lockable-resources -
© 2017 CloudBees, Inc. All Rights Reserved. 37 #DayOfJenkins Göteborg
2017 Resource Management • Throttle Concurrent Builds • Since 2.0, May 2017 • Supports only categories • Throttle inside run • Throttle across many runs https://plugins.jenkins.io/throttle-concurrents -
© 2017 CloudBees, Inc. All Rights Reserved. 38 #DayOfJenkins Göteborg
2017 Throttle parallel tasks inside Pipeline -
© 2017 CloudBees, Inc. All Rights Reserved. 39 #DayOfJenkins Göteborg
2017 Management of resources • Lockable Resources • Throttle Concurrent Builds • Pipeline Milestones – milestone() step • Priority Sorter Plugin • Job Fan-In plugin • … http://bit.ly/pipeline-compatibility -
© 2017 CloudBees, Inc. All Rights Reserved. 40 #DayOfJenkins Göteborg
2017 Tools Support • withEnv() for environment variables • tool() step in Pipeline was improved • Tool reference in the Job • Section in Declarative Pipeline -
© 2017 CloudBees, Inc. All Rights Reserved. 41 #DayOfJenkins Göteborg
2017 Tools Support • EnvInject Plugin – partially integrated • Only basic use-cases • No ETA • Custom Tools Plugin – partially integrated • Follow JENKINS-30680 • ETA – 2017 -
© 2017 CloudBees, Inc. All Rights Reserved. 42 #DayOfJenkins Göteborg
2017 Pipeline Libs • Early 2016 • Global Pipeline Library • Pipeline Remote Loader Plugin • Late 2016 • Pipeline Shared Libraries • Support of @Grab – Pipeline libs can pull common Groovy libs Library Structure: -
© 2017 CloudBees, Inc. All Rights Reserved. 43 #DayOfJenkins Göteborg
2017 /me as a Pipeline Adopter -
© 2017 CloudBees, Inc. All Rights Reserved. 44 #DayOfJenkins Göteborg
2017 Pipeline Early adoption It is working! * Image: http://devopsreactions.tumblr.com/post/151281670953/trying-the-first-prototype-good-enough -
Lessons Learnt. Obstacles? • Missing integrations • Pipeline DSL !=
Groovy • Error handling • Development Tools -
© 2017 CloudBees, Inc. All Rights Reserved. 46 #DayOfJenkins Göteborg
2017 Missing integration ØTools/plugins •http://bit.ly/pipeline-compatibility •Trusted pipeline libraries should help much ØBuild Promotion (Promoted Builds, Release, …) -
© 2017 CloudBees, Inc. All Rights Reserved. 47 #DayOfJenkins Göteborg
2017 For a reason… Pipeline DSL is not Groovy -
© 2017 CloudBees, Inc. All Rights Reserved. 48 #DayOfJenkins Göteborg
2017 Reason 1. Context retention – CPS/NonCPS • No way to propagate any data • Solution - @NonCPS steps within the library steps -
© 2017 CloudBees, Inc. All Rights Reserved. 49 #DayOfJenkins Göteborg
2017 Reason 2. Script Security Script approval Groovy Sandbox Mode • Mandatory for Jenkinsfile from SCM • Mandatory for untrusted libs Limited access to Java API • Script approvals by admins • Debug fun “Run & Fail & Approve & Rerun” -
Lessons Learnt. Obstacles? • Missing integrations • Pipeline DSL !=
Groovy • Error handling • Development Tools -
© 2017 CloudBees, Inc. All Rights Reserved. 51 #DayOfJenkins Göteborg
2017 Error Handling • IMHO: Pipeline is designed for Success Path scenarios When it “Just works” When it does not -
© 2017 CloudBees, Inc. All Rights Reserved. 52 #DayOfJenkins Göteborg
2017 Error Handling. Top-level Scripted Pipeline Declarative Pipeline • Global try/catch() When it does not… -
© 2017 CloudBees, Inc. All Rights Reserved. 53 #DayOfJenkins Göteborg
2017 Error Handling When it “Just works” When it does not The interesting part… Failover, repeat -
© 2017 CloudBees, Inc. All Rights Reserved. 54 #DayOfJenkins Göteborg
2017 Error Handling. Hardware failover • Pipeline allows re-running unreliable stages on other nodes. System failure (script returns 253) Node #1 Node #2 . . . . . . for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 253")) { // Fatal error fail("Test run failed") } } } Try all possible boards: -
© 2017 CloudBees, Inc. All Rights Reserved. 55 #DayOfJenkins Göteborg
2017 Error handling. Hardware failover for (def board : boards) { echo "trying board " + board; try { node(board) { checkout scm sh ‘./bin/run.sh’ // Call passed => DONE break; } } catch (Exception ex) { if (ex.message.contains ("exit code 255")) { // Fatal error fail("Test run failed") } } } ??? • Hard to extract error cause • Special handling is required for every case • Cannot just use existing parsers • Build Failure Analyzer • Log Parser • Warnings • Try-catch blocks pollute the code • Hard to test -
© 2017 CloudBees, Inc. All Rights Reserved. 56 #DayOfJenkins Göteborg
2017 Error handling. Hardware failover Declarative Pipeline Encapsulation via Libraries Simple use-case Other • Exception try/catch • timeout() wrapper • Proper failover • Logic in @NonCPS -
Lessons Learnt. Obstacles? • Missing integrations • Pipeline DSL !=
Groovy • Error handling • Development Tools -
© 2017 CloudBees, Inc. All Rights Reserved. 58 #DayOfJenkins Göteborg
2017 Pipeline. Dev Tools IDE Integration Unit Test Framework Static Analysis Library manager Debug tools Documen- tation Easy script deployment Freestyle => Pipeline Converter Good Not enough Missing -
© 2017 CloudBees, Inc. All Rights Reserved. 59 #DayOfJenkins Göteborg
2017 How do I work? Intellij IDEA Filesystem SCM Plugin (with patch) Disclaimer: There may be better ways • Syntax file • Built-in documentation Jenkins Test Instance Source Code (local .git repos) • Production System Snapshot • Custom Pipeline Library Settings • Repositories for libraries • Jenkinsfile repos -
© 2017 CloudBees, Inc. All Rights Reserved. 60 #DayOfJenkins Göteborg
2017 Getting the Syntax File -
© 2017 CloudBees, Inc. All Rights Reserved. 61 #DayOfJenkins Göteborg
2017 Filesystem SCM Plugin • Pending pull request: • https://github.com/jenkinsci/filesystem_scm-plugin/pull/2 Library definition in the test instance !!! -
© 2017 CloudBees, Inc. All Rights Reserved. 62 #DayOfJenkins Göteborg
2017 Pipeline Testing • No standard test framework offered by the Jenkins project • External project: https://github.com/lesfurets/JenkinsPipelineUnit “mvn verify” has been invoked… And the execution passed… For the source Jenkinsfile… -
Lessons learnt • No area-specific issues so far • High
barrier to entry • It worth investment -
© 2017 CloudBees, Inc. All Rights Reserved. 64 #DayOfJenkins Göteborg
2017 What I need? 2017 • Development Tools for Pipeline • IDE, Debugger, Test Tools • Community Pipeline Libraries • Common Embedded cases without plugins • Promotion engine for Pipeline • Ideally: Merge of standard Promoted Builds and Release plugins • Plugins for area-specific tools -
© 2017 CloudBees, Inc. All Rights Reserved. 65 #DayOfJenkins Göteborg
2017 Takeaways • Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances • Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases • Barrier to entry is still high • Just try it… -
© 2017 CloudBees, Inc. All Rights Reserved. 66 #DayOfJenkins Göteborg
2017 Thank you! Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev -
© 2017 CloudBees, Inc. All Rights Reserved. 67 #DayOfJenkins Göteborg
2017 More info • Embedded solutions page: • https://jenkins.io/solutions/embedded/ • Links to the articles and videos • Some categorization • To be updated soon… Embedded Solutions Page … -
© 2017 CloudBees, Inc. All Rights Reserved. 68 #DayOfJenkins Göteborg
2017 • Jenkins project • Website: https://jenkins.io • Meetup: https://www.meetup.com/Jenkins-online-meetup/ • Embedded Solutions page: https://jenkins.io/solutions/embedded/ • Jenkins Pipeline • Tutorial: http://bit.ly/pipeline-tutorial • Compatibility: http://bit.ly/pipeline-compatibility • Examples: https://github.com/jenkinsci/pipeline-examples • CloudBees • Community portal: https://go.cloudbees.com/ Useful Links -
© 2017 CloudBees, Inc. All Rights Reserved. 69 #DayOfJenkins Göteborg
2017 Thank you! Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev