Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced Code Coverage with Jenkins, GitHub and...

Advanced Code Coverage with Jenkins, GitHub and API Mocking

In 2015-2018, I talked about how to use the Jenkins Pipeline and custom libraries to do advanced integration tests and analyze code coverage. Coverage plugins were rather weak, and one needed some scripts and hacks to make it work, and to DIY for distributed testing. In 2021 the situation has changed significantly thanks to the Coverage and Checks API plugins. Distributed integration testing also became easier thanks to better coverage collectors and integrations with API mocking tools. So, good time to be alive… and use Jenkins!

We will talk about how modern Jenkins allows you to improve and analyze code coverage. We will talk about unit and integration testing with WireMock, the new Coverage Plugin, support for standard formats (Cobertura, JaCoCo, gcov, JUnit, etc.), parallelization for heavy integration tests and API mocking, and integration with GitHub Checks API. How can you analyze code coverage in Jenkins and when do you need to create your own libraries? And what’s the fuzz about Testcontainers and WireMock for integration testing?

Oleg Nenashev

January 21, 2024
Tweet

Resources

More Decks by Oleg Nenashev

Other Decks in Technology

Transcript

  1. / > whoami @oleg_nenashev oleg-nenashev Dr. Nenashev / Mr. Jenkins

    Developer tools hacker Community builder & DevRel consultant #RussiansAgainstPutin #StandWithUkraine
  2. / Agenda • Jenkins TL;DR • Code Coverage in Jenkins.

    2017 Edition • Modern Code Coverage in Jenkins, done right • Boosting coverage with Integration tests • GitHub Checks API in Jenkins • WireMock and Testcontainers These slides: speakerdeck.com/onenashev/advanced-code- coverage-with-jenkins-github-and-api-mocking
  3. / Disclaimer: Taking a break in Jenkins • On sabbatical

    in the community • Reasons: personal events, ongoing war • I participate in governance and GSoC • I will be back! CDF Statement We stand with the people of Ukraine. Please assist humanitarian efforts for the Ukrainian people and those affected by the military invasion of Ukraine by supporting international aid organizations, including UNICEF.
  4. / Система Docs Plugins Configs Infrastruc ture Backend Samples Demos

    CI/CD System from Jenkins Infrastruc ture Infrastruc ture
  5. / Система Docs Plugins Configs Infra- structure Backend Samples Demos

    Infra- structure Infra- structure ? without proper maintenance Any Automation Framework
  6. / Team A(utomation) Jenkins is a part of the CI

    system, Dev Productivity Team’s Best Friend (?proof)
  7. /

  8. / 1. Take library components 2. Build your own bicycle

    Pipeline Add some glue! Back to 2017… Using Pipeline libraries
  9. / Coverage Plugin • 2018 - Created as Code Coverage

    API Plugin during GSoC 2018 by Shenyu Zheng • jenkins.io/projects/gsoc/2018/code-coverage-api-plugin • 2020/2022 - Major UI/UX revamps • 2023 - replaced by the Coverage Plugin plugins.jenkins.io/coverage
  10. / Code Coverage API Plugin (EoL). Supported Adapters • JaCoCo

    • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
  11. / Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov

    (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
  12. / Usage • Records coverage produced by your build tool

    • Compatible with Jenkins Pipeline and Freestyle jobs
  13. / Coverage Plugin. Features • Coverage charts and trends •

    Fine grain reporting • Source code navigation • Parallel execution support • Reports merging • Flexible thresholds • REST API
  14. / … and Code Browsing* * Disabled by default due

    to space/CPU consumption. Enable for projects when you need them
  15. / Examples Coverage Plugin • Reference implementation & Dogfooding •

    github.com/jenkinsci/coverage-plugin • ci.jenkins.io/job/Plugins/job/coverage-plugin • With sources ci.jenkins.io/job/Plugins/job/code-coverage-api-plugin/view/change-requests/job/PR-651/10 Jenkinsfile Core: • github.com/jenkinsci/jenkins • Cobertura • Standard publishing
  16. / Examples Jenkinsfile Runner: • https://github.com/jenkinsci/jenkinsfile-runner/ • JaCoCo / JaCoCo

    Maven Plugin • https://github.com/jenkinsci/jenkinsfile-runner/blob/master/pom.xml • publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')]
  17. / And the Pipeline library • buildPlugin() and buildPluginWithGradle in

    the shared Pipeline Library • 400 codelines but easy for users • Default settings to meet project quality standards github.com/jenkins-infra/pipeline-library/blob/ master/vars/buildPluginWithGradle.groovy
  18. / • “GitHub Checks API for Jenkins project” by Kezhi

    Xiong • Google Summer of Code 2021 • Implementation: Checks API Plugin + GitHub Adapter • jenkins.io/projects/gsoc/2020/projects/github-checks/
  19. / Checks API 101 • Status reporting to GitHub •

    Available in GitHub and GitHub Enterprise • Requires GitHub App authentication • Checks API documentation: https://docs.github.com/en/rest/reference/checks
  20. / GitHub App Authentication • Supported in GitHub Branch Source

    Plugin 2.7.1+. Patch by Tim Jacomb in April 2020 • https://plugins.jenkins.io/github-branch-source • Announcement: https://www.jenkins.io/blog/2020/04/16/github-app-auth entication/
  21. / Setting up GitHub App Authentication Guide: https://github.com/jenkinsci/github-branch-source-plugin/blob/m aster/docs/github-app.adoc 1.

    Register a new GitHub App in GitHub 2. Download access tokens 3. Install the App in your organization/repository 4. Configure Jenkins credentials 5. Use credentials in Branch Source / Org folders
  22. / • Multiple checks • Status reporting • Summary and

    Details • GitHub Markdown support • Annotations https://github.com/jenkinsci/gradle- plugin/pull/382/checks?check_run_i d=19876936412
  23. /

  24. / Integration testing • Multiple technology stacks and toolchains •

    Many teams involved • Sacred knowledge • [Almost] always too late
  25. /

  26. /

  27. /

  28. / Build Unit tests Publish Coverage Integration tests Publish Coverage

    • High infrastructure requirements • Needs parallelization • Reports merging • Virtualization
  29. /

  30. / Build Unit tests Publish Reports Integration tests Publish Reports

    NOW “Shift Left” Fast integrations tests are critical
  31. / Ways to do API integration testing 1. Testing against

    Production/Staging servers 2. Testing against a simplified portable instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Accu- rate Nope
  32. /

  33. /

  34. / Containers are not always slow! • On-demand image build

    • Caching Docker image builds • Suspending containers between tests • Graceful termination ◦ github.com/testcontainers/moby-ryuk testcontainers.org
  35. /

  36. / wiremock.org/docs/stubbing IF (request_url) THEN (response) * * it gets

    MUCH more complex WireMock Config JSON: Client library response request Client App Mock API Server HTTP/2
  37. / Comparison [1/2] 1. Testing against Production/Staging servers 2. Testing

    against a simplified/containerized instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Nope Accu- rate
  38. /

  39. /

  40. / WireMock as a Proxy Tests Real API Provider •

    Fault injection • Recording • Protocol Verification Testcontainers
  41. /

  42. / Code Coverage API: Merging reports • Parameter in older

    versions • Default behavior in new versions • Keeps diffs and code browsing
  43. /

  44. / Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov

    (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
  45. /

  46. / Credits to • All Jenkins contributors • All WireMock

    contributors • Code Coverage API / Coverage Plugin ◦ Shenyu Zheng ◦ Ulli Hafner ◦ Florian Orendi - docs ❤ • GitHub Checks API ◦ Kezhi Xiong ◦ Ulli Hafner ◦ Tim Jacomb • Google Summer of Code!
  47. / Takeaways - Coverage Reporting • Jenkins is one of

    the strongest platforms for coverage reporting and analysis • Code Coverage API supports parallelization and reports mergings • Your technology stack can be supported OOTB or via report format converters • Jenkins integrates with GitHub Checks
  48. / Integration Testing • Great asset if done right •

    Shift left your integration testing • Mocking APIs and services, reducing external dependencies, containers • There are tools for that, including WireMock and Testcontainers
  49. / What to do after the talk? 1. Check out

    our CI in Jenkins repositories ◦ github.com/jenkinsci 2. Try out coverage reporting in Jenkins 3. Shift left your integration tests with WireMock and Testcontainers ◦ wiremock.org ◦ testcontainers.com 4. Share feedback with us! ◦ www.jenkins.io/participate/connect ◦ slack.wiremock.org
  50. / It’s [always] a great time to contribute ;) jenkins.io/participate

    wiremock.org/participate gradle.org/resources