[2020.11 Meetup] Lisa Crispin - Testing in DevOps

1. @lisacrispin An Agile Thinking Tool for Building a DevOps Culture DevOps Lisbon, November 16, 2020 Lisa Crispin With material by Janet Gregory Copyright 2020
2. @lisacrispin A little about me Part of OutSystems Quality Team lisa@agiletester.ca “Agile Testing for the Whole Team” now available as a remote course New!
3. @lisacrispin My message today How using the agile testing quadrants model can identify risks and testing activities to help grow a quality-focused DevOps culture and succeed with continuous delivery / deployment. 3
4. @lisacrispin What’s “DevOps”? • Whole team approach to quality – applied to Operations too • Small, frequent, low-risk changes • It’s a culture shift • No more throwing the release candidate over the wall to Ops • We build it – we run it – we own it!
5. @lisacrispin A quality-focused DevOps culture means …
6. @lisacrispin What is continuous delivery? Continuous Delivery is the ability to get changes of all types— including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way. We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. ~ Jez Humble and Dave Farley
7. @lisacrispin Continuous Delivery (CD)
8. @lisacrispin Continuous deployment (also CD)
9. @lisacrispin OK, continuous makes sense How will we ever fit testing into CD????
10. @lisacrispin the quadrants and how they evolved Photo by Suzanne D. Williams on Unsplash
11. @lisacrispin Adapted from original by Brian Marick 2003 2009
12. @lisacrispin 2014
13. @lisacrispin 2019 Q1 Q2 Q3 Q4
14. @lisacrispin Power of the quadrants • A thinking tool • Provides a common language • Emphasizes “whole team” • Plan testing at different levels • Tests for “Story Done” • Tests for “Feature Done” • Tests for “Release Done”
15. @lisacrispin Functional examples (expected behavior / misbehaviors) Examples – 3rd party integrations Contract testing (on APIs) Unit testing Static analysis Component integration Test doubles for external integrations Data integrity Smoke tests on dev env. Exploratory testing Usability Monitoring Observability Testing in Production Recoverability Performance Load / Stress / Reliability Scalability, Safety Infrastructure as code
16. @lisacrispin Try out the quadrants! Test a web app using Stripe to let donors make payments.
17. @lisacrispin Apply the quadrants to a deployment pipeline • Infrastructure as code • Test environments • Team testing skills Photo by Sven Kucinic on UnSplash
18. @lisacrispin Evaluate risks and plan mitigation strategies Release feature toggles – “hide” changes until confident to release • Enable incremental/iterative approach • For tests that critique the product What risks will you accept pending feedback in production? • scale, data, networking, configuration, user behavior
19. @lisacrispin Apply quadrants to a simple deployment pipeline Local Unit tests Static analysis Test doubles -Conversations -Get examples -Create tests -Form a shared understanding of the problem Run unit tests Run api auto tests Observability Monitoring Test in Prod Run API auto tests Security, load, performance, recovery…. Run workflow auto tests UAT
20. Create build failure alerts Script to auto build/deploy “preview” test envs Visualize & improve pipeline design Train everyone in exploratory testing Implement build status dashboard Define failure analysis runbook Define SLOs & SLIs Train everyone in failure analysis tools Script to send build results to Slack Evaluate auto test coverage Script to refresh test databases Evaluate SLI metrics in pre-prod
21. @lisacrispin Going back to the quadrants… What do we need to do to move towards continuous delivery?
22. @lisacrispin What are the benefits?
23. @lisacrispin Improved communications No matter how it looks at first, it's always a people problem. -- Jerry Weinberg Most software problems are communication problems! The quadrants give us a framework to help with: • Common language • Shared understanding • Whole team approach to testing
24. @lisacrispin Designers Getting every perspective • Programmers • SREs / platform engineers • Ops specialists • Business stakeholders • Testers • Designers • …. …….. SRE Ops Business Delivery Teams
25. @lisacrispin Applying test activities continuously Graphic by Janet Gregory, inspired by Dan Ashby
26. @lisacrispin Release with confidence 26 It’s all a balance, and it depends on your context Photo by Bekir Domex on Unsplash
27. @lisacrispin Further reading https://agiletester.ca/more-agile-testing-the-book/ Download Chapter 8: Using Models to Help Plan Download Chapter 23: Testing in DevOps https://agiletester.ca/agile-testing-condensed-a-brief- introduction/ also has information on both of these topics
28. Agile Testing Condensed, A Brief Introduction Agile Testing: A Practical Guide for Testers and Agile Teams More Agile Testing: Learning Journeys for the Whole Team By Janet Gregory and Lisa Crispin https://agiletester.ca https://agiletestingfellow.com Contact info https://lisacrispin.ca Email: lisa@agiletester.ca Twitter: lisacrispin 28