Release the Mutants and Validate your Tests

Slide deck from my talk at Dreamforce 2022 on mutation testing. Includes links to the CLI plugin and source code.

Transcript

1. Release the Mutants And validate your tests Keir Bowden, CTO BrightGen @bob_buzzard
2. About me CTO at BrightGen Salesforce MVP 14 years Salesforce 35 years experience
3. Agenda The need for mutation testing What is mutation testing? Mutation testing in practice bbmutants CLI Plug-in Key Learnings
4. The need for mutation testing
5. The Need Production Code Unit Tests Quality
6. The Need
7. The Need
8. The Need
9. The Need
10. The Need
11. The Need
12. Quis custodiet ipsos custodes? Who will guard the guards themselves? How do you tests your tests? Changing code should cause test failures Enter the mutants!
13. What is Mutation Testing?
14. 1. Competent Programmer • Code is close to correct • Most faults are simple mistakes 2. Coupling Effects • Simple faults are coupled to complex faults • Fixing simple faults will fix complex faults • Tests that detect mutants would also detect real bugs Two Hypotheses
15. Fault based testing - intentionally break code 1. Test original class 2. Create mutant class - change one aspect 3. Test mutant class 4. Compare test results “Open Box” testing - knows about internals Mutation testing R. Lipton, “Fault Diagnosis of Computer Programs”, Student Report, Carnegie Mellon University, 1971
16. Mutation Testing Original Mutant
17. Mutant Killed • Changing code breaks tests • Mutant is covered by test case Mutant Survived • Changing code did not break test • Mutant is not covered by test case, or • Mutant is equivalent to original class Mutation Testing
18. Syntax changed Functionality did not Suppress (aka delete) them Equivalent Mutant
19. Mutations Type Example Boolean Operator Replace && with || Comparison Operator Replace == with !=, >=, <= Increment Replace ++ with -- Method Call Replace System.now() with System.now.addHours(17) Constant Replace 250000 with 275000 Return Values Replace return true with return false Delete Code Remove lines containing DML statements
20. Mutation Testing in Practice
21. One change per mutant Avoid multiple mutations of the same line Limit mutations of a single class Large test suite means fewer mutants • Target specific classes Generating Mutants
22. Generate new mutants for each cycle • They can’t get any deader • Avoids maintenance overhead Chance of regenerating identical mutants • Change tactics • Target only updated files Generating Mutants
23. Computationally expensive For each mutant • Replace original code • Deploy mutant • Run entire test suite • Restore original code Execute, evaluate, repeat • Execute every mutant, every time Executing Mutants
24. Some mutants will not deploy • Not a weakness of tests • Developers should spot this! • Continuous integration will catch this Suppress (aka delete) them Executing Mutants
25. Expensive • Human Oracle required • Manual checks for all survivors Suppress equivalents Calculate mutation score • Indicator of test quality Fix tests • Ensure original code is still green Evaluating, or Counting the Bodies killed mutants total mutants Mutation Score * 100
26. Well suited to automation • What changes is less important than something changed • Simple “mistakes” introduced • Coupling effect says real bugs would be found Repeated deploy/execute tests Automation
27. New scratch org for cycle Source tracking detects changes • Replace production class with mutant on filesystem • Reverse after testing CLI to push source and run tests SFDX is Key
28. bbmutants CLI Plug-in
29. bbmutants:generate Once per cycle Configurable • Target specific files • Limit mutants per file • Specify strings to replace • Specify values to replace Generate Mutants
30. Avoid repetition - randomise: • Starting line in file (if limiting mutants) • Choice of mutation • Choice of constant • Choice of comparison operator Generate Mutants
31. bbmutants:execute Deploy and run tests • Delete mutants that won’t deploy List survivors Calculate mutation score Iterative, after fixing tests • Delete equivalents Execute Mutants
32. Plug-in Demo
33. Key Learnings
34. Allow sufficient time Easier with smaller files • Fewer mutants Mutation scores won’t surprise you! • Lots of test effort = high score • Focus on coverage = low score Key Learnings
35. Mutation test at the end of development Automatically generated perform as well as hand crafted Constants are less helpful than expected • Tests typically reference production values Removing DML didn’t break tests • Tendency to check returned value over database contents Key Learnings
36. Key Learnings Pros Cons Detect untested code Vast number of potential mutants Measurable mutation score Manual verification of surviving mutants Improve test suite quality Full test run for each mutant Tests how the code doesn’t work Knowledge of code internals required
37. CLI Plug-in on NPM - https://bobbuzz.me.uk/mutationplugin sfdx plugins:install bbmutation Source on Github - https://bobbuzz.me.uk/mutationsrc Resources
39. Connect with Developer Trailblazers from Anywhere Join the Community www.salesforce.com/devcommunity/ #SFDevsAMA
40. 40 Thank you