Testing in the postapocalyptic future

Talk presented at Scala Days 2019 in Lausanne.

One popular way of testing in the Scala community is property-based testing - by generating random, often unexpected test data, it can uncover shortcomings in our implementations. In this talk, you will learn about a completely different approach, mutation testing: By mutating your code, it tests your tests and tells you a lot more about the quality of your tests than metrics like code coverage. We're going to cover what mutation testing us, how you can use it in your Scala projects, how it compares to property-based testing, and the challenges of implementing this approach in Scala.

Transcript

1. 1 1 2 . 0 6 . 2 0 1 9 L A U S A N N E / S C A L A D A Y S Testing in the postapocalyptic future Daniel Westheide Twitter: @kaffeecoder
2. 2 Postapocalyptic present
3. 3 What is a strong test suite?
4. 4 Coverage?
5. Complex logic in need of a test 5 object Math { def nonNegative(x: Int): Boolean = x >= 0 }
6. 100% branch coverage 6 import minitest._ object MathTest extends SimpleTestSuite { import Math.nonNegative test("1 is non-negative") { assert(nonNegative(1)) } test("0 is non-negative") { assert(nonNegative(0)) } test("-1 is negative") { assert(!nonNegative(-1)) } }
7. Still 100% branch coverage 7 import minitest._ object MathTest extends SimpleTestSuite { import Math.nonNegative test("1 is non-negative") { assert(nonNegative(1)) } }
8. 100% coverage? Holy moly! 8 import minitest._ object MathTest extends SimpleTestSuite { import Math.nonNegative test("1 is non-negative") { println(nonNegative(1)) } }
9. 9 So what is a strong test suite?
10. Characteristics of a strong test suite 10 • tests actually have assertions • not just the happy path • covers corner cases
11. 11 Property-based testing
12. More maths 12 object Math { def nonNegative(x: Int): Boolean = x >= 0 def nonNegativeRatio(xs: List[Int]): BigDecimal = { val count = xs.count(nonNegative) BigDecimal(count) * 100 / xs.size } }
13. Example-based testing 13 import minitest._ object MathTest extends SimpleTestSuite { import Math._ test("100% non-negative ratio") { assertEquals(nonNegativeRatio(List(1, 2, 3)).toInt, 100) } test("0% non-negative ratio") { assertEquals(nonNegativeRatio(List(-5, -3, -2)).toInt, 0) } test("50% non-negative ratio") { assertEquals(nonNegativeRatio(List(-5, 1, 0, -1)).toInt, 50) } }
14. Property-based testing 14 import minitest.SimpleTestSuite import minitest.laws.Checkers import org.scalacheck.Prop.AnyOperators object MathProperties extends SimpleTestSuite with Checkers { import Math._ test("ratios of given numbers and inverted numbers adds up to 100") { check1((xs: List[Int]) => { val ys = xs.map(invert) (nonNegativeRatio(xs) + nonNegativeRatio(ys)).toInt ?= 100 }) } private def invert(x: Int): Int = x match { case 0 => -1 case Int.MinValue => Int.MaxValue case _ => x * -1 } }
15. Boom! 15 - ratios of given numbers and inverted numbers adds up to 100 *** FAILED *** Exception raised on property evaluation. (Checkers.scala:39) > ARG_0: List() > Exception: java.lang.ArithmeticException: Division undefined minitest.api.Asserts.fail(Asserts.scala:103)
16. 16 Who watches the watchers?
17. 17 Mutation testing
18. Mutation testing 18 How does it work? • It changes the code under test • Then runs your test suite against the modified code • Do your tests care?
19. Mutations 19 Examples: • Replace > with >= • Replace > with < • Replace >= with > • Replace >= with <=
20. From mutation to mutant 20 // function under test: def posRange(start: Int, end: Int): Boolean = start > 0 && end > start // mutant 1: def posRange(start: Int, end: Int): Boolean = start >= 0 && end > start // mutant 2: def posRange(start: Int, end: Int): Boolean = start < 0 && end > start // mutant 3: def posRange(start: Int, end: Int): Boolean = start > 0 && end >= start // mutant 4: def posRange(start: Int, end: Int): Boolean = start > 0 && end < start // mutant 5: def posRange(start: Int, end: Int): Boolean = start > 0 || end > start
21. Detecting a mutant 21 def nonNegative(x: Int): Boolean = x >= 0 // MUTANT 1: def nonNegative(x: Int): Boolean = x > 0 // MUTANT 2: def nonNegative(x: Int): Boolean = x < 0 // MUTANT 3: def nonNegative(x: Int): Boolean = x == 0 import minitest._ object MathTest extends SimpleTestSuite { import Math._ // FAILS FOR MUTANT 1 and 2! test("0 is non-negative") { assert(nonNegative(0)) } // FAILS FOR MUTANT 2 and 3 ! test("1 is non-negative") { assert(nonNegative(1)) } // FAILS FOR MUTANT 2 and 3! test("-1 is negative") { assert(!nonNegative(1)) } }
22. Missing a mutant 22 def nonNegative(x: Int): Boolean = x >= 0 // MUTANT 1: def nonNegative(x: Int): Boolean = x > 0 // MUTANT 2: def nonNegative(x: Int): Boolean = x < 0 // MUTANT 3: def nonNegative(x: Int): Boolean = x == 0 import minitest._ object MathTest extends SimpleTestSuite { import Math._ // SUCCEEDS FOR MUTANT 1! test("1 is non-negative") { assert(nonNegative(1)) } }
23. 23 Do I need this?
24. 24 Do I need this? Yes, you do!
25. 25 “Okay Daniel, you have convinced me! I need this mutation testing thing for my project! Where can I buy it?“ HAYDEN HIPSTER Purchasing Director at Brontosaurus Enterprise Solutions
28. Salander Mutanderer 28 Ingredients: SBT and Scalameta 1. Find all Scala source files under test 2. For each source file, yield mutants for applicable mutations 3. Run test suite for each mutant 4. Analyse and log results
29. Salander Mutanderer 29 def salanderMutanderer(incantation: Incantation)( implicit logger: ManagedLogger): Unit = { val results = for { sourceFile <- scalaSourceFiles(incantation) mutant <- summonMutants(sourceFile) } yield runExperiment(incantation, mutant) val stats = analyseResults(results) logResults(stats) }
30. Scalameta 30 • library for reading, analysing, transforming, and generating Scala code • Tree – syntax tree representation of Scala code – pattern matching – comparison – traversal
31. Traversing a tree 31 def collect[T](fn: PartialFunction[Tree, T]): List[T]
32. Defining a mutation 32 type Mutation = SourceFile => PartialFunction[Tree, List[Mutant]] val `Change >= to > or ==` : Mutation = sourceFile => { case original @ Term.ApplyInfix(_, Term.Name(">="), Nil, List(_)) => List( Mutant(UUID.randomUUID(), sourceFile, original, original.copy(op = Term.Name(">")), "changed >= to >"), Mutant(UUID.randomUUID(), sourceFile, original, original.copy(op = Term.Name("==")), "changed >= to ==") ) }
33. The spellbook 33 val spellbook: List[Mutation] = List( `Change >= to > or ==`, `Replace Int expression with 0` )
34. Summoning mutants 34 private def summonMutants(sourceFile: SourceFile): List[Mutant] = sourceFile.source.collect(Mutations.in(sourceFile)).flatten object Mutations { def in(sourceFile: SourceFile): PartialFunction[Tree, List[Mutant]] = spellbook .map(_.apply(sourceFile)) .foldLeft(PartialFunction.empty[Tree, List[Mutant]])(_.orElse(_)) }
35. Result of an experiment 35 sealed trait Result extends Product with Serializable final case class Detected(mutant: Mutant) extends Result final case class Undetected(mutant: Mutant) extends Result final case class Error(mutant: Mutant) extends Result
36. Running an experiment 36 private def runExperiment(incantation: Incantation, mutant: Mutant)( implicit logger: Logger): Result = { val mutantSourceDir = createSourceDir(incantation, mutant) val settings = mutationSettings( mutantSourceDir, incantation.salanderTargetDir / mutant.id.toString) val newState = Project .extract(incantation.state) .appendWithSession(settings, incantation.state) Project.runTask(test in Test, newState) match { case None => Error(mutant) case Some((_, Value(_))) => Undetected(mutant) case Some((_, Inc(_))) => Detected(mutant) } }
37. 37 Example
38. $ sbt salanderMutanderer 38 object Math { def nonNegative(x: Int): Boolean = x >= 0 } [info] Passed: Total 1, Failed 0, Errors 0, Passed 1 [info] Total mutants: 2, detected mutants: 0 (0%) [info] Undetected mutants: [info] /home/daniel/projects/private/salander/src/main/scala/lib/Math.scala:5:38: changed >= to == [info] /home/daniel/projects/private/salander/src/main/scala/lib/Math.scala:5:38: changed >= to > [success] Total time: 8 s, completed 6 Jun 2019, 11:24:13 import minitest._ object MathTest extends SimpleTestSuite { import Math._ test("-1 is non-negative") { assert(!nonNegative(-1)) } }
39. Weaknesses 39 • very limited set of mutations • only basic console reporting • performance
40. A slight case of thread necromancy 40
41. 41 “The future is already here — it‘s just not very evenly distributed.“ WILLIAM GIBSON
42. Supported mutations 42 • boolean literal: e.g. replace true with false • conditional expression: e.g. if (x < 3) ... => if (false) ... • equality operator: e.g. replace > with >= • logical operator: e.g. replace && with || • method expression: e.g. a.take(b) => a.drop(b) • string literal: e.g. replace “Magic” with “”
43. HTML report 43
44. Performance 44 • there are faster compilers than scalac • potentially great number of mutants – mutate source file – compile – run tests
45. Mutation switching 45 def nonNegative(x: Int): Boolean = sys.env.get("ACTIVE_MUTATION") match { case Some("0") => x > 0 case Some("1") => x <= 0 case Some("2") => x == 0 case _ => x >= 0 }
46. Usage 46 • available as a plugin for SBT or Maven • addSbtPlugin(“io.stryker-mutator” % “sbt-stryker4s” % “0.5.0”) • sbt stryker • configuration using stryker4s.conf file – disable certain mutations – narrow down source files to be considered for mutation – change base directory – set threshold for detection rate
47. Usage patterns 47 • do not put this into your delivery pipeline • do not enforce a minimum detection rate • run either – locally from time to time – in a scheduled job • plan time to go through the report and improve your test suite
48. 48 Challenges and limitations
49. Non-compiling mutants 49 • having a filter method doesn‘t guarantee that there is a filterNot method • even operators are just methods, e.g. > and >= • problem: no type information available • compile errors cause the whole mutation test run to fail
50. Other issues 50 • Infinite loops caused by mutations cannot be stopped • no native support for multi-module projects • performance: running the complete test suite for each mutant
51. Ideas and plans 51 • Performance: – keeping the test process alive – ignore mutants without test coverage – for each mutant, only run tests hitting the respective code • Robustness – rolling back mutations causing compile errors – exploring feasability of type analysis using SemanticDB
52. 52 Towards a postapocalyptic future!
53. Links 53 • https://github.com/dwestheide/salander • https://github.com/stryker-mutator/stryker4s • https://stryker-mutator.io/blog/2018-10-6/mutation-switching • https://scalameta.org/
54. Thank you! Questions? 54 Daniel Westheide daniel.westheide@innoq.com Twitter: @kaffeecoder Website: https://danielwestheide.com Krischerstr. 100 40789 Monheim am Rhein Germany +49 2173 3366-0 Ohlauer Str. 43 10999 Berlin Germany +49 2173 3366-0 Ludwigstr. 180E 63067 Offenbach Germany +49 2173 3366-0 Kreuzstr. 16 80331 München Germany +49 2173 3366-0 Hermannstrasse 13 20095 Hamburg Germany +49 2173 3366-0 Gewerbestr. 11 CH-6330 Cham Switzerland +41 41 743 0116 innoQ Deutschland GmbH innoQ Schweiz GmbH www.innoq.com