Mutation Testing

Brown Bag lightning talk

Transcript

1. Mutation Testing Chris Sinjakli
2. Testing is a good thing But how do we know our tests are good?
3. Code coverage is a start But it can give a “good” score with really dreadful tests
4. Really dreadful tests public int addTwoNumbers(int a, int b) { return a – b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } Coverage: 100% Usefulness: 0
6. A contrived example But how could we detect it?
7. Mutation Testing! “Who watches the watchmen?”
8. If you can change the code, and a test doesn’t fail, either the code is never run or the tests are wrong.
9. Going with our previous example public int addTwoNumbers(int a, int b) { return a – b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } Let’s change something
10. Going with our previous example public int addTwoNumbers(int a, int b) { return a + b; } ... @Test public void shouldAddTwoNumbers() { int result = addTwoNumbers(1, 1); assertTrue(true); } This still passes
11. So it caught a really rubbish test How about something slightly less obvious?
12. Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Coverage: 100% Usefulness: >0 But still wrong
13. Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a && b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Mutate
14. Slightly less obvious (and I mean slightly) public int checkConditions(boolean a, boolean b) { if (a || b) { return 42; } else { return 0; } } @Test public void testBothFalse() { int result = checkConditions(false, false); assertEquals(0, result); } @Test public void testBothTrue () { int result = checkConditions(true, true); assertEquals(42, result); } Passing tests
15. Mutation testing caught our mistake :D
16. Useful technique But still has its flaws
17. The downfall of mutation (Equivalent Mutants) int index = 0 while (someCondition) { doStuff(); index++; if (index == 100) { break; } } int index = 0 while (someCondition) { doStuff(); index++; if (index >= 100) { break; } } Mutates to But the programs are equivalent, so no test will fail
18. Tools Some Java, then some Ruby
19. Java • Loads of tools to choose from • Bytecode vs source mutation • Will look at PIT (seems like one of the better ones)
20. PIT - pitest.org • Works with “everything” – Command line – Ant – Maven • Bytecode level mutations (faster) • Very customisable – Exclude classes/packages from mutation – Choose which mutations you want – Timeouts • Makes pretty HTML reports (line/mutation coverage)
21. Ruby
22. Ruby • Mutant seems to be the new favourite • Runs in Rubinius (1.8 or 1.9 mode) • Only supports RSpec • Easy to set up rvm install rbx-head rvm use rbx-head gem install mutant • And easy to use mutate “ClassName#method_to_test” spec
23. Summary • Seems like it could identify areas of weakness in our tests • At the same time, could be very noisy • Might be worth just trying it against an existing project and seeing what happens
24. Questions?