It's time to up your test game

On paper, everyone agrees that tests are important. And yet, more often than not, tests are neglected. In the rare case where we take the time to write them, more often than not it’s considered a cumbersome task, and given less attention than the "real code".

Tests are often as important, if not more so, as the code that will be published on the Play Store. This talk will start with describing a productive mindset when writing tests, followed by a few examples of common pitfalls and how to avoid them, and will conclude with a couple of advanced testing paradigms that can help make your tests cleaner and better.

Transcript

1. It’s time to UP your test game Xavier F. Gouchet, Mobile RUM Team Lead @xgouchet / @datadoghq
2. Monitor your Web and Mobile applications from your users’ perspective • Performance Improvement • Error Management and Alerting • Customer Support • Application Analytics About… Datadog Real User Monitoring 2
3. Agenda • Common testing* pitfalls • Advanced topics : ◦ Fuzzy Testing ◦ Mutation Testing 3
4. 1. Common Testing Pitfalls Writing tests is hard!
5. Tests are hard to maintain 5
6. Causes for poor tests Lazy code It’s “just test”, why should we put too much effort in it? Not thought through Let’s just test the nominal case, what could go wrong? Tests without intent We need to increase our coverage, but what are we testing exactly ? 6
7. Write clean code tests 7
8. Keep your best practices • Code style and naming conventions • KISS • SOLID • Use Patterns • Avoid Hardcoded Values✘ 8
9. Keep in mind the goals of testing • Verify that the code works • Make the contract explicit • Guide the development (TDD) Watch your tests fail✘ • Prevent regressions • Ease maintenance 9
10. Covering all cases 10
11. What are you actually testing • Each test has a single purpose • Test name reflects the intent ◦ `given X when Y then Z` ◦ `A performs B on C event` 11
12. Think of any way the code might break • Rule of thumb ◦ 1+ test for nominal case ◦ 1+ test for each input (invalid, null) ◦ 1+ test for error state(s) • Add a test for every new bug 12
13. 2. Fuzzy Testing Real users aren’t all named Alice or Bob…
14. ✘Hardcoded Values 14
15. What could go wrong here? @Test fun `test get host`() { // Given val email = "username@example.com" // When val host = extractHost(email) // Then assertThat(host).isEqualTo("example.com") } 15
16. Constants are especially bad in Tests • You’re only testing one value … Every … single … time! • Humans are bad at generating “randomness” 16
17. Fuzzy testing 17
18. The main concept ARRANGE Generate “random” data ACT Perform some test ASSERT Verify the output / callbacks matches the test data 18
19. Using random test data… • Overall a wider range of data is tested • Understand boundaries of valid/invalid inputs • Write generic tests without assumptions • Find edge cases 19
20. Using the Elmyr library https://github.com/xgouchet/Elmyr 20
21. Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 21
22. Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 22
23. Basic tests class FooTest { @get:Rule val forge = ForgeRule() @IntForgery var i: Int = 0 // … } 23
24. Generate any kind of data • Boolean • Numeric • String • Collections (List, Set, Map) • Enums 24
25. Adapting to your use cases • Keep the control over the generation of primitives • Generate custom class instances 25
26. Controlling your random data class FooTest { @get:Rule val forge = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 26
27. Controlling your random data class FooTest { @get:Rule val forge = ForgeRule() @IntForgery(min = 13, max = 42) var i: Int = 0 // … } 27
28. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 28
29. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 29
30. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 30
31. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 31
32. Randomize your own classes class BookForgeryFactory : ForgeryFactory<Book> { override fun getForgery(forge: Forge): Book { return Book( bookId = forge.getForgery<UUID>(), title = forge.anAlphabeticalString(), author = forge.anAlphabeticalString(), email = forge.aStringMatching("\w+@\w+.com") ) } } 32
33. Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 33
34. Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 34
35. Keep the random within a controlled range class FooTest { @get:Rule val forge = ForgeRule() .withFactory(BookFactory()) @Forgery lateinit var book: Book // … } 35
36. Making tests reproducible <FooTest.testSomething()> failed with Forge seed 0xdeadfa11 36
37. Making tests reproducible class FooTest { @get:Rule val forge = ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 37
38. Making tests reproducible class FooTest { @get:Rule val forge = ForgeRule(0xdeadfa11) @IntForgery var i: Int = 0 // … } 38
39. 3. Mutation Testing How confident are you about your coverage?
40. SUCCESS: 26/26 (100%) Tests passed — @bloerwald 40
41. Code coverage only tells you which lines are visited, not which lines are tested 41
42. An 80% code coverage doesn’t mean “80% of your code is tested”; it means “at least 20% of your code is not tested” 42
43. ✘Watch your tests fail 43
44. Code review your tests Code has bugs Tests are Code ∴ Tests have bugs 44
45. The Mutation Testing Algorithm • Have some (valid) tests • Mutate the production code i.e. introduce some bugs • Run the tests again • … • Profit 45
46. Using the PITest library https://github.com/koral--/gradle-pitest-plugin 46
47. Add the plugin to your build.gradle apply plugin: 'pl.droidsonroids.pitest' pitest { targetClasses = ['com.example.*'] outputFormats = ['XML', 'HTML'] } 47
48. 48 Mutation Coverage
49. if (x > y) … if (a && b) … return t i++ Before / After Mutation if (x ≤ y) … if (a || b) … return null i-- 49
50. Think about your tests as an incident response team; Mutation testing is similar to running a drill emergency. 50
51. It won’t find bugs in the code, just reveal test issues Useful but not critical Only simulates atomic faults 51
52. Key takeaway 52
53. “You need to be as confident in the tests you code as you are in the code you test.” 53
54. • Test is code • Vary your test inputs • Make sure your tests are actually testing something Being confident in your tests 54
55. https://github.com/DataDog/dd-sdk-android 55
56. 56 Free Trial dtdg.co/droidcon Win a Google Pixel 4
57. Thanks! Any questions? @xgouchet / @datadoghq 57 Presentation template by SlidesCarnival · Icons by Aleksandra Wolska @tutsii159