High Cost Tests and High Value Tests: GOTO Chicago 2018

There is a value in writing tests and there is also a cost. The currency is time. The trade-offs are difficult to evaluate because the cost and value are often seen by different people. The writer of the test bears much of the short term cost while long term benefits and cost are borne by the rest of the team. By planning around both the the cost and value of your tests, you’ll improve your tests and your code. How much do slow tests cost? When is it worth it to test an edge case? How can you tell if testing is helping? Here are some strategies to improve your tests and code.

Transcript

1. This slide saves me time in the long run even though it took time to create Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
2. Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
3. When I get a feature request Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
4. Inventory management software Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
5. I need to decide how to test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
6. I could Outside-In TDD Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
7. I could just write integration tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
8. I could not test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
9. I'm making a decision Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
10. Testing will be worth it Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
11. What does worth it mean? ! Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
12. How do I know? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
13. High Cost Tests & High Value Tests Noel Rappin (@noelrap) https://www.tablexi.com http://techdoneright.io http://www.noelrappin.com http://pragprog.com/book/nrtest3 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
14. How can you measure cost and value? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
15. Tests are at the intersection of: Code Process Design Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
16. Time is our metric Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
17. How do tests cost time? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
18. You have to write the test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
19. The test runs A lot Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
20. The test needs to be understood Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
21. The test needs to be fixed Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
22. How do tests save time? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
23. Writing the test improves code design Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
24. Running the test is faster than manual testing Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
25. The test validates the code Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
26. Catches Bugs Faster Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
27. Cost ! vs Value ! Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
28. Dev: ! " Forever: ! " # Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
29. Spoiler alert: There is no right answer Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
30. Strategy not Tactics Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
31. Some data Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
32. End-to-End Integration Capybara Starts With Ends With User Input HTML Output Write Time Run Time 30 mins 0.5 - 3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
33. Workflow Intermediate Object Starts Ends Params and workflow.run Database changes Write Time Run Time 15 mins 0.05 - 0.3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
34. Unit One method Starts Ends Call a method Output of that method Write Time Run Time 1-5 mins 0.001 - 0.04 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
35. Type Specs Total run time Avg run time Write time System 22 12.72 0.570 ~11 hrs Workflow 40 2.36 0.059 ~10 hrs Unit 119 1.86 0.015 ~10 hrs Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
36. System tests are 12% of the tests and 75% of the run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
37. The slowest 4 tests are 40% of the run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
38. The run times have a wider range Kind Min Max Variance Write 1 min 30 min 30x Run 0.001 sec 3 sec 3000x Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
39. Another project Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
40. Type Specs Total time Avg time System 409 579 1.42 Workflow 534 206 0.38 Unit 773 93 0.12 Total 1716 878 0.51 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
41. End to end tests are 23% of tests and 66% of run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
42. What does that suggest? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
43. Balance time spent Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
44. As you write similar tests, costs go down Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
45. Short term cost ! " is not related to test type Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
46. Long term cost ! " # is Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
47. Long term cost: runtime failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
48. In other words: complexity Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
49. Long term savings come from focus Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
50. A small fraction of your tests can be the bulk of your cost Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
51. Big payoff in avoiding the slowest tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
52. That's a long way to get to "don't write slow tests" Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
53. No individual test causes a slow suite Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
54. It's an aggregate set of decisions Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
55. Is a slow test suite inevitable? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
56. Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
57. ! Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
58. Test suites get longer as the code gets more complicated. Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
59. Only CI runs all the tests... we can throw hardware at that Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
60. As long as I can run the tests I'm working on quickly, I'm fine Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
61. A short history of the Rails community's thoughts about testing... Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
62. 1. Testing is great Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
63. 2. Testing is slow Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
64. 3. Let's try and make testing faster Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
65. 4. That's hard. Let's throw CI at it. Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
66. There's still a cost to a long suite Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
67. Giving up causes you to lose the value of tests in improving code Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
68. Inventory management software Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
69. First test: Capybara integration Fails on: • View failure • Controller failure • Handoff from controller to logic failure • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
70. Next test: workflow object Fails on: • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
71. Maybe: Unit test Fails on • specific bit of logic • database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
72. Failure paths: Bad Input "A", -3, "" Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
73. System Test Workflow Test Unit test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
74. Unit test A partial test of the workflow Or a model test Or a logic object Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
75. New feature blank row that can become a new item Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
76. Workflow test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
77. New bug a new row that duplicates a name Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
78. Not a system test workflow or partial workflow test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
79. What if you don't like unit tests? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
80. "TDD is Dead" Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
81. Argument against unit tests: Unit tests cost too much to write Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
82. Often true in a legacy context Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
83. True if framework doesn't handle tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
84. You might hear: Unit tests cause hard to understand designs Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
85. True if you don't like small units Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
86. You might just be writing bad unit tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
87. A lot of Copy/Paste Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
88. A lot of unrelated assertions Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
89. Logic change is far away from method under test Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
90. Capybara is not a unit test framework Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
91. Lack of unit tests: Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
92. Good legacy code strategy Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
93. At the cost of harder to diagnose tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
94. Strategies Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
95. What will make a test fail? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
96. If it can't fail uniquely Do you need it? Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
97. Create the minimum amount of objects needed to trigger the failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
98. Use multiple test failures as an opportunity Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
99. Sometimes you can delete tests Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
100. Use integration tests to save development time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
101. Tests have costs ! " Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
102. Tests have value ! " Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap
103. Noel Rappin (@noelrap) http://pragprog.com/book/nrtest3 workshops@tablexi.com http://techdoneright.io/ Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap