High Cost Tests and High Value Tests

There is a value in writing tests and there is also a cost. The currency is time. The trade-offs are difficult to evaluate because the cost and value are often seen by different people. The writer of the test bears much of the short term cost while long term benefits and cost are borne by the rest of the team. By planning around both the the cost and value of your tests, you’ll improve your tests and your code. How much do slow tests cost? When is it worth it to test an edge case? How can you tell if testing is helping? Here are some strategies to improve your tests and code.

Transcript

1. This slide saves me time in the long run even though it took time to create Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
2. When I get a feature request Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
3. Inventory management software Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
4. I need to decide how to test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
5. I could Outside-In TDD Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
6. I could just write integration tests Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
7. I could not test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
8. I'm making a decision Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
9. Testing will be worth it Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
10. What does worth it mean? ! Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
11. How do I know? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
12. High Cost Tests & High Value Tests Noel Rappin (@noelrap) https://www.tablexi.com http://techdoneright.io http://www.noelrappin.com http://pragprog.com/book/nrtest3 Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
13. How can you measure cost and value? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
14. Tests are at the intersection of: Code Process Design Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
15. Time is our metric Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
16. How do tests cost time? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
17. You have to write the test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
18. The test runs A lot Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
19. The test needs to be understood Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
20. The test needs to be fixed Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
21. How do tests save time? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
22. Writing the test improves code design Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
23. Running the test is faster than manual testing Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
24. The test validates the code Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
25. Catches Bugs Faster Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
26. Cost ! vs Value ! Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
27. Dev: ! " Forever: ! " # Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
28. Spoiler alert: There is no right answer Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
29. Strategy not Tactics Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
30. Some data Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
31. End-to-End Integration Capybara Starts With Ends With User Input HTML Output Write Time Run Time 30 mins 0.5 - 3 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
32. Workflow Intermediate Object Starts Ends Params and workflow.run Database changes Write Time Run Time 15 mins 0.05 - 0.3 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
33. Unit One method Starts Ends Call a method Output of that method Write Time Run Time 1-5 mins 0.001 - 0.04 seconds Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
34. Type Specs Total run time Avg run time Write time System 22 12.72 0.570 ~11 hrs Workflow 40 2.36 0.059 ~10 hrs Unit 119 1.86 0.015 ~10 hrs Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
35. System tests are 12% of the code and 75% of the output Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
36. The slowest 4 tests are 40% of the run time Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
37. The run times have a wider range Kind Min Max Variance Write 1 min 30 min 30x Run 0.001 sec 3 sec 3000x Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
38. Another project Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
39. Type Specs Total time Avg time Cucumber 116 520 4.48 System 175 110 0.62 Workflow 493 209 0.42 Unit 772 92 0.19 Total 1571 987 6.28 Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
40. End to end tests are 18% of tests and 67% of run time Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
41. What does that suggest? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
42. Balance time spent Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
43. As you write similar tests, costs go down Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
44. Short term cost ! " is not related to test type Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
45. Long term cost ! " # is Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
46. Long term cost: runtime failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
47. In other words: complexity Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
48. Long term savings come from focus Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
49. A small fraction of your tests can be the bulk of your cost Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
50. Big payoff in avoiding the slowest tests Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
51. That's a long way to get to "don't write slow tests" Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
52. No individual test causes a slow suite Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
53. It's an aggregate set of decisions Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
54. Inventory management software Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
55. First test: Capybara integration Fails on: • View failure • Controller failure • Handoff from controller to logic failure • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
56. Next test: workflow object Fails on: • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
57. Maybe: Unit test Fails on • specific bit of logic • database access failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
58. Failure paths: Bad Input "A", -3, "" Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
59. System Test Workflow Test Unit test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
60. Unit test A partial test of the workflow Or a model test Or a logic object Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
61. New feature blank row that can become a new item Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
62. Workflow test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
63. New bug a new row that duplicates a name Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
64. Not a system test workflow or partial workflow test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
65. Signs your test is too big? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
66. A lot of Copy/Paste Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
67. A lot of unrelated assertions Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
68. Logic change is far away from method under test Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
69. Capybara is not a unit test framework Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
70. What if you don't like unit tests? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
71. "TDD is Dead" Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
72. Saw this as JS testing advice: Write tests Not Too Many Mostly integration Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
73. Unit tests cost too much to write Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
74. Often true in a legacy context Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
75. True if framework doesn't handle tests Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
76. Unit tests force hard to understand designs Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
77. True if you don't like small units Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
78. ! matters less if you are not running tests while developing Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
79. Really good legacy code strategy Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
80. Cost of harder to diagnose tests Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
81. Strategies Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
82. What will make a test fail? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
83. It it can't fail uniquely Do you need it? Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
84. Create the minimum amount of objects needed to trigger the failure Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
85. Use multiple test failures as an opportunity Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
86. Sometimes you can delete tests Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
87. Use integration tests to save development time Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
88. Tests have costs ! " Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
89. Tests have value ! " Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap
90. Noel Rappin (@noelrap) http://pragprog.com/book/nrtest3 (15% off with code "RubyConf NOLA") & http://techdoneright.io/ Noel Rappin | High Cost Tests and High Value Tests | RubyConf 2017 | https://www.tablexi.com | @noelrap