Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Cost Tests and High Value Tests: GOTO Chic...

Noel Rappin
April 26, 2018
Technology
2
140

High Cost Tests and High Value Tests: GOTO Chicago 2018

There is a value in writing tests and there is also a cost. The currency is time. The trade-offs are difficult to evaluate because the cost and value are often seen by different people. The writer of the test bears much of the short term cost while long term benefits and cost are borne by the rest of the team. By planning around both the the cost and value of your tests, you’ll improve your tests and your code. How much do slow tests cost? When is it worth it to test an edge case? How can you tell if testing is helping? Here are some strategies to improve your tests and code.

Noel Rappin

April 26, 2018
Tweet

More Decks by Noel Rappin

See All by Noel Rappin
Rails on Ruby: How Ruby Makes Rails Great
noelrap
0
310
In Defense of Metaprogramming
noelrap
0
420
Rails 7 Front End Tooling
noelrap
1
540
More Engineers, More Problems: Solutions for Big Teams
noelrap
0
190
The Rails 6 Front End: Building with Webpacker and Stimulus
noelrap
3
1.2k
The Developer's Toolkit
noelrap
0
160
High Cost Tests and High Value Tests
noelrap
1
160
The Road To Legacy is Paved With Good Intentions
noelrap
1
160
Test Doubles are Not To Be Mocked
noelrap
1
370

Other Decks in Technology

See All in Technology
【若手エンジニア応援LT会】ソフトウェアを学んできた私がインフラエンジニアを目指した理由
kazushi_ohata
0
150
適材適所の技術選定 〜GraphQL・REST API・tRPC〜 / Optimal Technology Selection
kakehashi
1
170
Amazon CloudWatch Network Monitor のススメ
yuki_ink
1
200
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
2
3.2k
複雑なState管理からの脱却
sansantech
PRO
1
140
AWS Lambda のトラブルシュートをしていて思うこと
kazzpapa3
2
170
Engineer Career Talk
lycorp_recruit_jp
0
160
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
5
750
OCI Security サービス 概要
oracle4engineer
PRO
0
6.5k
Taming you application's environments
salaboy
0
180
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
Python(PYNQ)がテーマのAMD主催のFPGAコンテストに参加してきた
iotengineer22
0
470

Featured

See All Featured
Being A Developer After 40
akosma
86
590k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
How To Stay Up To Date on Web Technology
chriscoyier
788
250k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
Into the Great Unknown - MozCon
thekraken
32
1.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
93
16k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.7k
Become a Pro
speakerdeck
PRO
25
5k
Optimizing for Happiness
mojombo
376
70k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
6
410
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
Fashionably flexible responsive web design (full day workshop)
malarkey
405
65k

Transcript

  1. This slide saves me time in the long run even

    though it took time to create Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  2. Noel Rappin | High Cost Tests and High Value Tests

    | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  3. When I get a feature request Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  4. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  5. I need to decide how to test Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  6. I could Outside-In TDD Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  7. I could just write integration tests Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  8. I could not test Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  9. I'm making a decision Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  10. Testing will be worth it Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  11. What does worth it mean? ! Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  12. How do I know? Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  13. High Cost Tests & High Value Tests Noel Rappin (@noelrap)

    https://www.tablexi.com http://techdoneright.io http://www.noelrappin.com http://pragprog.com/book/nrtest3 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  14. How can you measure cost and value? Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  15. Tests are at the intersection of: Code Process Design Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  16. Time is our metric Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  17. How do tests cost time? Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  18. You have to write the test Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  19. The test runs A lot Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  20. The test needs to be understood Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  21. The test needs to be fixed Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  22. How do tests save time? Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  23. Writing the test improves code design Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  24. Running the test is faster than manual testing Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  25. The test validates the code Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  26. Catches Bugs Faster Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  27. Cost ! vs Value ! Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  28. Dev: ! " Forever: ! " # Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  29. Spoiler alert: There is no right answer Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  30. Strategy not Tactics Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  31. Some data Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  32. End-to-End Integration Capybara Starts With Ends With User Input HTML

    Output Write Time Run Time 30 mins 0.5 - 3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  33. Workflow Intermediate Object Starts Ends Params and workflow.run Database changes

    Write Time Run Time 15 mins 0.05 - 0.3 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  34. Unit One method Starts Ends Call a method Output of

    that method Write Time Run Time 1-5 mins 0.001 - 0.04 seconds Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  35. Type Specs Total run time Avg run time Write time

    System 22 12.72 0.570 ~11 hrs Workflow 40 2.36 0.059 ~10 hrs Unit 119 1.86 0.015 ~10 hrs Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  36. System tests are 12% of the tests and 75% of

    the run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  37. The slowest 4 tests are 40% of the run time

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  38. The run times have a wider range Kind Min Max

    Variance Write 1 min 30 min 30x Run 0.001 sec 3 sec 3000x Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  39. Another project Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  40. Type Specs Total time Avg time System 409 579 1.42

    Workflow 534 206 0.38 Unit 773 93 0.12 Total 1716 878 0.51 Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  41. End to end tests are 23% of tests and 66%

    of run time Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  42. What does that suggest? Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  43. Balance time spent Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  44. As you write similar tests, costs go down Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  45. Short term cost ! " is not related to test

    type Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  46. Long term cost ! " # is Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  47. Long term cost: runtime failure Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  48. In other words: complexity Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  49. Long term savings come from focus Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  50. A small fraction of your tests can be the bulk

    of your cost Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  51. Big payoff in avoiding the slowest tests Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  52. That's a long way to get to "don't write slow

    tests" Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  53. No individual test causes a slow suite Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  54. It's an aggregate set of decisions Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  55. Is a slow test suite inevitable? Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  56. Noel Rappin | High Cost Tests and High Value Tests

    | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  57. ! Noel Rappin | High Cost Tests and High Value

    Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  58. Test suites get longer as the code gets more complicated.

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  59. Only CI runs all the tests... we can throw hardware

    at that Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  60. As long as I can run the tests I'm working

    on quickly, I'm fine Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  61. A short history of the Rails community's thoughts about testing...

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  62. 1. Testing is great Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  63. 2. Testing is slow Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  64. 3. Let's try and make testing faster Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  65. 4. That's hard. Let's throw CI at it. Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  66. There's still a cost to a long suite Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  67. Giving up causes you to lose the value of tests

    in improving code Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  68. Inventory management software Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  69. First test: Capybara integration Fails on: • View failure •

    Controller failure • Handoff from controller to logic failure • Logic failure • Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  70. Next test: workflow object Fails on: • Logic failure •

    Database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  71. Maybe: Unit test Fails on • specific bit of logic

    • database access failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  72. Failure paths: Bad Input "A", -3, "" Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  73. System Test Workflow Test Unit test Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  74. Unit test A partial test of the workflow Or a

    model test Or a logic object Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  75. New feature blank row that can become a new item

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  76. Workflow test Noel Rappin | High Cost Tests and High

    Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  77. New bug a new row that duplicates a name Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  78. Not a system test workflow or partial workflow test Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  79. What if you don't like unit tests? Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  80. "TDD is Dead" Noel Rappin | High Cost Tests and

    High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  81. Argument against unit tests: Unit tests cost too much to

    write Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  82. Often true in a legacy context Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  83. True if framework doesn't handle tests Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  84. You might hear: Unit tests cause hard to understand designs

    Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  85. True if you don't like small units Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  86. You might just be writing bad unit tests Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  87. A lot of Copy/Paste Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  88. A lot of unrelated assertions Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  89. Logic change is far away from method under test Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  90. Capybara is not a unit test framework Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  91. Lack of unit tests: Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  92. Good legacy code strategy Noel Rappin | High Cost Tests

    and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  93. At the cost of harder to diagnose tests Noel Rappin

    | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  94. Strategies Noel Rappin | High Cost Tests and High Value

    Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  95. What will make a test fail? Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  96. If it can't fail uniquely Do you need it? Noel

    Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  97. Create the minimum amount of objects needed to trigger the

    failure Noel Rappin | High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  98. Use multiple test failures as an opportunity Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  99. Sometimes you can delete tests Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  100. Use integration tests to save development time Noel Rappin |

    High Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  101. Tests have costs ! " Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  102. Tests have value ! " Noel Rappin | High Cost

    Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap

  103. Noel Rappin (@noelrap) http://pragprog.com/book/nrtest3 [email protected] http://techdoneright.io/ Noel Rappin | High

    Cost Tests and High Value Tests | GOTO Chicago 2017 | https://www.tablexi.com | @noelrap