Are Your Tests Slowing You Down?

Testing is a Good Thing, right? Especially automated testing. But "Good things come to those who wait" is not something that's going to appeal to the busy developer. You want results, and you want them now. You're in The Zone working on a problem, and the last thing you want is to break your flow wrestling with your testing framework or waiting for the tests to finish running.

More code means more tests. More coverage means more tests. More tests mean more time. Time that you want to spend being productive, creative, innovative. How can you balance the need for quality with the need for speed?

In this talk, Trisha will identify issues that slow down developers when writing, running and debugging tests, and look at tools that can help developers with each of these problems. There will be live coding, analysis of social media poll results, an overview of solutions in this space, "best practice" recommendations, and machine learning will be mentioned at some point.

Transcript

1. Are Your Tests Slowing You Down? Trisha Gee
2. https://trishagee.com/books/
3. Tests Make You More Productive? Tests Make You More Productive
6. So it’s OK not to write tests, right?
7. “Testing is Xanax for programmers, it’s an anxiety reducing technique.” https://youtu.be/guycIP56YeY Kent Beck
8. What slows down your testing?
9. What slows down your testing?
10. What slows down your testing?
11. 1. Writing 2. Troubleshooting 3. Running
12. 1. Writing Tests Takes Time
13. Use Your IDE
14. AI Will Make You Super Productive
15. AI is good for generating mundane and/or predictable code
16. AI is good for generating test data
17. AI can consider edge cases we haven’t thought of
18. AI is a decent companion …but it doesn’t replace you
19. AI is a lying toad
20. “The keyboard is not the bottleneck.” Dave Farley
21. “The productivity myth suggests that anything we spend time on is up for automation…. The importance and value of thinking about our work and why we do it is waved away as a distraction. The goal of writing… is filling a page rather than the process of thought that a completed page represents.” https://www.techpolicy.press/challenging-the-myths-of-generative-ai/ Eryk Salvaggio
22. Bring a friend
23. “We recommend that the developers, who are good at writing efficient, maintainable code, work together with the testers, who are good at specifying test cases, to automate tests.” Agile Testing Condensed: A Brief Introduction Lisa Crispin, Janet Gregory
24. https://bit.ly/3RG85vc
25. “Creativity is a wild mind and a disciplined eye.” Dorothy Parker
26. Invest the time in writing to save time troubleshooting
27. 2. Troubleshooting Takes Time
28. What is this test anyway?
30. Your test needs to tell you what is expected and why, so you understand what to do when it fails.
34. Why do tests fail? Body text
35. ‣ Code is wrong ‣ Test is wrong
36. Something changed
37. • Production code • Test code • Refactoring/structure • Data • Infrastructure • Load or usage
38. 1. Whose fault is it?
39. 1. Triage
40. • Production code • Test code • Refactoring/structure • Data • Infrastructure • Load or usage
41. You need data
45. 2. Fix it
46. “Your best and wisest refuge from all troubles is in your science.” Ada Lovelace
49. Data is key
51. Tools for seeing more
54. Automation is your friend
58. Learn how to use your debugger!
62. Aim for fast feedback and easy reproduction
63. “99.9% of the time the best way to troubleshoot a test is to re-write it so it’s smaller and faster.” Israel Boza Rodriguez (@IsraKaos)
64. Fixed! ✅
65. Let’s talk about “unreliable test results”
66. I have made the decision to trust you Body text
67. 1. Find Your Flaky Tests
69. This is NOT a fix!
71. So now you can ignore those flaky tests
73. 2. Quarantine your flaky tests
74. 3. Fix your flaky tests!!!
75. If you add a new test and it’s flaky Or are working on a test and it becomes flaky Fix it now!!
76. 1. When they impact you directly 2. Attach them to related functionality 3. Flaky Test Days 4. Add flaky tests to the backlog 5. 20% time, tech debt time, cleanup time… Make Time To Fix Flaky Tests
77. 1. Environment, Network, and Resources 2. Integration Points 3. Concurrency, Asynchronous Programming, and Waiting 4. Setup/Teardown and Test Data 5. System Behaviour Causes of flakiness Known good practices and tools Not always in my control Try to minimise these Can be REALLY HARD
78. https://youtu.be/vHBzZHE4tJ0 1. Environment 2. Test Data 3. Versioning 4. Resource Use 5. System Behaviour
79. 3. Running Tests Takes Time
80. What slows down your testing?
81. How do you speed up your tests?
82. Use a Build Cache
83. Very effective for multi-module projects
84. No cache: Full build required
85. Build cache: Changing a leaf module
86. Build cache: Changing a module’s implementation
87. Build cache: Changing core implementation
88. Remote Build Cache
89. Run tests in parallel (local)
91. Run tests in parallel (CI)
93. Test Distribution
94. Test Distributor
95. Test Distribution Results ‑ ~50% ‑ ~50% ‑ ~50% Doubling the number of executors cuts build time in half
96. Let’s use AI!
98. “The test selection procedure selects fewer than a third of the tests that would be selected on the basis of build dependencies” Predictive Test Selection - Meta Research https://research.facebook.com/publications/predictive-test-selection/ Mateusz Machalica, Alex Samylkin, Meredith Porth, Satish Chandra
99. Predictive Test Selection time savings
101. What if we just make our tests faster?
102. Like any performance problem, focus on the worst-performers first
108. “Once things like databases become a hindrance, optimise the testing loop, mock things out to run things fast. You can run locally and you don't have all the same infra dependencies as someone else.” On DevEx, PaaS, and Testing in Production https://bit.ly/3ZA0uQM Kelsey Hightower
109. Look out for expensive setup and tear down
110. Invent Time Travel
111. Write Better Tests
113. ⬢ Separation of concerns ⬢ Encapsulation ⬢ High cohesion and loose coupling ⬢ SOLID ⬢ DRY ⬢ Descriptive naming ⬢ Modularisation ⬢ Designing for performance
114. Tension between these ideals
115. Refactor and Delete
116. “Yes, I regularly delete tests that no longer add unique value.” Angie Jones (@techgirl1908)
117. Summary
118. Automate as much as possible
119. Tooling can help you
120. There is a time investment in writing tests, because there should be thinking involved
121. Invest time in writing better tests. They will be easier to debug, less flaky, and better document your system’s expected behaviour
122. “Testing is not only for checking the system does what it’s supposed to, it’s for documenting the expected behaviour.” Israel Boza Rodriguez (@IsraKaos)
123. bit.ly/3MGAMnZ bit.ly/slow-tests Win a copy of Getting to Know IntelliJ IDEA!