.NET Day 2024: 100% Unit Test Coverage and beyond

More info at: https://dotnetday.ch

Transcript

1. 100% Unit Test Coverage and beyond Marc Sallin 27 August 2024
2. Speaker Marc Sallin Solution Architect marc.sallin@swisspost.ch 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 2 Swiss Post Ltd I/T, IT20.2 Wankdorfallee 4, 3014 Bern swisspost.ch
3. Swiss Post Parcel Sorting Daillens Vétroz Härkingen PZ & BBZ Urdorf Frauenfeld Untervaz Cadenazzo Pratteln Ostermundigen Rümlang Wallisellen Buchs 1.5 Years 3 Centers 6 Million Parcels 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 3
4. Some Numbers ~5k Deployments 0 Regressions 1 / 1,5h Change Failure & TTR ~1.5d Commit Lead Time 18 Dev & Arch 99.9% Unit Test Coverage 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 4
5. 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 5 “80% Coverage is enough.” Too much people in our industry..
6. 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 6 Dear Software Engineers, we need to talk...
7. Software Engineering “The systematic application of scientific and technological knowledge, methods, and experience to the design, implementation, testing, and documentation of software.” IEEE Standard Glossary of Software Engineering Terminology 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 7
8. Fallacy 1: Nirvana fallacy or the danger of false dichotomies Premise: Tests do not proof the absence of defects It follows: Even 100% test coverage doesn’t proof the absence of defects Conclusion: it follows that 100% isn’t worth it 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 8
9. Fallacy 2: There is a threshold 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 9 𝑓 𝑥 = 2(1 − 𝑒−2𝑥)
10. Fallacy 3: BUT not all areas are important 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 10 Premise: Not all areas are important to test It follows: We not need to test all areas Conclusion: We only cover 80% To make the conclusion’s true, the following assumptions have to be true: Assumption I: Important areas are objectively definable Assumption II: The ration of important vs. unimportant areas stays the same
11. Fallacy 4: We just need End-2-End 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 11
12. Fallacy 5: Tests are to expensive and we have no time 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 12 https://martinfowler.com/bliki/DesignStaminaHypothesis.html (26.08.2024)
13. Fallacy 6: Test shouldn’t be written by devs 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 13 https://builtin.com/software-engineering-perspectives/v-model (28.08.2024)
14. Beyond 100%... But you accepted that tests do not guarantee functional correctness! Can we do better? 2. September 2024 Die Schweizerische Post – Titel der Präsentation – Vorname Name – C1 Öffentlich | C2 Intern | C3 Eingeschränkt | C4 Vertraulich (Fusszeile ändern über «Einfügen > Kopf- und Fusszeile») 14
15. How many Tests? 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 15
16. Still missing... Startup Flows Contracts Test Quality
17. Startup – IoC Container Validation 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 17
18. Flows – Integration Tests 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 18
19. Test Quality - Mutation Testing 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 19
20. Contracts – Contract Testing 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 20
21. Exclude from Coverage! 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 21
22. Conclusion 100% coverage is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning.
23. Thank You! 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 23
24. Danke, Merci, Grazie, Thank you 27. August 2024 Swiss Post – 100% Unit Test Coverage – Marc Sallin – C1 Public 24