Zero-bug policy success

Zero-bug policies are usually controversial, misunderstood, or both. Meanwhile, you probably live with a backlog of open bugs that you don’t expect to resolve any time soon. But you could. Zero-bug policies matter, because swarms of unfixed bugs waste time, hurt team morale, cause soul-destroying meetings, make development slow and unpredictable, and destroy customer trust.

This experience report from a Norwegian start-up in 2024 describes the journey from firefighting daily bug reports, to fixing the last open bug, and the joy of development with zero open bugs. Attendees will learn about the practical side of implementing a zero-bug policy, from introducing the idea, to achieving its goals. Our story includes the team dynamics of agreeing to give it a try, the objections and risks, and other implementation challenges. You’ll learn about why it’s worth the effort to fix (almost) all of the bugs, and which ones you can ignore.

Transcript

1. @hilton.org.uk http://hilton.org.uk/ Zero-bug policy success
2. Skal vi prioritere det? Fiks det nå! NEI FIKS DET ELLER KAST DET DET ULTIMATE BUGSYSTEMET Bør vi fikse det? JA JA NEI Kast! AV YASSAL SUNDMAN OVERSETTELSE AV MINA TAAJE FIXITNOWORDELETEIT.COM YDS.SE
3. Ska vi prioritera den? DET ULTIMATA BUGGHANTERINGSSYSTEMET Fixa nu! Borde vi fixa den? Kasta! AV YASSAL SUNDMAN FIXITNOWORDELETEIT.COM
4. Will we prioritize it? THE DEFINITIVE BUG MANAGEMENT SYSTEM Fix it now! Should we fix it? Delete it! BY YASSAL SUNDMAN FIXITNOWORDELETEIT.COM
5. Policies, systems & targets
6. Policies, targets & systems The notion of a zero-bug policy is sometimes controversial. Zero bugs is a target, not a policy. ‘Zero-bug policy’ is an inaccurate name for a bug management system for achieving a target. (naming things is hard !) 7 @PeterHilton •
7. Context - who, what, where
8. Context matters Most of what you hear and read about developing so!ware supposedly applies to… the whole industry " But as every consultant (and product manager) knows: it depends This is why we need experience reports 9 @PeterHilton •
9. Peter Ambar & Simon Gabi Nejat Ahmad Babbili Thierno Ruddy 6 developers 2 designers 1 product manager
13. Context Team Fully-remote 9 people + CTO So!ware Consumer retail Web shop Supply chain CRM + ERP Company 20 employees Mission-driven (sustainability) 14 @PeterHilton •
14. Getting started
15. The Joel Test: 12 Steps to Better Code (2000) 1. Do you use source control? 2. Can you make a build in one step? 3. Do you make daily builds? 4. Do you have a bug database? 5. Do you fix bugs before writing new code? 6. Do you have an up-to-date schedule? 7. Do you have a spec? 8. Do programmers have quiet working conditions? 9. Do you use the best tools money can buy? 10. Do you have testers? 11. Do new candidates write code during their interview? 12. Do you do hallway usability testing? 1. Do you use source control? 2. Can you make a build in one step? 3. Do you make daily builds? 4. Do you have a bug database? 5. Do you fix bugs before writing new code? # 6. Do you have an up-to-date schedule? 7. Do you have a spec? 8. Do programmers have quiet working conditions? 9. Do you use the best tools money can buy? 10. Do you have testers? 11. Do new candidates write code during their interview? 12. Do you do hallway usability testing?
17. From idea to adoption (2024) January Team retro idea to ‘Adopt a zero-bug policy’. Peter’s blog post shared on Slack. February Zero-bug policy team discussion. March My first day, discussed it again in the team retro. April Marketing campaign leads to many bug reports. 18 @PeterHilton •
18. Bugs hurt We suddenly spent too much time on $ support requests $. We struggled to keep track of what was broken. Discussing and prioritising bugs doesn’t ✨ spark joy ✨. We couldn’t conduct product experiments to improve customer activation (first purchase) when it didn’t work. 19 @PeterHilton •
20. The edge case discussion trap: what if… ? 1. … we have an existing bug backlog → fix them all! 2. … we didn’t fix the bug quickly → evaluate why not! 3. … the bug doesn’t affect customers → fix it anyway! 4. … the fix requires even more time → re-evaluate! 5. … we have multiple open bugs → prioritise! 6. … there’s a critical bug → interrupt other work! https://hilton.org.uk/blog/zero-bug-scenarios 21 @PeterHilton •
21. Goals Predictable effort to support and maintain a stable system. Stop wasting time discussing and prioritising open bugs. Meaningful product feedback from feature adoption data (because customers don’t use broken features). Developer happiness & 22 @PeterHilton •
22. Safe to try We realised that we had to experiment to find out, and that finding out had to be safe. So we adopted a zero-bug policy and started fixing bugs ' ' ' 23 @PeterHilton •
23. The bug workload
24. 0 50 100 150 200 250 300 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 253 reported 252 fixed ' Bug workload 2024
25. ' Bug workload 2024 spring 2024 marketing campaign bugs fixed per week Value Axis 0 5 10 15 20 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec open bugs 100% focus on fixing bugs 50% focus on fixing bugs
26. ‘Zero bugs policy pissed me off this week’ – team retro, August. 27 @PeterHilton •
27. spring 2024 marketing campaign bugs fixed per week Value Axis 0 5 10 15 20 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec open bugs ' Bug workload 2024 100% focus on fixing bugs 50% focus on fixing bugs zero-bug policy success zero-bug day no open bugs for the first time
28. bugs fixed Value Axis 0 5 10 15 20 Sep Oct Nov Dec Jan Feb Mar Apr May open bugs ' September 2024 – March 2025
29. Few critical bugs ' 161 bugs fixed from May 2024 – April 2025 ( 3 were critical bugs We only drop what we’re working on for critical bugs (normal bug reports are just next in the queue) We use team programming for critical bugs, because FOMO 30 @PeterHilton •
30. $ Deleted 9% ✅ Fixed 91% ' Reported bugs May 2024 – April 2025
31. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Mon Tue Wed Thu Fri Sat Sun & & ' ' ' & & & & & & & * * * * * & & & & & A P R I L 2 0 2 5
32. How we work 1. Slack #bugs channel → bug reports and updates # + ✅ 2. Weekly @product-support group rota for first responder 3. Notion board view → bug reports '*, 4. Separate Bug inbox board column (hidden when empty) 5. ‘Air-gapped’ separate product feedback database 33 @PeterHilton •
33. Using a bug tracker Not tracking bugs
34. Previously-answered questions
35. How do we define bugs vs other kinds of change? We don’t. We don’t need a written definition, because we evaluate one bug at a time. What counts is whether we’re going to fix it. You only need definitions for prioritisation games. 36 @PeterHilton •
37. What if we’d started with hundreds of open bugs? Then we’d have had to consider the ☢ nuclear option ☢ 38 @PeterHilton •
38. 39 @PeterHilton • DELETE YOUR BACKLOG
39. How do we prioritise for business stakeholders? As product manager, prioritisation is my responsibility . Other business stakeholders have never wanted to discuss bug prioritisation. If I did end up in a prioritisation discussion: I’d talk about business outcomes, such as customer sales, customer satisfaction, average order value, etc. 40 @PeterHilton •
40. Do we worry about slipping back into many bugs? Yes, a little, but less for each month it still doesn’t happen. I sometimes remind the team that we still fix bugs first. We avoid big releases that might cause many bug reports. Release feature flags decouple deployments from releases. 41 @PeterHilton •
41. Did we refactor, and improve testing, while fixing? Yes – much more unit test and integration test coverage. Better code reviews. Intermittent web shop checkout failures: root cause identified, redesigned to prevent inconsistent state on failure. 42 @PeterHilton •
42. If we delete a bug report, might a series of small bugs compound and become a big problem? Didn’t happen. A!er all, we only deleted 9% of bug reports. Deleting several related bug reports would be… unlikely. 43 @PeterHilton •
43. Lessons learned
44. Transition Fix It Now Or Delete It only works once you reach zero bugs. But we had no idea how long fixing all bugs would take. What if we’d had hundreds of open bugs to start with? Then we’d have had to consider the ☢ nuclear option ☢ 45 @PeterHilton •
45. When you only have one bug at a time, you can deal with them case-by-case. 46 @PeterHilton •
46. Why you can’t do this too Your team probably hasn’t got all of these: 1. enough pain that your team attributes to bugs, 2. the agency to delete the backlog, 3. a culture of quality that doesn’t accept buggy releases, 4. a product manager who understands that you can’t test product hypotheses with buggy releases. 47 @PeterHilton •
47. Zero-bug policy summary 1. Getting started required support from the whole team. 2. Our motivation came from the pain of managing bugs. 3. Everyone had questions, and wanted to overthink it. 4. A shared vision for a better future helped (maybe). 5. The fix-all-bugs transition phase was hard work. 6. Once we got zero bugs for the first time, it all got easier. 7. Now we worry less and talk less about bugs ' 49 @PeterHilton •
48. @hilton.org.uk http://hilton.org.uk/ https://hilton.org.uk/blog/zero-bug-policy