Technical Debt - The code monster in your closet - PyParis 2018

PyParis - November 14, 2018

Technical debt is the code monster hiding in every Python programmer’s closet. If you ignore it, it will terrorize you at night.

I’ve worked at many institutions with many programming languages over the past 12 years. They all have technical debt. Putting on a band-aid and ignoring the real issues can be disastrous. We’ll go through several case studies, review big red flags, and learn how to start chipping away at the problem with confidence.

Transcript

1. PyParis 2018 Nina Zakharenko @nnja bit.ly/ParisTechDebt
2. Livetweet! use #PyParis @nnja
3. Technical Debt The code monster in your closet slides: bit.ly/ParisTechDebt @nnja
4. What is technical debt? @nnja
5. A series of bad decisions (Both business & technical) @nnja
6. Which lead to -> Error prone code & architecture @nnja
7. ... and using more Resources to accomplish Less @nnja
8. What decisions were made in the past that prevent me from getting sh** done today? @nnja
9. What causes technical debt? @nnja
10. Me. And you. @nnja
11. Mistakes I Made Early On 4 Not seeing the value in unit tests 4 Not knowing how to say NO to features @nnja
12. Mistakes I Made Early On 4 Overly optimistic estimates 4 Putting releases over good design & reusable code @nnja
13. Time Crunch That project was due yesterday! I'll take a shortcut, and clean up the mess tomorrow. @nnja
14. Unneeded Complexity Lines of code committed != amount of work accomplished @nnja
15. Lack of understanding 1. Have a problem 2. Look up a solution on stackoverflow 3. Copy & paste it into your code 4. ??? 5. Bugs! @nnja
16. Culture of Despair This is already a heap of trash. Will anyone really notice if I add one more thing to the top? @nnja
17. Red Flags Houston, we have a problem. @nnja
18. Code Smells 4 Not Bugs 4 An indication of a deeper problem @nnja
19. Code Smells 4 Half implemented features 4 No documentation, or poor documentation @nnja
20. Code Smells 4 Commented out code 4 Incorrect comments 4 No tests, or worse: broken tests @nnja
21. Restore deleted code with git! Find by content: $ git log --summary -G'(D|d)jango' Find the commit that deleted a file: ```shell git log --diff-filter=D --summary -- <filename> @nnja
22. No more commented out code! @nnja
23. Poor Documentation class OrganicGlutenFreePizzaFactory: def get_dough(self): """ Return amazing, organic, GMO and Gluten Free Dough """ # ran out of organic gluten free, use the other stuff. # return 'organic gluten free dough' return 'gmo pesticide processed gluten-full dough' @nnja
24. Architecture & Design... Smells 4 Parts of the code no one wants to touch 4 Brittle codebase -- changing code in one area breaks other parts of the system 4 Severe outages caused by frequent & unexpected bugs @nnja
25. Good Design -> Implementing new features comes easily Poor Design -> New features are shoe- horned into the system @nnja
26. Python Specific @nnja
27. Functionality changes, but variable names don't employees = ['John', 'Mary', 'Dale'] employees = 'Bob' employees[0] @nnja
28. Monkey Patching !" def new_init(self): pass some_library.SomeClass.__init__ = new_init @nnja
29. What exactly does this decorator do? def decorator_evil(func): return False @decorator_evil def target(a,b): return a + b >>> target(1,2) TypeError: 'bool' object is not callable >>> target False
30. Circular Dependencies # Circumvent circular dependency warnings def some_function(x): from some.module import some_method some_method(x) @nnja
31. Case Studies @nnja
32. IRS Chief: "We still have applications that were running when JFK was President" Tech at the IRS
33. 50 Year Old Technology "And we continue to use the COBOL programming language, it is extremely difficult to find IT experts who are versed in this language." @nnja
34. It's not just the IRS 4 Banks & Financial Institutions 4 Universities 4 Air Traffic Control 4 ... many still use COBOL @nnja
35. Story Time 4 I used to work in finance. 4 At the time I was there, all of the banking systems were run on mainframes. 4 The bankers were getting frustrated. They wanted a UI. @nnja
36. Big Idea! 4 Let’s write a fancy new web front end 4 It’ll do ALL the things @nnja
37. But 4 Rewriting the backend is too expensive 4 It already does what we need 4 Let's leave the mainframe as the backend @nnja
38. Cursors 4 The mainframe would output a text screen from a program result, based on a query. 4 The results would be parsed by reading variables from the screen in certain positions. @nnja
39. Result? 4 The new system was incredibly slow 4 And error prone 4 After months of work, the multi-million dollar rewrite was scrapped @nnja
40. You can try to cover up debt... (but it probably won't work) @nnja
41. The MVP 4 (Minimum Viable Product) 4 Get the product to market as soon as possible @nnja
42. A Great Idea 4 A successful project that was created by a lone developer in a coffee fueled 48 hours. @nnja
43. There Was a Problem 4 Years went on, but the initial code and design didn’t go away. 4 Instead, it became the base for an expanding project, with expanding features. 4 There was never any time to refactor. @nnja
44. !"# @nnja
45. Scope Creep 4 Features that someone thought was a good idea one day, stuck around forever. 4 > “In case we need them. Later.” @nnja
46. Sad Developers 4 Minimal working tests (no time to write them). 4 When a release was pushed, something was bound to break. 4 Made everything feel like it was your fault. @nnja
47. Grinding To a Halt 4 Development time for new features skyrocketed 4 The project was deemed too difficult to maintain 4 ... and cancelled. @nnja
48. Sometimes you need to burn it. With fire. @nnja
49. Battling The Monster @nnja
50. Don't point fingers Technical debt is a team-wide problem. Everybody needs to be part of the solution. @nnja
51. Work Together 4 Code Standards 4 Pair Programming 4 Code Reviews @nnja
52. Unless something is on fire, or you’re losing money, don't merge unreviewed code into master. @nnja
53. Be Accountable 4 Unit & Integration Tests 4 Pre-Commit Hooks 4 Continuous Integration @nnja
54. Make a Commitment Company tried to fight debt, but they didn't make a commitment. @nnja
55. Ended up with twice as many technologies in their stack as needed, and twice as big of a mess. @nnja
56. Sell It To Decision Makers By allocating project time to tackling debt, the end result will be less error prone, easier to maintain, and easier to add features to. @nnja
57. Not broken, why fix it? Source
58. Ski Rental Problem You’re going skiing for an unknown number of days. It costs $1 a day to rent, or $20 to buy. Source
59. Hiring developers is hard. Technical debt frustrates developers. Frustrated developers are more likely to leave. @nnja
60. Some lingering debt is inevitable. Don't be a perfectionist. Figure out the project tolerance, and work with it. @nnja
61. Use these arguments to justify the additional time it takes to do things right @nnja
62. To Win The Fight, Pay Down Your Debt @nnja
63. Refactoring The single greatest tool in your toolbox @nnja
64. What is it? Systematically changing the code without changing functionality, while improving design and readability. @nnja
65. Refactoring 4 Slow and steady wins the race. 4 The end goal is to refactor without breaking existing functionality. @nnja
66. Refactoring 4 Replace functions and modules incrementally. 4 Test as you go. 4 Tests are mandatory at this step. @nnja
67. github.com/Yelp/undebt, yelp refactoring
68. Use proper design patterns github.com/faif/python-patterns
69. Use depreciation patterns Like openstack debtcollector class removed_property(object): """Property descriptor that deprecates a property. This works like the ``@property`` descriptor but can be used instead to provide the same functionality and also interact with the :mod:`warnings`module to warn when a property is accessed, set and/or deleted. """ @nnja
70. Use vulture.py to find dead or unreachable code $ pip install vulture $ vulture script.py package/ or $ python -m vulture script.py package/ github.com/jendrikseipp/vulture
71. sample code def foo(): print("foo") def bar(): print("bar") def baz(): print("baz") foo() bar() vulture.py output › python -m vulture foo.py foo.py:7: unused function 'baz' (60% confidence)
72. Prioritize What causes the biggest & most frequent pain points for developers? @nnja
73. Just like with monetary debt, pay off the high interest loan first. @nnja
74. Shelf Life What's the life expectancy of this project? Longer shelf life -> higher debt interest @nnja
75. Technical debt can be strategic If you don't have to pay it off, you got something for nothing. @nnja
76. Making time for refactoring depends on the size of your team, and the size of your problem. @nnja
77. Guidelines 4 Small 4 Devote a week every 6-8 weeks 4 Medium 4 Devote a person every 1-4 weeks, rotate 4 Large @nnja
78. A Few Last Tips @nnja
79. Code should be for humans @nnja
80. Boy Scout Rule "Always check in a module cleaner than when you checked it out." Source
81. Expect To Be Frustrated The process of cleaning up days / months / years of bad code can be analogous with untangling a ball of yarn. Don't give up. @nnja
83. Thank You! Python @ Microsoft: bit.ly/parispython @nnja @nnja