Continuous Delivery for Legacy Code (v3.1) 🇬🇧 @LeanAgile Scotland 2024

This is based on a true story.

My day job is software archeology. I find joy in recovering and analyzing code bones and culture as well as making the skeleton walk again. A short time ago, however, I was confronted with the most horrible code base I have ever seen. This talk is about how we managed to save it and achieve bi-weekly deployments with a high level of confidence.

Five million lines of code in multiple languages (Classic ASP, .NET, VBScript, VBA, JavaScript, T-SQL, PL-SQL) in one monolith. The business logic stretched from the UI (WebForms, Scripting, SQL Queries) down to the database (Stored Procedures), there was no test coverage and an enormous amount of hidden coupling. A version control system was not used, we had no test environment, deployments required developers to copy their local compilation to production and multiple customer installations are supported by uncommenting and commenting code.

Together we will explore what to do when you inherit such a thing: how to identify hotspots, find hidden coupling, explore how connascence can help you, ways to test as well as refactor and how to achieve a regular deployment schedule.

Transcript

1. 25.09.24 Richard Gross (he/him) Archaeology + Health Checks richargh.de/ speakerdeck.com/richargh @arghrich Hypermedia TestDSLs Continuous Delivery for Legacy Code
2. richargh.de Slide 2 by richargh.de from What are the properties of low- quality software?
3. richargh.de Slide 3 by richargh.de from Dave Farely, author of the best-selling book “Continuous Delivery” The Quality of a System is Defined by Our Ability to Change it! Dave Farley @davefarley77
4. richargh.de Slide 4 by richargh.de from Bad (legacy) software is software where we no longer have any options
5. richargh.de Slide 5 by richargh.de from A story of the worst software I have ever and you will never see. And we still got to Continuous Delivery ☺
6. richargh.de Slide 6 by richargh.de from Along comes a customer … Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ Please Health Check Customer Me
7. richargh.de Slide 7 by richargh.de from 4M LoC Delivery Excel Webservice Website Fat Client Jobs Database T-SQL JavaScript VBA C# Asp.NET + C# Classic Asp + VBScript WinForms + C# MIA 5M No Code delivered
8. richargh.de Slide 8 by richargh.de from Copy-Based Version ControlTM with production server as origin File tree from actual project
9. richargh.de Slide 9 by richargh.de from Cycle Time of 6 months weeks Dev Machine Cp from Prod Develop Compile Don‘t Test Release Cp to Prod 1 hour Bugfixing Return to Start 5 months 6 months
10. richargh.de Slide 10 by richargh.de from Too much to fix Icon by Franziska Haaf binarykitten.de Not Versioned & Doesn‘t Compile „Interesting“ Deployment Horrible Cycle Time No Tests Conflated Logic, UI & SQL 15 years of Uncontrolled growth
11. richargh.de Slide 11 by richargh.de from Icons by Franziska Haaf binarykitten.de COTS „Apocalypto“ In-house „Fenix“ Buy I‘ll Customer Money
12. richargh.de Slide 12 by richargh.de from Icons by Franziska Haaf binarykitten.de In-house „Fenix“ Please Fix Me Customer
13. Oh Boy
14. richargh.de Slide 14 by richargh.de from Ken Mugrage @kmugrage Legacy Code is code that’s too scary to update and too profitable to delete. It‘s not CD if you can‘t deploy to production right now Dylan Beattie @dylanbeattie
15. richargh.de Slide 15 by richargh.de from Can we deploy this to production right now? Nope
16. richargh.de Slide 16 by richargh.de from Icon by Franziska Haaf binarykitten.de 15 years of Uncontrolled growth
17. richargh.de Slide 17 by richargh.de from Where would I even start? If I’m lucky I have a long list of issues: • Issue 1 in file x • Issue 2 in file y • Issue 3 in file z • Issue … • Issue … • Issue … • Issue … • Issue … • Issue … • Issue … • Issue … • Issue … • Issue … • Issue 9001
18. richargh.de Slide 18 by richargh.de from Don’t Fix everything Everyhwere All at once
19. richargh.de Slide 19 by richargh.de from Map your code. Extract your insights. Master your Legacy.
20. richargh.de Slide 20 by richargh.de from Art by Zaufishan
21. richargh.de Slide 21 by richargh.de from Visualize metrics with CodeCharta buildings Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ SomeService.kt Lines of Code f.ex. Complexity f.ex. Number of authors
22. richargh.de Slide 22 by richargh.de from Find potential hotspots Open-Source Tool CodeCharta: https://maibornwolff.github.io/codecharta/ Lot‘s of code, very complex, and everyone has to change it
23. richargh.de Slide 23 by richargh.de from We know what to focus on Not Versioned & Doesn‘t Compile
24. richargh.de Slide 24 by richargh.de from Put everything into version control • Code • Configuration • Deployment scripts • Infrastructure Provisioning • Alerting • Monitoring • Documentation • Onboarding.sh Scripts • … Git
25. richargh.de Slide 25 by richargh.de from Establish a Pipeline Commit Stage Compile Git
26. richargh.de Slide 26 by richargh.de from We know what to focus on Versioned & Automated Compilation „Interesting“ Deployment
27. richargh.de Slide 27 by richargh.de from Scripts to Provision Infrastructure Same but parameterized Scripts for TST, STG and PRD • Machine • Web Server • Application Server • Database • Email Server • FTP Server • Pipeline Server Phoenix Infra: With one click we can (re)create any infrastructure Gives us
28. richargh.de Slide 28 by richargh.de from Deploy via Pipeline to Test Env Commit Stage Compile Git Deploy To TST (Cont.) STG (Click) Smoke Test
29. richargh.de Slide 29 by richargh.de from Can we deploy this to production right now? We have a button to do it, but do we want to?
30. richargh.de Slide 30 by richargh.de from No Tests Conflated Logic, UI & SQL We know what to focus on Versioned & Automated Compilation „Interesting“ Deployment
31. richargh.de Slide 31 by richargh.de from Characterization test1: Write Tests for code that you don’t understand 1 Also called approval or golden master test. 2 Put system in a known (database) state is also an input. 3 Log messages you add are also an output. vX - redesigned v1 - legacy Input for UI/API/DB 2 Output 3 Output 3 Continuous Compare
32. richargh.de Slide 32 by richargh.de from Start with tests for critical use cases and hotspots • ~10 E2E Tests For critical use cases • DB-Integration Tests For direct-db-access hotspots that you will refactor • Unit tests (should give 80% confidence) For refactored hotspots For no-direct-db-access hotspots For new code
33. richargh.de Slide 33 by richargh.de from The tests guide the way 1. Establish a test dsl1,2 ( the easy part) 2. Try to unit test legacy element and … fail 3. Realize that coupling prevents you from proper testing 4. Write a too-big integration test 5. Fix coupling problems ( the hard part) 6. Use Dsl to switch integration to unit test 1 https://github.com/Richargh/testdsl 2 Structure-cementing tests and how to avoid them (in German )
34. richargh.de Slide 34 by richargh.de from Understand Coupling and Cohesion High Coupling Low Cohesion Low Coupling High Cohesion Unknown source
35. richargh.de Slide 35 by richargh.de from Uncouple your Code 1 https://www.maibornwolff.de/en/know-how/strangler-fig-pattern/ Delete Obsolete (unreachable or unused) Code Characterization Test, then Refactor Code Sprout new tested Code Break Dependencies (Seams) Strangler Fig1 Unfixable code Feature Flag Legacy Code Write ArchUnit Tests
36. richargh.de Slide 36 by richargh.de from Can we deploy this to production right now? Hmm, it’s somewhat tested but can we really be sure nothing breaks in production?
37. richargh.de Slide 37 by richargh.de from Separated Logic, UI & Data in Hotspots Critical Use Case Tests Growing Unit Tests Unit tested new code Horrible Cycle Time We know what to focus on Versioned & Automated Compilation „Interesting“ Deployment
38. richargh.de Slide 38 by richargh.de from Do Monitor in Production
39. richargh.de Slide 39 by richargh.de from RED Metrics 1 https://thenewstack.io/monitoring-microservices-red-method/ Community Dashboard https://community.grafana.com/t/how-can-i-configure-red-dashboard/69885 R -> Request Rate E -> Errors / second D -> Duration per request
40. richargh.de Slide 40 by richargh.de from New Cycle Time, without bug-cycle Slide adapted from Dave Farley Commit Stage Compile Unit Test Lint Security Docker Release Git Monitor Logs Events Metrics Alerts Automated Performance testing 20 min Deploy To TST (Cont.) STG (Click) Smoke Test <1 min Manual Testing 30 min Automated Acceptance Testing User Acceptance Test DB-Integration Test 20 min 5 min 60 min
41. richargh.de Slide 42 by richargh.de from Can we deploy this to production right now? Yes. Depending on where we make the change, even within 30min.
42. richargh.de Slide 43 by richargh.de from But there is still a problem…
43. richargh.de Slide 44 by richargh.de from How did the previous developers get here?
44. richargh.de Slide 45 by richargh.de from Healthy Software Ideas Product Process Culture Predicts
45. richargh.de Slide 46 by richargh.de from GIGO Ideas Product Process Culture Predicts
46. richargh.de Slide 47 by richargh.de from A Feature Factory helps no one https://cutlefish.substack.com/p/tbm-4952-your-calendar-your-priorities
47. richargh.de Slide 48 by richargh.de from Henrik Kniberg @henrikkniberg Culture is the sum of everyone's attitude and actions, so model the behaviour you want to see! Observe behavior: • Who does the management promote? • Is there fear of failure? • Is quality appreciated or story points or …? • What is the delivery „process“? • Is there some kind of learning exchange?
48. richargh.de Slide 49 by richargh.de from Make the desired behavior … Obvious • Define Principles together • Shared purpose Attractive • Reduce (time) pressure • (Refactoring) score Atomic Habits 4 Laws of behavior change Easy • Tech-coach • Reduce friction Satisfying • Pair up • Never break the chain (twice)
49. richargh.de Slide 50 by richargh.de from CD is a change in mindset
50. richargh.de Slide 51 by richargh.de from CD = Working in a way that software is always in a releasable state It’s not about the tools, even though they are getting better all the time
51. richargh.de Slide 52 by richargh.de from Separated logic, UI & data in hotspots Critical E2E Tests; Growing unit test suite; New code always unit tested We know what to focus on Versioned & Automated Compilation „Interesting“ Deployment Nice Cycle Time In case you need some help here…
52. richargh.de Slide 53 by richargh.de from Code Quality Insights Scan to get your insights
53. richargh.de Slide 54 by richargh.de from Thank you @arghrich Richard Gross (he/him) IT Archaeology + Health Checks Hypermedia TestDSLs Works for maibornwolff.de/ Slides, Code, Videos Contact me <firstname>.<lastname> at maibornwolff de Contact Ask me about these topics ☺ richargh.de/ rigross Code Quality Insights
54. richargh.de Slide 55 by richargh.de from Backup
55. richargh.de Slide 56 by richargh.de from
56. richargh.de Slide 57 by richargh.de from 10 customer installations Frobozz Co. Oceanic Airlines … Umbrella Corp Soylent Corp
57. richargh.de Slide 58 by richargh.de from And also 10 separate codebases … Oceanic Airlines Soylent Corp Frobozz Co. Umbrella Corp Similar C/P Copy/ Paste Feature Copy/Paste
58. richargh.de Slide 59 by richargh.de from 10 codebases And 10 database schemata Similar Similar
59. richargh.de Slide 60 by richargh.de from Our Legacy CD Principles • Build quality in • Focus on current tree, not on the forest • Tackle fear head on • Focus on reducing batch size • Decouple Deployment and Release • Computers perform repetitive tasks, people solve problems CD Principles by Jez Humble Four Principles of Low-Risk Software Releases by Jez Humble
60. richargh.de Slide 61 by richargh.de from Our branching strategy • Don‘t Branch • Don‘t Even Branch • Branch and you’re screwed • Never Branch to Refactor • Branches indicate Ci =Continuous Isolation
61. richargh.de Slide 62 by richargh.de from The True Cost of Rewrites Original Article by Doug Bradbury The True Cost of Rewrites Features Releases over time Catch Up Missing Features Sub-Par Parity Enhancements Planned Rewrite Actual Features Old App Adoption