Averting Tragedy on the Architectural Commons (ScanDev 2013)

Much has been made of the need to establish software architectures to provide the firm foundations on which successful products are constructed, but if a product is successful over the long term its architecture will not only need to evolve, but must be actively defended against malignant forces. The fact that software architectures tend to outlive the tenure of developers, architects, management teams and even companies makes the maintenance of software architectures over the long term crucial for the ability of products to deliver ongoing value. What happens to the architecture of established systems in an environment where concurrent feature delivery projects make architectural ‘adjustments’ for their own ends? How can erosion of this shared architecture be managed across multiple teams? How can projects acting in their own best interests avoid over-exploiting the common architectural resource on which they all depend? Should ownership of the architecture be distributed and shared, or centralised and tightly controlled? Perhaps most importantly, can we detect when the architecture has been violated? This talk explores how shared resources in other fields are managed for the common good, and draws analogies and lessons which can be applied to the shared ‘resource’ of a software architecture. Examples garnered from over twelve years working with greenfield and legacy software systems illustrate how to diagnose, understand the causes of, and address the erosion of application architectures so that products can flourish and be productive for future generations.

Transcript

1. Avoiding Tragedy on the Architectural Commons 1 Robert Smallshire Roxar Software Solutions, Oslo @robsmallshire Tuesday, 5 March, 13
2. Systems and their architectures are long lived Lifetimes in the software industry 2 Developers Windows XP Applications CEOs Lines of code FTSE100 Classes Modules 0 15 30 45 60 58 37 22 13 6.8 6.2 4.7 3.1 Sources: W3schools, Software Lifetime and its Evolution Process over Generations, CEO Succession Practices: 2012 Edition, Investors Chronicle, Half-lives of software related entities The number of years over which half the entities are replaced Tuesday, 5 March, 13
3. 3 “Farming looks mighty easy when your plow is a pencil and you're a thousand miles from the corn field.” Dwight Eisenhower Tuesday, 5 March, 13
4. 4 Tuesday, 5 March, 13
5. 5 Sustainable Development Provision meets ongoing needs whilst preserving the supporting environment. Tuesday, 5 March, 13
6. 6 Tuesday, 5 March, 13
7. 7 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognise and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Tuesday, 5 March, 13
8. Garrett Hardin 8 “Resources held for common use by all will ultimately be destroyed.” Hardin, G. (1968) The Tragedy of the Commons. Science 162 1243-1248 “There is no technical solution.” Tuesday, 5 March, 13
9. 9 Tuesday, 5 March, 13
10. 10 Tuesday, 5 March, 13
11. Potential and actual tragedies The Commons 11 Global Climate Fisheries Groundwater Grazing Habitat Forests Radio Spectrum Atmosphere The Internet Roads Parks Museums Silence Software Systems Welfare State Tuesday, 5 March, 13
12. 12 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognise and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Tuesday, 5 March, 13
13. What takes us from this ... How does this happen? 13 Tuesday, 5 March, 13
14. ... to this? How does this happen? 14 Tuesday, 5 March, 13
15. The tragedy unfolds inexorably, one dependency at a time. How does this happen? 15 using App.Foo.Bar import App.Foo.Bar from App.Foo import Bar #include <app/foo/bar.h> require “app/foo/bar.h” Tuesday, 5 March, 13
16. has second order effect on outcomes Organization 16 Command and Control Self Organising Tuesday, 5 March, 13
17. Large numbers of smart people - does smartness scale? 17 Organization Tuesday, 5 March, 13
18. Smartness is not an additive quantity. 18 Organization Tuesday, 5 March, 13
19. 100 1000 10000 100000 1000 10000 100000 1000000 10000000 Productivity (Lines of Code / Year) Total Lines of Code Use published productivity data to forward model code size. Modelling team and code evolution 19 Sources: COCOMO II At any given system size we can predict a distribution for developer productivity. Tuesday, 5 March, 13
20. 20 3 years 54 kLoC Overall Employees : 4 Team Size : 2 LoC : 54 k Author present : 60% Tuesday, 5 March, 13
21. 21 3 years 152 kLoC Cumulative team size : 11 ± 2 @ 1σ Team Size : 7 LoC : 157 k ± 23 k @ 1σ Author present : 70% ± 14% @ 1σ Tuesday, 5 March, 13
22. 22 20 years 1.8 MLoC Cumulative team size : 114 ± 9 @ 1σ Team Size : 21 LoC : 1.8 M ± 0.08 M @ 1σ Author present : 19% ± 4% @ 1σ Tuesday, 5 March, 13
23. Most authors of your product quit way back when. Who can you still talk to? 23 days Proportion of code written by current colleagues. 20% after 20 years Tuesday, 5 March, 13
24. Step back and take in the big picture Detecting compromised architectures 24 Tuesday, 5 March, 13
25. 25 Tuesday, 5 March, 13
26. How is complexity distributed in your system? Mapping complexity 26 Tuesday, 5 March, 13
27. How are defects distributed in your system? Mapping quality 27 Tuesday, 5 March, 13
28. Do you understand your actual dependencies? 28 Dependency Structure Tuesday, 5 March, 13
29. 29 Tuesday, 5 March, 13
30. 30 Tuesday, 5 March, 13
31. Reap all the immediate benefit but share the deferred cost 31 Introduced dependencies Tuesday, 5 March, 13
32. 32 Tuesday, 5 March, 13
33. 33 “any situation in which one person makes the decision about how much risk to take, while someone else bears the cost if things go badly” Paul Krugman, Economist Tuesday, 5 March, 13
34. 34 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognize and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Tuesday, 5 March, 13
35. We have met the enemy and he is us. Tragedy on the Architectural Commons 35 Short Tenure Developer turnover outpaces change of the software system. Moral Hazard Developers acting rationally for local gain aren’t exposed to the deferred and global cost of their actions. The Tragedy The common environment of the software architecture erodes, making ongoing development more frustrating and costly. Productivity falls. 1 2 3 Tragic Cycle Tuesday, 5 March, 13
36. Often used in combination Solutions to the tragedy 36 Education Encourage ‘enlightened self-interest’. Foster cooperation through the architecture. Privatisation Create a ‘core’ team who own the architecture. Does this confuse architecture with infrastructure? Taxation Tax feature projects by requiring them to contribute beyond their narrow project scope. Regulation Give clear ownership to a central authority. Introduce regulating software. Tuesday, 5 March, 13
37. Education 37 ‣ Enlightened self interest • “do well by doing good” ‣ Deferred gratification • sacrifice short-term interests in favour of long-term interests Tuesday, 5 March, 13
38. The code is the design? “most complete, unambiguous, and irreducible expression of a design” 38 The code is the most, Maybe. But only the de facto design, not the intended design. Tuesday, 5 March, 13
39. Privatisation 39 ‣ Transfer ownership of the architecture to a single organisation • ownership obligates stewardship • acts on behalf of feature teams, who must rent or purchase services • does this confuse architecture with infrastructure? Tuesday, 5 March, 13
40. 40 Creating a core team who owns the architecture usually results in another feature team. It’s hard to maintain stakeholder support for infrastructure or design artefacts. Tuesday, 5 March, 13
41. Taxation 41 ‣ Introduce transaction costs • attach an immediate cost to potentially detrimental changes • changes reflect their long-term, shared cost • discourage frivolous or merely convenient dependencies Tuesday, 5 March, 13
42. 42 Not all changes should be treated equally Tuesday, 5 March, 13
43. 43 Not all changes should be treated equally Have new dependencies require review from a wider constituency Tuesday, 5 March, 13
44. Regulation 44 ‣ Central authority • architects are easy to ignore • architects can’t be omniscient • architects should encode design rules in such a way that they can’t be ignored Tuesday, 5 March, 13
45. 45 Encode design constraints into automated systems: • vigilant build systems • commit triggers • custom static analysis tools Tuesday, 5 March, 13
46. 46 The microeconomics of change By understanding the incentives which drive architectural erosion we can avoid tragic decline. Tuesday, 5 March, 13
47. Dependencies and complexity are too cheap in the short term Confront the moral hazard 47 Short Tenure Developer turnover outpaces change of the software system. Moral Hazard Developers acting rationally for local gain aren’t exposed to the deferred cost of their actions, of technical debt. The Tragedy The common environment of the system erodes, making ongoing development more frustrating and costly. Productivity falls. 1 2 3 Tragic Cycle Tuesday, 5 March, 13
48. 48 Tuesday, 5 March, 13
49. Thank you Questions? 49 Robert Smallshire Roxar Software Solutions, Oslo @robsmallshire starting in May Tuesday, 5 March, 13