Averting Tragedy on the Architectural Commons (JavaZone 2012)

Much has been made of the need to establish software architectures to provide the firm foundations on which successful products are constructed, but if a product is successful over the long term its architecture will not only need to evolve, but must be actively defended against malignant forces. The fact that software architectures tend to outlive the tenure of developers, architects, management teams and even companies makes the maintenance of software architectures over the long term crucial for the ability of products to deliver ongoing value.

What happens to the architecture of established systems in an environment where concurrent feature delivery projects make architectural ‘adjustments’ for their own ends? How can erosion of this shared architecture be managed across multiple teams? How can projects acting in their own best interests avoid over-exploiting the common architectural resource on which they all depend? Should ownership of the architecture be distributed and shared, or centralised and tightly controlled? Perhaps most importantly, can we detect when the architecture has been violated?

This talk explores how shared resources in other fields are managed for the common good, and draws analogies and lessons which can be applied to the shared ‘resource’ of a software architecture. Examples garnered from over twelve years working with greenfield and legacy software systems illustrate how to diagnose, understand the causes of, and address the erosion of application architectures so that products can flourish and be productive for future generations.

Transcript

1. Avoiding Tragedy on the Architectural Commons 1 Robert Smallshire Roxar Software Solutions, Oslo @robsmallshire Thursday, 13 September, 12
2. Systems and their architectures are long lived Lifetimes in the software industry 2 Developers Windows XP Applications CEOs Lines of code FTSE100 Classes Modules 0 15 30 45 60 58 37 22 13 6.8 6.2 4.7 3.1 Sources: W3schools, Software Lifetime and its Evolution Process over Generations, CEO Succession Practices: 2012 Edition, Investors Chronicle, Half-lives of software related entities The number of years over which half the entities are replaced Thursday, 13 September, 12
3. 3 Thursday, 13 September, 12
4. 3 “Farming looks mighty easy when your plow is a pencil and you're a thousand miles from the corn field.” Dwight Eisenhower Thursday, 13 September, 12
5. 4 Thursday, 13 September, 12
6. 5 Sustainable Development Provision meets ongoing needs whilst preserving the supporting environment. Thursday, 13 September, 12
7. 6 Thursday, 13 September, 12
8. 7 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognise and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Thursday, 13 September, 12
9. Garrett Hardin 8 “Resources held for common use by all will ultimately be destroyed.” Hardin, G. (1968) The Tragedy of the Commons. Science 162 1243-1248 Thursday, 13 September, 12
10. Garrett Hardin 8 “Resources held for common use by all will ultimately be destroyed.” Hardin, G. (1968) The Tragedy of the Commons. Science 162 1243-1248 “There is no technical solution.” Thursday, 13 September, 12
11. 9 Thursday, 13 September, 12
12. 9 Thursday, 13 September, 12
13. 9 Thursday, 13 September, 12
14. 9 Thursday, 13 September, 12
15. 9 Thursday, 13 September, 12
16. 9 Thursday, 13 September, 12
17. 9 Thursday, 13 September, 12
18. 10 Thursday, 13 September, 12
19. 11 Thursday, 13 September, 12
20. 11 4      047 880 visitors to Yosemite National Park in 2010. Thursday, 13 September, 12
21. 12 Thursday, 13 September, 12
22. 13 Thursday, 13 September, 12
23. 14 Thursday, 13 September, 12
24. 15 Collapse of Newfoundland Cod Stocks Thursday, 13 September, 12
25. Potential and actual tragedies The Commons 16 Global Climate Fisheries Groundwater Grazing Habitat Forests Radio Spectrum Atmosphere The Internet Roads Parks Museums Silence Welfare State Thursday, 13 September, 12
26. Potential and actual tragedies The Commons 16 Global Climate Fisheries Groundwater Grazing Habitat Forests Radio Spectrum Atmosphere The Internet Roads Parks Museums Silence Software Architectures Welfare State Thursday, 13 September, 12
27. 17 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognise and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Thursday, 13 September, 12
28. What takes us from this ... How does this happen? 18 Thursday, 13 September, 12
29. ... to this? How does this happen? 19 Thursday, 13 September, 12
30. The tragedy unfolds inexorably, one dependency at a time. How does this happen? 20 Thursday, 13 September, 12
31. The tragedy unfolds inexorably, one dependency at a time. How does this happen? 20 using App.Foo.Bar import App.Foo.Bar from App.Foo import Bar #include <app/foo/bar.h> require “app/foo/bar.h” Thursday, 13 September, 12
32. has second order effect on outcomes Organization 21 Command and Control Self Organising Thursday, 13 September, 12
33. Large numbers of smart people - does smartness scale? 22 Organization Thursday, 13 September, 12
34. Large numbers of smart people - does smartness scale? 22 Organization Thursday, 13 September, 12
35. Smartness is not an additive quantity. 23 Organization Thursday, 13 September, 12
36. Reap all the immediate benefit but share the deferred cost 24 Introduced dependencies Thursday, 13 September, 12
37. Reap all the immediate benefit but share the deferred cost 24 Introduced dependencies Thursday, 13 September, 12
38. 25 Thursday, 13 September, 12
39. 26 “any situation in which one person makes the decision about how much risk to take, while someone else bears the cost if things go badly” Paul Krugman, Economist Thursday, 13 September, 12
40. Step back and take in the big picture Detecting compromised architectures 27 Thursday, 13 September, 12
41. 28 Thursday, 13 September, 12
42. 28 Thursday, 13 September, 12
43. How is complexity distributed in your system? Mapping complexity 29 Thursday, 13 September, 12
44. How are defects distributed in your system? Mapping quality 30 Thursday, 13 September, 12
45. How are defects distributed in your system? Mapping quality 30 Thursday, 13 September, 12
46. Do you understand your actual dependencies? 31 Dependency Structure Thursday, 13 September, 12
47. 32 Thursday, 13 September, 12
48. 33 Thursday, 13 September, 12
49. 34 The tragedy of the commons What is the tragedy and how does it relate to software architecture? The causes and effects of architectural erosion How to recognize and detect compromised architectures. Defending the architecture Managing the architecture and its stakeholders. 1 2 3 Thursday, 13 September, 12
50. We have met the enemy and he is us. Tragedy on the Architectural Commons 35 Short Tenure Developer turnover outpaces change of the software system. Moral Hazard Developers acting rationally for local gain aren’t exposed to the deferred and global cost of their actions. The Tragedy The common environment of the software architecture erodes, making ongoing development more frustrating and costly. Productivity falls. 1 2 3 Tragic Cycle Thursday, 13 September, 12
51. Often used in combination Solutions to the tragedy 36 Education Encourage ‘enlightened self-interest’. Foster cooperation through the architecture. Privatisation Create a ‘core’ team who own the architecture. Does this confuse architecture with infrastructure? Taxation Tax feature projects by requiring them to contribute beyond their narrow project scope. Regulation Give clear ownership to a central authority. Introduce regulating software. Thursday, 13 September, 12
52. Education 37 ‣ Enlightened self interest • “do well by doing good” ‣ Deferred gratification • sacrifice short-term interests in favour of long-term interests Thursday, 13 September, 12
53. The code is the design? “most complete, unambiguous, and irreducible expression of a design” 38 The code is the most, Maybe. But only the de facto design, not the intended design. Thursday, 13 September, 12
54. Privatisation 39 ‣ Transfer ownership of the architecture to a single organisation • ownership obligates stewardship • acts on behalf of feature teams, who must rent or purchase services • does this confuse architecture with infrastructure? Thursday, 13 September, 12
55. 40 Creating a core team who owns the architecture usually results in another feature team. It’s hard to maintain stakeholder support for infrastructure or design artefacts. Thursday, 13 September, 12
56. 40 Creating a core team who owns the architecture usually results in another feature team. It’s hard to maintain stakeholder support for infrastructure or design artefacts. Thursday, 13 September, 12
57. Taxation 41 ‣ Introduce transaction costs • attach an immediate cost to potentially detrimental changes • changes reflect their long-term, shared cost • discourage frivolous or merely convenient dependencies Thursday, 13 September, 12
58. 42 Not all changes should be treated equally Thursday, 13 September, 12
59. 42 Not all changes should be treated equally Thursday, 13 September, 12
60. 42 Not all changes should be treated equally Thursday, 13 September, 12
61. 43 Not all changes should be treated equally Have new dependencies require review from a wider constituency Thursday, 13 September, 12
62. Regulation 44 ‣ Central authority • architects are easy to ignore • architects can’t be omniscient • architects should to encode design rules in such a way that they can’t be ignored Thursday, 13 September, 12
63. 45 Encode design constraints into automated systems: • build systems • commit triggers • custom static analysis tools Thursday, 13 September, 12
64. 46 The microeconomics of change By understanding the incentives which drive architectural erosion we can avoid tragic decline. Thursday, 13 September, 12
65. Dependencies and complexity are too cheap in the short term Confront the moral hazard 47 Short Tenure Developer turnover outpaces change of the software system. Moral Hazard Developers acting rationally for local gain aren’t exposed to the deferred cost of their actions, of technical debt. The Tragedy The common environment of the system erodes, making ongoing development more frustrating and costly. Productivity falls. 1 2 3 Tragic Cycle Thursday, 13 September, 12
66. To avert the tragedy 48 Hire smart people Be vigilant Dependencies should immediately reflect their lifetime costs Automate regulations and enforcement and organise them any which way you like. At large scales and over the long term team structures and processes are second order effects. use wide-angle tools to monitor system-scale dependencies and qualities. Follow-up at code level. Avoid a ‘the code is the design’ mentality. dedicate your effort to encoding and automation of local rules and enforcement. Special cases aren’t special enough. more closely by having new dependencies trigger design or code reviews to bring some of their cost forwards in time. Thursday, 13 September, 12
67. 49 Thursday, 13 September, 12
68. 50 Robert Smallshire Roxar Software Solutions, Oslo @robsmallshire Thursday, 13 September, 12
69. Thank you Questions? 50 Robert Smallshire Roxar Software Solutions, Oslo @robsmallshire Thursday, 13 September, 12