Designing and Visualizing Cloud Architectures for Humans

Today's software architectures are organic, scaling with cloud infrastructure and changing with agile development cycles. As software matures and teams grow, so does complexity. Combining d3.js visualizations and monitoring, we'll explore different dimensions of cloud architectures, both in design and operation to focus less on technical details and more on what matters: people.

Transcript

1. Designing and Visualizing Cloud Architectures for Humans • Julie Ng Cloud Solution Architect Microsoft • Frontconf Munich 27 April 2019 Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 1
2. What is Good Architecture? In the opinion of a former Enterprise Architect 1
3. Hello World! ONCE UPON A TIME…
4. Decouple Frontends! DIFFERENT UI REQUIREMENTS
5. Backends for Frontends (BFF) DIFFERENT UX REQUIREMENTS
6. • Decoupling has continued: • microservices, containers, serverless… what’s next? • on-prem à cloud à where’s my data? • Failures more complex • Debugging harder • Does it have to be this way? • WHY is it this way? Did I order this Architecture Spaghe>? IT-TRANSFORMATION AT SCALE Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 6
7. • One that works • Has no downtime • Enterprise Architecture = bridging technology and business • Allianz Germany 2018 • 36.4 € Billion Revenue • 2.4 € Billion Operating Profit • 129 years old • So how much does 1 hour of downtime cost? What is a good architecture? LARGE SCALE - ENTERPRISE ARCHITECTURE COMPANY STATISTICS FROM h/ps://www.allianzdeutschland.de/die-allianz-deutschland-im-portraet/ Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 7
8. Cloud Context The Human Element What causes DownAme? PER GOOGLE NEXT (YELLOW), INDEPENDENT ORACLE USER GROUP (BLUE) 3rd Party or Cloud Server Failures Storage Failures Application Errors Usage Spikes/Surges Power Outages Network Outage Human Error Bad Deployments Monitoring Gaps Scaling Boundaries Uneven Sharding ' Humans cause downtime The cloud just works… when configured properly (by humans) Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 8
9. • Measure quality of architecture by downtime • Assume the cloud just works • Assume we humans caused the failure • Goal is to understand human factor in architecture • How do we behave? • Is there a pattern? • Why do we behave the way we do? The Human Factor on Architecture WHY I BUILT NEWTON.JS We’re pushing the buttons Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 9
10. How do you Document Architecture? What did we intend to build? 2
11. Architecture Diagrams CLASSIC DOCUMENTATION Diagram Source – Scalable Web Application Reference Architecture https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/app-service-web-app/scalable-web-app • Classic diagrams • Show intent • But very static • Not real-time. Quickly outdated. • What do arrows indicate? • Dependencies? • Data Flow? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 11
12. Typical Real Time Monitoring Dashboards INTERACTIVE DOCUMENTATION? • Latency • CPU Usage • Uptime • Storage Consumption • Memory Consumption • Nodes Count • Temperature Does this make sense for a cloud application? No. The cloud just works, remember? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 12
13. • For 10.000 feet overview - too much information • Missing relationships – remember those arrows? • To understand how everything is connected, we have to go return to the humans… Monitor uptime for hundreds of components? JUST UPTIME Graphic Source https://community.grafana.com/t/status-panel-for-more-than-200-servers/2710 Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 13
14. OrganizaAons in the Cloud The people behind the code 3
15. What we imagine – 1 team ORGANIZATIONS – UNIFORM AND STANDARDIZED IN LARGE OPEN SPACE OFFICES Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 15
16. Reality – many chaoAc teams, but diverse ORGANIZATIONS So do you really expect 1 architecture? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 16
17. Conway’s Law BEHAVIORAL PSYCHOLOGY “ Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations. - Melvin Conway. "How Do Committees Invent?” Datamation (April 1968). So despite independent teams, is there only a single possible overall result? Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 17
18. • Installed on base images so every team uses them – easy win? • Large organizations and systems à too much noise • Arrows indicate request flow, but not intent - the relationships we wanted to build • How can we indicate intent? Automated graphs with trace agents AUTOMATE RELATIONSHIPS Example automated graph Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 18
19. Junior designers… make it look good JULIE ZHOU, VP PRODUCT FACEBOOK ON GOALS https://medium.com/the-year-of-the-looking-glass/junior-designers-vs-senior-designers-fbe483d3b51e Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 19
20. Senior designers… make new value JULIE ZHOU, VP PRODUCT FACEBOOK ON GOALS https://medium.com/the-year-of-the-looking-glass/junior-designers-vs-senior-designers-fbe483d3b51e Before we automate, let’s create. How can we reintroduce intent to real time monitoring? Remember, that’s how we choose to measure good architecture. Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 20
21. Visualizations with d3.js Data Driven Transformations on the DOM 4
22. • There are no loops over the DOM • Based on data() • selectAll() does it all: • Adds Circles • Updates Circles • Removes Circles Data Driven – describe what, not how D3.JS - NO FOR LOOPS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 22
23. • Nodes array • IDs must be unique • Links Array • source & target reference IDs • References are key based, not Array index based #PROTIP Source and targets are not array index based – although d3.js documents and tutorials say they are. Example Data D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 23
24. General Update Pattern D3.JS DATA DOM ELEMENTS Enter Exit Update Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 24
25. • General Update Pattern • nodes.data(data) // join • nodes.exit() • nodes.enter() • Nodes.merge() // update Nodes D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 25
26. • selection.data(data, (d) => d.id) // does not work for links * Links, Key Functions D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 26
27. • Default Force, Particles Engine • Too uniform? D3.js Force Engine D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 27
28. Webcola Force Layout D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 28
29. • D3 Layout iterations: tick • Custom events via EventEmitter • this.emit('update') • Standard events • mouseover, mouseout • dragstart • etc. Event Driven D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 29
30. • Runtime vs. State • Transactions Visualize Dependencies – intent! D3.JS Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 30
31. Live Demo! Newton.js Cloud Architecture Graphs for Humans https://github.com/julie-ng/newtonjs-graph Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 31
32. • Follow Me! • julie.io • github.com/julie-ng • twitter.com/jng5 • Credits - all stock photos are from amazing photographers on Unsplash https://unsplash.com/collections/4638940/frontconf-talk-cloud-architectures- visualizations/98850e9cfebb9cc0d350ce787e6986e4 • Note - all opinions are my own and do not reflect those of my employers past and present. Thank you! BYE BYE Frontconf - 27 April 2019 Designing and Visualizing Cloud Architectures for Humans 32