Share
Embed
USING GRAPH THEORY & NETWORK SCIENCE TO EXPLORE YOUR MICROSERVICES ARCHITECTURE
Talk Given at GOTO Copenhagen 2019
- https://gotocph.com/2019/sessions/1112/using-graph-theory-and-network-science-to-explore-your-microservices-architecture
Nicki Watt
November 18, 2019
Transcript
81 slides
-
Nicki Watt - CTO @techiewatt USING GRAPH THEORY &
NETWORK SCIENCE TO EXPLORE YOUR MICROSERVICES ARCHITECTURE -
@techiewatt Hands-on software delivery Consulting provides services is CTO at
Cloud Native / Microservices Architectures Data Engineering Platforms Co-author of with expertise in lead projects on includes experience gained building -
AGENDA OVERVIEW • INTRO & HYPOTHESIS • A BIT OF
THEORY • MICROSERVICE EXPLORING • SUMMARY @techiewatt -
Graph theory is already being used to drive efficiencies in,
and produce more reliable software systems INTRODUCTION @techiewatt -
INTRODUCTION @techiewatt Infrastructure: Reliably managing and updating cloud and other
infrastructure-as-code resources Credit: Paul Hinze (HashiConf 2016) Applying Graph Theory to Infrastructure-as-code -
Can we also use it to help inspect and drive
us towards improvements in our microservices architecture? INTRODUCTION @techiewatt -
In microservice architectures, the most important aspects are not properties
of the code but properties of the system - Adam Tornhill #GOTOcph SHOUT OUT! @techiewatt -
HYPOTHESIS @techiewatt General Hypothesis: Data Driven Architectural Improvement You
can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture -
HYPOTHESIS @techiewatt For this talk … Can we use
these metrics to detect bad microservice architectural smells and anti-patterns like a tightly coupled architecture (distributed monolith) -
GRAPH THEORY 101 Nodes / Vertices Relationships /
Edges @techiewatt A graph is a way to formally represent a network, or collection of related objects, in a mathematical way -
• Graph Analytics ◦ An Action Performed: The act of
analysing connected data - using any appropriate graph-based approach or tools (visualisations, queries, statistics, algorithms) • Graph Algorithms ◦ Programmable process or set of rules: Leverages the mathematical properties of graphs to explore, classify and interpret connected data. A subset of tooling used to do graph analytics. @techiewatt GRAPH INSIGHTS 101 -
GRAPH INSIGHTS 101 @techiewatt New academic field, circa 21st century
includes study of: • Real-world representations in order to understand the universal properties of networks (Biological, social, transport) • Large, complex networks Roots date back to 1786, includes study of: • Abstract theoretical graph forms (e.g. random graphs, trees, directed graphs) • Graphs of any size NETWORK SCIENCE GRAPH THEORY -
“Based on the mathematics of graph theory, graph algorithms use
the relationships between nodes to infer the organization and dynamics of complex systems. Network scientists use these algorithms to uncover hidden information, test hypotheses, and make predictions about behavior”. - Graph Algorithms by M Needham, A Hodler @techiewatt GRAPH INSIGHTS 101 -
Show me some of this graph theory / network science
stuff ... @techiewatt -
DEGREE How connected is a specific node? A is more
highly connected than B and C 6 2 2 @techiewatt 1 -
CLUSTER COEFFICIENT How tightly is a group clustered, compared to
how tightly it could be clustered? @techiewatt 2 -
COMMUNITY DETECTION Used to find related communities, uncover groupings, and
quantify the quality of groupings @techiewatt 3 -
@techiewatt V1.0 - NETWORK STATS Density: 0.19 Number of nodes:
20 Number of edges: 36 Average degree: 3.6000 Average Clustering Co-eff: 0.30 -
@techiewatt We’ve been directed to look a bit closer at
a few potential problem services … Where do they fit in architecturally? -
V1.0 - ARCHITECTURE DIAGRAM @techiewatt Front End Service Back End
Service External Integration Adapter Service Database API API API API API API -
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS (DEGREE) @techiewatt Front End Back End External Integration -
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS (CLUSTER COEFF) @techiewatt Front End Back End External Integration -
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C v1 - degree & cluster coefficient degree cluster coefficient V1.0 ANALYSIS (AVERAGES) @techiewatt Average degree: 3.6 Average Clustering Coefficient: 0.3 Front End Back End External Integration -
V1.0 INVESTIGATION @techiewatt - Entity Services Anti-Pattern - Distributed
Monolith Front End Back End External Integration -
V1.0 INVESTIGATION @techiewatt - Entity Services Anti-Pattern - Distributed
Monolith Front End Back End External Integration Order User Product -
V1.0 INVESTIGATION @techiewatt - Entity Services Anti-Pattern - Distributed
Monolith Front End Back End External Integration -
V1.0 - OBSERVATIONS CONTINUED @techiewatt - Help detect tightly coupled
areas - Not always DDD aligned -
COMMUNITIES NOT ALWAYS DDD ALIGNED @techiewatt Process & Data Criteria
Organisation Operations Erich Eichinger Blog: “Heuristics for Identifying Service Boundaries” https://opencredo.com/blogs/identify-service-boundary-heuristics/ -
Recap … V1.0 @techiewatt User Front End Back End External
Integration Database -
V1.0 —> V2.0 @techiewatt U1 U2 U3 U4 U5 U6
User Front End Back End External Integration Database -
V1.0 —> V2.0 @techiewatt User U1 U2 U3 U4 U5
U6 User Mob API User Web API Front End Back End External Integration Database -
V2.0 @techiewatt User V1.0 —> V2.0 Front End Back End
External Integration Database -
V2.0 - NETWORK STATS @techiewatt Density: 0.11 Number of nodes:
29 Number of edges: 46 Average degree: 3.1724 Average Clustering Co-eff: 0.20 -
@techiewatt COMPARING Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8
1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient -
0 0.2 0.4 0.6 0.8 1 1.2 0 2 4
6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 User Service -
0 0.2 0.4 0.6 0.8 1 1.2 0 2 4
6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 User Service New V2 Refactored User Services -
Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8 1 1.2
0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient @techiewatt COMPARING New V2 Refactored User Services V1 User Service V2 Mono User Service (allow distmon to continue functioning) -
@techiewatt COMPARING Degree Cluster Coefficient 0 0.2 0.4 0.6 0.8
1 1.2 0 2 4 6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient -
0 0.2 0.4 0.6 0.8 1 1.2 0 2 4
6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V1 (5) V2 (4) -
0 0.2 0.4 0.6 0.8 1 1.2 0 2 4
6 8 10 12 14 16 ShippingFacade Order PricingCalculator User Product Paym entFacade W eb API M ob API B2B API UPS DHL Royal M ail Shipper X Shipper Y Shipper Z M asterCard PayPal Provider A Provider B Provider C M onoUserService W eb User API M ob User API EventPublisher RegisterNew User UpdateCoreDetails UpdateM arketingPrefs ReportUserAbuse GetUserProfile SearchUser EventConsum er UserEventM onoConsum er Combined Measurements (V1 & V2) v1 degree v2 degree v1 cluster coefficient v2 cluster coefficient Degree Cluster Coefficient @techiewatt COMPARING V2 (1.0) V1 (0.9) -
OTHER MEASURES & ALGORITHMS @techiewatt - Strongly Connected
Components (SCC) - Help detect circular dependencies -
OTHER MEASURES & ALGORITHMS @techiewatt - Path Finding Algorithms Detect
who is calling deprecated services -
Early days … Can’t be applied blindly … Common sense
and understanding of architectural approaches and patterns still required!! CONCLUSION @techiewatt -
HYPOTHESIS @techiewatt General Hypothesis: Data Driven Architectural Improvement You
can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture -
HYPOTHESIS @techiewatt General Hypothesis: Data Driven Architectural Improvement You
can extract metrics and KPIs from a microservices architecture using graph theory AND use these to gain insight into the structure and characteristics of your microservices architecture -
HYPOTHESIS @techiewatt Specifically Can we use these metrics to
detect bad microservice architectural smells and anti-patterns like a tightly coupled architecture (distributed monolith)? -
CONCLUSION @techiewatt Demonstrated using degree and cluster coefficient measures to
detect tightly coupled (distributed monolith) architectures Demonstrated using community detection algorithms to uncover related groupings (boundaries) of microservices -
@techiewatt Come chat with me! Want Some Help? Questions ?
Want to find out more?