Autocon3 - Workshop Data Modeling

Transcript

1. Workshop Data Modeling & Network Source of Truth (B2) Autocon 3 - Prague - May 2025
2. About me : Damien Garros Co-Founder and CEO of OpsMill Focused on Infrastructure as Code, Automation & Observability for 12+ years Previously leading Technical Architecture at Network to Code @dgarros damiengarros
3. OpsMill Team Benoit Kohler Baptiste Girard Alex Gittings
4. Agenda 1/2 • Introduction 10 Min 2pm • Part 1 - Data Management 90 Min ◦ Introduction to Data Management ◦ Schema | Key Concepts ◦ Different type of databases ◦ LAB 1 BREAK 3:45pm ◦ Schema | Advanced Concepts ◦ Beyond the Schema
5. Agenda 2/2 • Part 2 - Network Infrastructure modeling 90 Min 4:30pm ◦ Data in Layers ◦ Business & Operational Context ◦ Data Federation / Aggregation ◦ Design for Idempotency ◦ LAB 2 • End 6pm
6. Targets & Goals for this workshop This workshop is targeted for automation builders with some experience building scripts or applications. The goals of this workshop are : • Introduce the fundamental technologies to store, organize and consume data (schema and database) and to present the differences between them • Present the best practices and challenges to model a network infrastructure in a Source of Truth
7. What about you ? • You are familiar with SQL ? • You have already used GraphQL ? • You already tried Infrahub ? • You love XML too ? • You are familiar with Neo4j ?
8. Introduction
9. Automation starts with Data Source of Truth Observability Telemetry SLA Compliance Reporting Service Catalogue User Interface Data Governance Deployment Automation
10. Source of Truth Configuration Templates IPAM Roles/Statuses Routing Information Inventory Circuit Cabling / Topology DCIM What do you need to rebuild it if the network was completely lost ? Services
11. Typical Network Source of Truth Git In House solution Netbox / Nautobot Infrahub
12. Key Pillars To Successful Automation Flexible Data Model Versioning CI Pipeline
13. Multiple layers of infrastructure data Component Layer Each element is managed individually. Technical Layer Global representation of the infrastructure elements interconnected Service / Intent Layer Definition of what services the infrastructure needs to deliver Service Server Firewall Network Server Firewall Network
14. Capture both Service and Technical information Service Definition Generate all technical spec (In memory) Generate the Configurations User Input Generate the Configurations User Input Generate all technical spec Based on a design in software Technical Specification
15. Flexible data models Source of Truth Scale Horizontally to manage more elements New Use cases New device Scale Vertically to capture higher level objects, business context, design & services
16. Part 1 Data Management
17. Schema : Definition A schema defines the structure, format, and constraints of data, specifying how data is organized and interpreted in databases or data models. It is important because it ensures data consistency, integrity, and facilitates communication between different systems by providing a shared understanding of the data’s structure.
18. It’s not about which one is the best, it’s about which one provide the best trade off for your use case
19. Data management trade offs Flexibility Integrity Consistence Accessibility Performance Change Management Availability
20. Schema : Definition --- - location: USA site_name: TechHub partial_address: 123 Main St - country: Germany site: BlueOcean address: 456 Ocean Dr - country: Japan address: 789 Sunset Blvd Does it has a schema ?
21. Schema : Definition Read Search Query Read file Query information Exchange Data Whether it’s intentional or not, there is always a schema
22. Schema : Purpose Documentation Data Integrity Validation Data Storage
24. A schema can be defined / implemented at various levels in an application stack Each level has its own set of advantages and trade offs Different implementations Storage Application User Place where the schema can be define / implemented
25. Pros and Cons of having a schema
26. Schema, data format or query ? SQL Excel JSON Schema XML YANG PromQL GraphQL JSON Protobuf Pydantic YAML TOML XPath Thrift Avro JMESPATH
27. Schema, data format or query ? Schema Data Format Query Language XML SQL GraphQL XPath PromQL TOML YAML JSON JSON Schema Protobuf YANG Pydantic Thrift CSV Excel
28. Schema Key Concepts
29. • Structure: A schema defines how data is organized. It specifies what kind of data can go where. • Relationships: Schemas describe how different pieces of data are connected, such as linking customers to their orders. • Constraints: It sets rules for the data, such as what values are allowed or required. This helps ensure data accuracy. Schema Principles
30. A schema is composed of Entities or Nodes Each node is usually composed of some attributes Attributes can have various types depending on what is supported: - Integer or Number - Text or String - Date - JSON Blob Structure
31. Relationships define how nodes are connected together DEVICE is connected to SITE DEVICE is connected to TAG Relationship types AKA Cardinality - One to One - One to Many (DEVICE - SITE) - Many to Many (DEVICE - TAG) Relationships are often called “Edges” Relationships
32. Constraints sets rules for acceptable values for each attributes or relationships. Constraints can include • Required fields • Unique values • Default values • Format • Maximum and minimum values, • Length restrictions • Maximum and minimum number of related nodes Constraints & Validations Rules
33. Constraints - An interface must be related to a device - The speed of an interface must be integer - The address of a site must include a zip code - The status of a device must be one of [active, maintenance or offline] - 2 interfaces with the same name can’t be associated with the same device - 2 devices can’t have the same name Examples
34. Schema vs Instance Prague router1 router2 Ethernet1 Ethernet2 Ethernet1 Site Device Interface Schema Instance
35. Different type of databases
37. Different type of databases Relational DBMS Key-Value Store Documents Store Graph DBMS Time Series DBMS Schema Mandatory SQL No Optional Query Powerful SQL - Simple Powerful (GQL) Powerful No Optional JSON schema Other - Optimized for Speed Optimized for Scale - Domain specific
38. Relational vs KV vs Graph
39. Database popularity
40. Evolution of popularity
41. Query execution time
42. Cypher is the query language for Neo4j, the leading graph database, and its specification are public under the OpenCypher spec. GQL is a standard language to query Graph database, it’s been recently standardized by ISO. GQL aims to be the SQL of graph databases. GQL is heavily inspired by Cypher, and Neo4j has contributed to its development, meaning GQL shares many similarities with Cypher. CYPHER & GQL
43. Part 1 • Explore and compare different schema languages ◦ JSON Schema, GraphQL, Pydantic Part 2 • Explore and compare different databases (Relational and Graph) ◦ SQLite & Neo4j Lab 1 - Goals and Agenda
44. Lab 1 opsmill/workshop-data-modeling #ac3-ws-b2 http://bit.ly/3H6jtxj
45. Labs are running on Instruqt platform, everything is already installed and ready to go! Between each step or challenge, you’ll find helpful slides with more info and context about the task. You can open these notes anytime using We know it, laptop screens can feel a bit tight! You can show or hide the assignment panel using Lab tips and tricks
46. BREAK
47. Schema Advanced concepts
48. • Migrations: Update the data to match the new version of the schema • Inheritance allows object to inherit structure or attributes from a parent or base object / entity. • Polymorphism Allows systems to handle different types of related entities through a shared interface or structure. Schema Advanced Concepts
49. Updating a schema is easy Updating the existing data to match the new schema is the hard problem. Migrations
50. What is a migration ? When do I need one ? Anything that changes the structure of the existing data will require some migrations. If there is no data associated with the schema, no migration is required. Migrations
51. • Add an attribute • Change the type of an attribute : String > Boolean • Change the name of an object • Change the relationships between objects Migrations Example
52. Migration strategy per platform Git In House Application Netbox / Nautobot Infrahub Application developer need to update the data manually or with a script. Application developer need to provide the migrations for each update, some libraries are available to help Migrations are built into the core products and the plugins. Some migrations are automatically handled by the platform. For the other one, the platform is running some validation to ensure the data has been updated prior to the migration
53. Inheritance / Polymorphism Location name description Building name description address city Rack name description rack_type height Inheritance allows object to inherit structure or attributes from a parent or base object / entity. Polymorphism allows systems to handle different types of related entities through a shared interface or structure. Use cases • Reusability • Precise schema per object • Hierarchical Data • Simplify relationships between object • Easier to extend schema over time
54. Inheritance is about the data-structure and reusing/sharing attributes and relationship between objects, it ensure consistency Polymorphism is about supporting multiple types of objects behind the same relationships Inheritance / Polymorphism
55. Polymorphism interface name description Physical Interface name description peer Connector type Logical interface name description ip addresses Device name description interfaces Physical Interface name description peer Connector type Logical interface name description ip addresses Device name description physical_interfaces logical_interfaces
56. GraphQL Interface & Fragments interface NetworkInterface { id: ID! name: String! description: String! parent: Device! } type PhysicalInterface implements NetworkInterface { peer: Device! connector_type: String! } type LogicalInterface implements NetworkInterface { ip_addresses: [IPAddress]! } query { network_interface { id name parent { name } ... on PhysicalInterface { connector_type } ... on LogicalInterface { ip_addresses { address } } } } Query Schema GraphQL natively support inheritance via Interface and Fragments
57. Inheritance / Polymorphism Support Schema Language Inheritance Polymorphism Comments SQL Partial Partial Not part of SQL , supported by Postgres JSON Schema Yes Yes through oneOF GraphQL Yes Yes through Interface Yang Yes Partial through grouping and augment Infrahub Yes Yes through Generic
58. Beyond the Schema
59. Model (Software) Model vs Schema A model is a superset of a schema which includes application specific logic and validation General Purpose schema only describes the structure, constraints, and rules of the data. A model encompasses not only the structure but also behavior and logic around the data. Schema Data Integrity Domain Specific Structure Presentation Display Constraints
60. General Purpose vs Domain Specific Schema Structure Presentation Constraints Business Logic Object level Integrity Dataset level Integrity General Purpose Schema Software Application Domain Specific Schema Software Application
61. Part 2 Network Infrastructure modeling in a Source of Truth
62. Data in Layers
63. Multiple layers of infrastructure data Component Layer Each element is managed individually. Technical Layer Global representation of the infrastructure elements interconnected Service / Intent Layer Definition of what services the infrastructure needs to deliver Service Server Firewall Network Server Firewall Network
64. Information vs Intent Component Technical Service / Intent
65. The data funnel Intended State Production System CMDB Documentation ITAM / ITOM
66. Firewall rule, ALLOW, port 389 from IP of server 1 & server 2 to IP of server 8 & server 10 Multiple layers of infrastructure data I want LDAP from server hosting Application YY to communicate with All Domain Controllers Firewall rule, ALLOW, port 389 from IP1 & IP2 to IP5 & IP8 Configuration Artifacts Source of Truth Service Layer Technical Layer Component Layer
67. Business & Operational Context
68. Business Context / Classification Interface Link Status Ethernet 1 Up Ethernet 2 Down Ethernet 3 Up Ethernet 4 Down Ethernet 5 Up Role Uplink Uplink Uplink Server Server Status Active Maintenance Active Provisioning Active
69. The 3 primary attributes Role Status Kind Capture the primary function of an object Capture all the stages of the lifecycle of an object Capture the nature of an object
70. Role defines the main function of an element: • For a server : is it a database or web server • For a network device : is it a core router or an access switch • For a site : is it a manufacturing site or an office. In some cases a given object may have multiple roles, if it’s delivering multiple functions, as an example : a server hosting both a web portal and a file server. Role
71. The status is meant to capture all the stages of the lifecycle of each object • Active • Provisioning • Maintenance • Software-Upgrade • Closed-for-Business The list of possible statuses will vary greatly between a site and a server, but the idea remains the same. Status
72. The kind (or type) captures implementation differences: • For a server : is it running linux or windows • For a network device : is it running Cisco or Arista • For a site : is it a large office or a small one. The kind is very important because usually it defines the implementation and it helps manage vendor specific requirements. Kind
73. Mapping workflows to Role, Status & Kind - name: Reboot network devices hosts: status_maintenance gather_facts: false tasks: - name: Reboot EOS device arista.eos.eos_command: commands: [ "reload now" ] when: platform == "eos" - name: Reboot Junos device juniper.junos.command: commands: [ "request system reboot" ] when: platform == "junos" Every playbook should map to a group of hosts defined but their role and status Specific actions should be controlled by the kind
74. Typically a workflow (manual or automated) is required to change the status from one value to another. This approach help to map a declarative approach and a workflow based automation. Mapping workflows to Role, Status & Kind
75. Design for idempotency
76. • Idempotent > Always the same results • Version Control Friendly > Input as text file, peer review • Safe & Predictable > Plan everything before, know what changes will be made before you run it. Infrastructure as Code principles
77. • Ensure all objects have a unique identifier that is independent of any systems ◦ Unique names ◦ Unique combination of names / relationships • Support declarative API Infrahub’s schema integrate idempotency natively with the Human Friendly Identifier (HFID) Design for idempotency
78. Data synchronization present its own set of challenges • How can we mapped objects from system A to system B • What is the state of the destination system before the sync ? Data Synchronization Source of Truth System of Record A B
79. Lab 2 opsmill/workshop-data-modeling #ac3-ws-b2 http://bit.ly/3H6jtxj
80. Thank You
81. Schema A closer look SQL, JSON Schema, GraphQL, Yang & Infrahub
82. Introduced in 1970 The language of ALL relational databases. Both a schema and a query language SQL Key Concepts: • A schema is mandatory • Data is organized in Tables • Additional features ◦ Permissions ◦ Transactions
83. SQL - Tables
84. SQL CREATE TABLE Products ( ProductID INT PRIMARY KEY, ProductName VARCHAR(50), Description TEXT, Price DECIMAL(10, 2), Quantity INT ); SELECT ProductName, Price FROM Products WHERE Price < 50; Schema Query Attribute Constraints
85. SQL Key Differentiators • Standardization and Wide Adoption • Used for Schema, Query and Storage • Support for ACID
86. Introduced in 2010 - First draft in 2013 The de-facto standard for JSON validation and structure definition Supported by many libraries, frameworks, and tools across programming ecosystems. JSON Schema Key Concepts: • Data Structure Definition • Extensibility and Modularity • Leverage the concept of “REF”
87. JSON Schema { "$id": "https://example.com/blog-post.schema.json", "$schema": "https://json-schema.org/draft/2020-12/schema", "description": "A representation of a blog post", "type": "object", "required": ["title", "content", "author"], "properties": { "title": { "type": "string" }, "content": { "type": "string" }, "publishedDate": { "type": "string", "format": "date-time" }, "author": { "$ref": "https://example.com/user-profile.schema.json" }, "tags": { "type": "array", "items": { "type": "string" } } } } { "title": "New Blog Post", "content": "content of the blog...", "publishedDate": "2023-08-25T15:00:00Z", "author": { "username": "authoruser", "email": "author@example.com" }, "tags": ["Technology", "Programming"] } Data JSON Schema External Ref Constraints
88. JSON SChema Key Differentiators • Object definition can be imported from a remote location • Work with most data structure, not just JSON • Support complex structure (heterogeneous / oneOf, allOf)
89. Developed at Facebook, released in 2015 Schema, Data query and manipulation language for APIs Designed to make APIs fast, flexible, and developer-friendly. Complementary / Alternative to REST API GraphQL Key Concepts: • Strongly typed Schema • Support Query & Mutation • Designed to be integrated with a storage engine
90. GraphQL Schema / Query type Query { posts: [Post] # Get a list of all posts post(id: ID!): Post # Get a single post by its ID users: [User] # Get a list of all users user(id: ID!): User # Get a single user by their ID } # Types representing the data structures in the system. type Post { id: ID! title: String! content: String! author: User! # Relationship to the User type comments: [Comment] # List of related Comment types } type User { id: ID! name: String! email: String! posts: [Post] # List of posts authored by this user } type Comment { id: ID! content: String! } query { posts { id title author { id name } comments { id content } } } Query Schema
91. GraphQL Schema / Mutation type Mutation { createPost(input: CreatePostInput!): Post createUser(input: CreateUserInput!): User addComment(input: AddCommentInput!): Comment } input CreatePostInput { title: String! content: String! authorId: ID! } input CreateUserInput { name: String! email: String! } input AddCommentInput { postId: ID! content: String! authorId: ID! } mutation { createPost( input: { title: "Introduction to GraphQL", content: "GraphQL is a query language for [...] executing those queries.", authorId: "1" } ) { id title content author { id name } } } Mutation Schema Response Input
92. GraphQL GraphQL Schema Query Query data Mutation Manipulate Data Create/Update/Delete Action Subscription Real Time Update (Over WebSocket) SQL Database Graph Database No-SQL Database Resolvers
93. GraphQL Key Differentiators • You get (only) what you query • Support Inheritance (Interface) • Native support for subscriptions • Designed to be integrated with a backend by default
94. YANG Key Concepts: • Hierarchical, tree-based structure for defining data • Easy to extend • Supports reusability and modularity via groupings and augments. • Integrates with protocols like NETCONF, RESTCONF, and gNMI for configuration and state management. YANG (Yet Another Next Generation) is a data modeling language designed for defining network configurations, state data, and operational behavior. Standardized by the IETF and widely used in network automation and management.
95. YANG module example-network { namespace "http://example.com/network"; prefix "ex"; container device { leaf hostname { type string; } leaf ip-address { type inet:ipv4-address; } leaf model { type string; } list interfaces { key "name"; leaf name { type string; } leaf enabled { type boolean; } } } } Groups related nodes Attributes
96. YANG Key Differentiators • Extensibility • Integration with Network Protocols • Hierarchical and Modular Data Modeling
97. Domain specific schema which goes beyond a generic schema language. Designed with Infrastructure Modeling in mind. Infrahub, provide Schema, Query and Storage out of the box, similar to SQL Infrahub Schema Key Concepts: • Domain specific schema • Captures how to store, query and represent data • Natively support inheritance / polymorphism • Support hierarchical nodes & IPAM
98. Infrahub Schema --- nodes: - name: Device namespace: Dcim label: Network Device icon: clarity:network-switch-solid inherit_from: - DcimGenericDevice - DcimPhysicalDevice attributes: - name: name kind: Text unique: true order_weight: 1000 - name: height label: Height (U) optional: false default_value: 1 kind: Number order_weight: 1400 relationships: - name: platform peer: DcimPlatform cardinality: one kind: Attribute order_weight: 1300 Presentatio n Structure
99. Relationship kind Kind Description Generic A flexible relationship with no specific functional significance. It is commonly used when an entity doesn't fit into specialized categories like Component or Parent. Attribute A relationship where related entities' attributes appear directly in the detailed view and list views. It's used for linking key information, like location Parent This relationship defines a hierarchical link, with the parent entity often serving as a container or owner of another node. Parent relationships are mandatory and allow filtering in the UI, such as showing all components for a given parent. Component This relationship indicates that one entity is part of another and appears in a separate tab in the detailed view of a node in the UI. It represents a composition-like relationship where one node is a component of the current node.
100. Infrahub Key Differentiators • Extensibility • Migrations built in • Natively support for inheritance & polymorphism • Support hierarchical nodes & IPAM
101. Summary JSON Schema GraphQL Yang SQL Infrahub Flexibility High Schema-less and easily adaptable High Defined in the API Layer High Models are easy to extend Low Schema changes require migrations Medium Some schema changes require migrations Data Integrity Limited Lacks strong constraints Medium Client-driven, depends on backend logic Medium Strong Enforced with keys, constraints, and ACID Strong Enforced keys, constraints, Nested Data Strong Suited for complex, nested structures Strong Suited for complex, nested structures Strong Suited for complex, nested structures Weaker Requires complex table structures Strong Suited for complex, nested structures Use Cases Dynamic or semi-structured data, flexible schemas Dynamic API data with customizable queries Network specific API Netconf, RESTConf, OpenConfig Structured data with strict integrity needs Infrastructure Source of Truth
102. Stateless Schema only Not coupled with a storage solution by design Stateless & Stateful JSON Schema GraphQL Yang SQL Infrahub Flexibility High High High Low Medium Data Integrity Limited Medium Medium Strong Strong Nested Data Strong Strong Strong Weaker Strong Stateful The schema is coupled with a system to store the data which means that any change in the schema may require some changes in the data as well (Migration)
103. MATCH (a:Person)-[:KNOWS]->(b) RETURN a.name, b.name CYPHER MATCH (a:Person {name: 'Alice'})-[:KNOWS]-(b:Person) RETURN b.name MATCH (p:Person)-[:KNOWS]->(f) RETURN p.name, count(f) as friends_count Pattern matching Filtering Count
104. Data Federation / Aggregation
106. Many systems of Record Cloud Network Security Cisco ISE
107. Simplify Intent Consumption for all systems Abstract the complexity away Intent Federation / Aggregation Intent Federation / Aggregation
108. Data Federation / Aggregation A B C A C B Individual Datasets Schema
109. Data Federation / Aggregation A B C A C B A C B Individual Datasets Schema Connected Datasets (Relationships)
110. Stateless Federation C B A
111. Data Synchronization A C B B C C B