Environment as Code: Moving away from Imperative Pipelines

Cloud Native tools like Infrastructure as Code (IaC) & Helm have made managing infrastructure as well as cloud-native apps easier in a lot of ways, but there are many challenges that companies accept as the cost of adopting them. IaC is good at provisioning individual infrastructure components (a few resources together), and tools like helm are good at deploying applications, but engineering teams want an entire environment with various components like networking, platform (ec2/eks), database, k8s apps, etc. to run their business applications.

To provision and teardown an entire environment, these teams have two options. They can either hand-roll imperative pipelines to manage individual components and then manage complex dependencies between these components or create a monolith deployment for the entire environment. These approaches are inefficient and slow down feature development and innovation. They also make replicating, visualizing & understanding environments difficult. What if there were a better way?

This talk digs into these challenges and then looks at how to resolve them. We will introduce Environment as Code (EaC) which is an abstraction over cloud-native tools that provides a declarative way of defining an entire environment. It has a Control Plane that manages the state of the environment, including component dependencies, and detects drift, and enables reconciliation.

Key Takeaways:

* Challenges adopting Cloud Native tools like Infrastructure as Code, Helm, etc.
* What is Environment as Code?
* How can Environment as Code help resolve those challenges?

Transcript

1. Environment as Code: Moving away from Imperative Pipelines Adarsh Shah Engineering Leader, Coach, Public Speaker Founder & CEO at CloudKnit @shahadarsh 
 https://cloudknit.io @shahadarsh https://bit.ly/eac-ghent Open Source Repo: https://github.com/cloudknit-io/cloudknit
2. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Some of the Current Tools AWS SDK
3. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit CNCF Landscape
4. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit CNCF Landscape
5. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent
6. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Evolution of Environment Setup networking + eks + …. Monolith Deployment networking eks website postgres networking networking + eks + …. ec2 website eks postgres ec2 Breaking into Separate & Smaller components + +
7. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Evolution of Environment Execution From Engineers machine Deployment Pipeline/GitOps execution from a Shared Environment
8. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit • Option 1 • Create a Monolith IaC • Option 2 • Hand-roll Pipelines - Imperative • Manage Complex Dependencies Options for Entire Environment Provisioning
9. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Other Challenges scaling IaC Replicating Environments is a pain Not easy to visualize/understand Environments Drift Detection Not straightforward to 
 promote changes Dev QA Prod
10. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Terminologies Components: A logical grouping of 1 or more Infrastructure Resources or Applications that get provisioned together. For example, Networking is an Infrastructure Component with various Infrastructure resources like Virtual Private Cloud(VPC), Subnets, Internet Gateways, Route Tables, etc. Environment: A logical grouping of all the Components that are needed to run business applications. The grouping includes components like networking, eks, database, k8s apps, etc.
11. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent What is Environment as Code?
12. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Environment as Code networking eks website postgres networking website eks postgres Declarative Environment as Code +
13. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Environment as Code
14. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit IaC vs EaC Infrastructure as Code Automates various Lego Pieces 
 (i.e. Infrastructure Resources) Environment as Code Automates how those Lego Pieces 
 are connected to make up a Lego Toy 
 (i.e. Entire Environment)
15. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Environment as Code Environment as Code (EaC) is an abstraction over Cloud native tools that provides a declarative way of de fi ning an entire Environment. It has a Control Plane that manages the state of the environment, including relationships between various resources, Detects Drift as well enables Reconciliation.
16. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Imperative to Declarative HOW Pipelines EaC WHAT
17. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Provision Environment Environment as Code Environment State (Manages dependencies, status etc. ) State networking eks State website postgres State networking website eks postgres Provision Reconcile Declarative Detect Drift Control Plane +
18. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent networking website eks postgres Teardown Environment Environment as Code Environment State (Manages dependencies, status etc. ) State networking State website eks postgres State Teardown Reconcile Declarative Detect Drift Control Plane +
19. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Attributes of Environment as Code
20. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Ability to de fi ne Entire Environment
21. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Environment as Code networking platform-k8s k8s-addons postgres networking eks website postgres Declarative Loosely Coupled
22. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Manage State for the entire Environment Example Environment State File
23. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Idempotent and Immutable for entire Environment
24. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Non-Idempotent Provision VM VM VM { } IaC 3 VMs End State
 Expected = 3 Actual = 6 Idempotent End State
 Expected = 3 Actual = 3 Reapply VM VM VM Provision VM VM VM Reapply No change { } IaC 3 VMs
25. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Provision v1 v1 v1 User Mutable Infrastructure { } IaC Apply changes v2 v2 v2 Change { } IaC v1 v1 v1 Provision User Provision v2 v2 v2 User Immutable Infrastructure { } IaC Change { } IaC Deploys v2 to same Infrastructure Provisions new Infrastructure with v2
26. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Visualize and Understand Environments
27. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Drift Detection and Reconciliation Reconcile (Preferably with Approval step that shows the plan) Desired State Current State Control Loop Observe/Detect Drift
28. @shahadarsh https://bit.ly/eac-ghent https://github.com/cloudknit-io/cloudknit Compare and Promote Changes between Environments Compare & promote changes across environments GitOps for Entire Environment
29. @shahadarsh https://github.com/cloudknit-io/cloudknit https://bit.ly/eac-ghent Create Pull Request, EaC Validate Validate Approve & 
 Merge PR Control Plane Reconcile GitOps for Environment EaC Trunk networking platform-k8s k8s-addons postgres Observe Branch networking eks website postgres
30. CloudKnit Open Source Repo: https://github.com/cloudknit-io/cloudknit
31. Thank You @shahadarsh https://bit.ly/eac-ghent Adarsh Shah Engineering Leader, Coach, Public Speaker Founder & CEO at CloudKnit @shahadarsh 
 https://cloudknit.io Open Source Repo: https://github.com/cloudknit-io/cloudknit