Cloud Management Superpowers with Pulumi and .NET

Transcript

1. Cloud Management Superpowers with .NET Mikhail Shilkov
2. DevOps Developer IT Infrastructure User
3. DevOps in Cloud age Developer Cloud Infrastructure User IT
4. Cloud tooling Developer
5. About me • Mikhail Shilkov • Software engineer at Pulumi .NET SDK, Azure, Core platform • Serverless • Functional programming, F# • Microsoft Azure MVP @MikhailShilkov mikhail@pulumi.com
6. Intro • Pulumi: Modern Infrastructure as Code • .NET languages to manage cloud resources Cloud Superpowers • Provisioning • Architecture • Testing • Policy • Delivery Agenda
7. .NET for All the Clouds Provision infrastructure anywhere using C#, F#, or VB.NET
8. General-purpose Programming Languages
9. Providers • AWS • Azure • GCP • Digital Ocean • Cloudflare … and more • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
10. Static Website on AWS Demo
11. Provisioning with Desired State Configuration 10
12. Cloud Management API Management API (e.g., Azure Resource Manager) Cloud Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party
13. var launchRequest = new RunInstancesRequest { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
14. aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \ --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools
15. SDK and Scripting: Challenges 15 Imperative Step-by-step provisioning process Complex Depend on both the current and the target state Error-prone How to handle failures?
16. Desired State Configuration Target Current Tool
17. Managing Resource Graphs Target Current Tool
18. var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example
19. Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });
20. How Pulumi Works CLI & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD
21. Markup-based Languages 21 AWS CloudFormation YAML Azure RM Templates JSON Terraform HCL "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ] Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group .rg.location resource_group_name = azurerm_resource_group .rg.name account_tier = "Standard" account_replication_type = "LRS" } Every alternative has its own language, own tools, own conventions
22. Markup Authoring Experience 22 Tooling Lack of tooling on day 1. No fast feedback from editors. Discovery Which properties can be set, and which values are valid? Size Thousands of lines of markup are hard to write, read, and maintain
23. Example App: Serverless URL Shortener KV Store Table “URLs” Function “Add URL” Function “Open URL” Tool Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples
24. Tools That You Love Developers can apply their existing skills to infrastructure
25. Pulumi ❤ .NET • .NET Core 3.1 • OS: Windows, macOS, Linux • Language: C#, F#, VB.NET • Editor and IDE: Visual Studio, Code, Rider • IntelliSense, ReSharper, StyleCop, DocFX • Package Manager: NuGet, MyGet, Paket • NUnit, xUnit.net, Moq
26. var storageAccount = new Account("sa", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard" }); var appServicePlan = new Plan("asp", new PlanArgs { ResourceGroupName = resourceGroup.Name, Kind = "App", Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” } }); Cloud Resources with C#
27. let storageAccount = Account("sa", AccountArgs (ResourceGroupName = io resourceGroup.Name, AccountReplicationType = input "LRS", AccountTier = input "Standard")) let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1") let appServicePlan = Plan("asp", PlanArgs (ResourceGroupName = io resourceGroup.Name, Kind = input "App", Sku = input sku)) Cloud Resources with F#
28. ' Create an Azure Resource Group Dim resourceGroup = New ResourceGroup("resourceGroup") ' Create an Azure Storage Account Dim storageAccount = New Account("storage", New AccountArgs With { .ResourceGroupName = resourceGroup.Name, .AccountReplicationType = "LRS", .AccountTier = "Standard" }) Cloud Resources with VB.NET
29. Architecture Reusable Abstractions
30. Resources to Provision an Azure Function Blob App Plan Container Storage Account Function App
31. // A resource group to contain our Azure Functions var resourceGroup = new ResourceGroup("functions-rg"); // A .NET Azure Function var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish") }); .NET Azure Function
32. Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create
33. // A Node.js Azure Function var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./javascript"), Runtime = "node" }); Node.js Azure Function
34. var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ResourceGroupName = linuxResourceGroup.Name, Kind = "Linux", Reserved = true, Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" } }); var python = new ArchiveFunctionApp(...); Python Azure Function on Linux
35. var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ... }); var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs { ResourceGroup = linuxResourceGroup, Archive = new FileArchive("./python"), Plan = linuxPlan, Runtime = "python" }); Python Azure Function on Linux
36. Building a Cosmos App Create custom abstractions
37. Cosmos App (1)
38. Cosmos App (2)
39. Cosmos App (3)
40. Cosmos App (4)
41. var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications
42. var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications
43. Testing and Policy Validate deployments
44. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync<MyStack>(); var resource = resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
45. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync<MyStack>(); var resource = resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
46. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync<MyStack>(); var resource = resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
48. Policy as Code (.NET coming later) const policies = new PolicyPack("aws", { policies: [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });
49. Delivery Multi-cloud cross-stack automation
50. State Management CLI and Engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD Pulumi Service S3 / Blob Storage Local disk
51. Pulumi Components 51 CLI Open-source Free for any use Cross-platform Written in Go Providers Open-source Free for any use Can be created by 1st or 3rd party SaaS backend Web API + portal “GitHub for infra” Free for individuals Paid plans for teams Useful but not required
52. Deployment process Allow developers to iterate quickly on dev environments. Gain visibility and change management with Continuous Delivery.
53. Cloud Transition 53 Kubernetes Azure Functions AWS S3 Azure Analytics Google ML MySQL DataDog App Docker DataDog New Relic App MySQL EARLY CLOUD Mostly Static N-Tier Apps VMs Private Cloud CURRENT CLOUD Partly Dynamic Less Monolithic VMs and Containers* Hybrid - Public/Private *ExperimentaDon FUTURE CLOUD Fully Dynamic Hyper-Connected Services Containers and Serverless Mostly Public Cloud
54. Infrastructure Landscape 54 Foundation Security IAM KMS Networking VPC Subnets Firewalls Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD
55. Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application
56. Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1
57. Kubernetes & Multi-stack Solutions Demo
58. Conclusions
59. PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence
60. Useful Links http://bit.ly/pulumilinks
61. Q&A