Pulumi: Cloud Infrastructure as C# and F#

Modern cloud platforms offer amazing capabilities for application developers. However, cloud services are hard to manage without employing Infrastructure as Code tools for automation.

Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse.

What if instead of configuration files, you could use your favorite programming languages like C# and F#? In this talk, Mikhail will introduce you to Pulumi, which uses actual code to manage infrastructure! See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. Cloud Infrastructure as C# and F# with Pulumi Mikhail Shilkov
2. • Software engineer at Pulumi: .NET SDK, Azure, Core platform • Cloud, Serverless • Functional programming, F# • Microsoft Azure MVP https://mikhail.io @MikhailShilkov About me
3. Agenda
4. Agenda
5. Agenda
6. Cloud Automation 5
7. Unified Management API Management API (e.g., Azure Resource Manager) Cloud Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party
8. REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15
9. var launchRequest = new RunInstancesRequest { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
10. aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \ --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools
11. Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone SDK and Scripting: Challenges
12. Desired State Configuration 10
13. Desired State Configuration Target Current Tool
14. Managing Resource Graphs Target Current Tool
15. Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Correctness Transparency Challenges
16. AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Vendor Tools
17. Markup Languages 14 YET ANOTHER MARKUP LANGUAGE
18. Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]
19. AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled
20. resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)
21. Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience
22. Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples
23. { "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation
24. { "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates
25. locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform
26. Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability
27. Inspect the resulting resource tree before deployment (against mocks) Deploy a part of your graph to a test environment, inspect the result Test the result of a complete deployment Unit Integration End-to-end Testability
28. I think we solved these problems before?
29. IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring
30. Pulumi Cloud infrastructure using real languages Developers and operators working together 21
31. General-Purpose Programming Languages Node.js Go Python .NET
32. • AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
33. var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# example
34. Still, Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });
35. CLI drives the deployment How Pulumi works CLI & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD
36. Static Website on AWS Demo 26
37. Serverless Functions on Azure Infrastructure Components 33
38. Resources to Provision an Azure Function Blob Plan Container Storage Account Function App
39. • Encapsulate low-level resources into a higher-level abstraction • Reuse, shared abstractions, versioning, package management • Grouping and structure Component Resources
40. // A resource group to contain our Azure Functions var resourceGroup = new ResourceGroup("functions-rg"); // A .NET Azure Function var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish") }); .NET Azure Function
41. Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create
42. // A Node.js Azure Function var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./javascript"), Runtime = "node" }); Node.js Azure Function
43. var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ResourceGroupName = linuxResourceGroup.Name, Kind = "Linux", Reserved = true, Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" } }); var python = new ArchiveFunctionApp(...); Python Azure Function on Linux
44. var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ... }); var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs { ResourceGroup = linuxResourceGroup, Archive = new FileArchive("./python"), Plan = linuxPlan, Runtime = "python" }); Python Azure Function on Linux
45. Cosmos App Creating custom abstractions 37
46. Cosmos App (1)
47. Cosmos App (2)
48. Cosmos App (3)
49. Cosmos App (4)
50. var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications
51. var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications
52. How it works Behind the scenes 41
53. N providers, M runtimes
54. N providers, M runtimes
55. O(N*M) complexity
56. O(N+M) complexity Intermediate Representation
57. O(N+M) complexity
58. State Management and Backends 41
59. State is a persisted graph State Management CLI and Engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD Pulumi Service S3 / Blob Storage Local disk
60. Open-source Free for any use Cross-platform Written in Go Open-source Free for any use Can be created by 1st or 3rd party Web API + portal “GitHub for infra” Free for individuals Paid plans for teams Useful but not required CLI Providers SaaS backend Pulumi components
61. Cloud credentials must be configured for CLI to run AWS profiles az login Tokens can be set via environment variables CLI Local logins Environment Cloud Authentication
62. Deployment process Allow developers to iterate quickly on dev environments. Gain visibility and change management with Continuous Delivery.
63. Track Infrastructure Changes from Commit to Deployment End to End Visibility
64. Dependency Visualization Understand Dependencies Between Resources
65. APIs and Webhooks Integrate Infrastructure Changes via Webhooks
66. Role-Based Access Controls Manage Access Based on Stacks and Teams
67. Kubernetes Deploy to multiple clouds 45
68. Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application
69. Organizations, Projects, and Stacks Org: acme-corp Project: vpc Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: k8s-cluster Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: svc-userprofile Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: svc-email Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2
70. Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1
71. Advanced scenarios Demo 46
72. F# 48
73. let storageAccount = Account("sa", AccountArgs (ResourceGroupName = io resourceGroup.Name, AccountReplicationType = input "LRS", AccountTier = input "Standard")) let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1") let appServicePlan = Plan("asp", PlanArgs (ResourceGroupName = io resourceGroup.Name, Kind = input "App", Sku = input sku)) Resources in F#
74. let d = deployment "mydep" { deploymentSpec { replicas 3 selector { matchLabels appLabels } podTemplateSpec { metadata { labels appLabels } podSpec { container { name "nginx" image "nginx" containerPort 80 } } } } } Computation Expression DSL? https://github.com/pulumi/pulumi/issues/3644
75. Quality Control 50
76. Programming Languages for Infrastructure? You sure?
77. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync<MyStack>(); var resource = result.Resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
78. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync<MyStack>(); var resource = result.Resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
79. [Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync<MyStack>(); var resource = result.Resources.OfType<SecurityGroup>().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing
80. Policy as Code (.NET coming soon) const policies = new PolicyPack("aws", { policies: [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });
81. Alternatives 53
82. TypeScript Python Java .NET Transpiled to and deployed by AWS CloudFormation AWS only Languages CloudFormation AWS Cloud Development Kit (CDK)
83. var queue = new Queue(this, "MyFirstQueue", new QueueProps { VisibilityTimeoutSec = 300 }); var topic = new Topic(this, "MyFirstTopic", new TopicProps { DisplayName = "My First Topic Yeah" }); AWS CDK example
84. // This is F# let myStorage = storageAccount { name "mystorage" // set account name sku Storage.Sku.PremiumLRS // use Premium LRS } let myWebApp = webApp { name "mysuperwebapp" // set web app name sku WebApp.Sku.S1 // use S1 size setting "storage_key" myStorage.Key // app setting depends_on myStorage // webapp is dependent on storage } Azure? ARM Generators (ex: Farmer) https://github.com/CompositionalIT/farmer
85. Conclusions 55
86. Define Infrastructure as Code Making Cloud Apps?
87. Fine-grained components Better Automation
88. Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code
89. м Infrastructure as Code Architecture as Code
90. • Get Started with Pulumi: https://pulumi.com/dotnet • Pulumi + .NET blog post: “Building Modern Cloud Applications using Pulumi and .NET Core” https://devblogs.microsoft.com/dotnet/building-modern-cloud- applications-using-pulumi-and-net-core/ • Pulumi + Cosmos blog post: “How to build globally distributed applications with Azure Cosmos DB and Pulumi” https://azure.microsoft.com/en-gb/blog/how-to-build-globally- distributed-applications-with-azure-cosmos-db-and-pulumi/ Useful Links
91. Meetups, Talks and Workshops with Pulumi Upcoming Events @PulumiCorp pulumi http://bit.ly/2OQ00Tt 18 Feb Copenhagen, DK 18 Feb Kaunas, LT 19 Feb Malmo, SE 20 Feb Amsterdam, NL 21 Feb Madrid, ES 25 Feb Prague, CZ 26 Feb Frankfurt, DE 30 Mar Amsterdam, NL
92. Amsterdam – 30 March The Pulumi team will provide hands-on instruction in a 4-hour workshop showing you how to setup your infrastructure for Kubernetes and other useful configurations. Includes lunch, workshop, labs and happy hour. Coming to a city near you Pulumi IaC Workshops @PulumiCorp pulumi http://bit.ly/2HgD9wd You don’t have to attend KubeCon to join this workshop
93. @MikhailShilkov https://mikhail.io