Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure as Software

Infrastructure as Software

n this talk, Mikhail will demonstrate why writing infrastructure in general programming languages is a better way to is a better choice for infrastructure management. Pulumi is an open source tool that allows users to write their infrastructure code in TypeScript, Python, .NET or Go.

General-purpose languages allow infrastructure code to have integrated testing, compile-time checks as well as being able to create infrastructure APIs and is more suited to infrastructure management than DSLs, JSON or YAML. In addition, he will demonstrate how to build infrastructure that manages Serverless, Kubernetes, PaaS and IaaS systems across multiple cloud providers.

Mikhail Shilkov

March 25, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. • Software engineer at Pulumi: .NET SDK, Azure, Core platform

    • Cloud, Serverless • Functional programming, F# • Microsoft Azure MVP https://mikhail.io @MikhailShilkov About me
  2. Unified Management API Management API (e.g., Azure Resource Manager) Cloud

    Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party
  3. REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization:

    AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15
  4. var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType

    = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
  5. aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \

    --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools
  6. Step-by-step provisioning process Depend on both the current and the

    target state How to handle failures? Imperative Complex Error-prone Scripting: Challenges
  7. Be able to describe the desired state in full, concisely,

    and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Dependencies Transparency Challenges
  8. Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type":

    "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]
  9. AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties:

    BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled
  10. resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location

    resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)
  11. Lack of fast feedback about errors Which options need to

    be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience
  12. Example: Code to infrastructure ratio KV Store Table “URLs” Function

    “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples
  13. { "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress":

    [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation
  14. { "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]",

    "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates
  15. locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" {

    count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform
  16. Different shape of infrastructure for different environments Bundling several related

    resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability
  17. Inspect the resulting resource tree before deployment (against mocks) Deploy

    a part of your graph to a test environment, inspect the result Test the result of a complete deployment Unit Integration End-to-end Testability
  18. IDEs Code completion Unit tests Types Functions Package managers Versioning

    Classes Code reviews Contracts SDKs Workflows Refactoring
  19. • AWS • Azure • GCP • Digital Ocean •

    Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
  20. const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage",

    { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example
  21. const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage",

    { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); Still, Desired State!
  22. Command-Line Interface (CLI) drives the deployment How Pulumi works CLI

    & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD
  23. • Encapsulate low-level resources into a higher-level abstraction • Reuse,

    shared abstractions, versioning, package management • Grouping and structure Component Resources
  24. // A resource group to contain our Azure Functions const

    resourceGroup = new azure.core.ResourceGroup("rg"); // A Node.js Azure Function const nodeApp = new azure.appservice.ArchiveFunctionApp("http", { resourceGroup, archive: new pulumi.asset.FileArchive("./javascript"), }); export const nodeEndpoint = nodeApp.endpoint; Node.js Azure Function
  25. Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack

    functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create
  26. // A .NET (C#) Azure Function const dotnetApp = new

    azure.appservice.ArchiveFunctionApp("dotnet", { resourceGroup, archive: new pulumi.asset.FileArchive("./dotnet/bin/Debug/"), appSettings: { runtime: "dotnet", }, }); .NET Azure Function
  27. const linuxResourceGroup = new azure.core.ResourceGroup("linuxrg"); const linuxPlan = new azure.appservice.Plan("linux-asp",

    { resourceGroupName: linuxResourceGroup.name, kind: "Linux", sku: { tier: "Dynamic", size: "Y1" }, reserved: true, }); const pythonApp = new azure.appservice.ArchiveFunctionApp(...); Azure Function in Python on Linux
  28. const linuxResourceGroup = new azure.core.ResourceGroup("linuxrg"); const linuxPlan = new azure.appservice.Plan("linux-asp",

    {...}); const pythonApp = new azure.appservice.ArchiveFunctionApp("http", { resourceGroup: linuxResourceGroup, plan: linuxPlan, archive: new pulumi.asset.FileArchive("./python"), appSettings: { runtime: "python", }, }); Azure Function in Python on Linux
  29. const f = new azure.appservice.HttpEventSubscription("greeting", { resourceGroup, callback: async (context,

    req) => { return { status: 200, body: `Hello ${req.params.name}!`, }; }, }); Serverless Functions as Callbacks
  30. const queue = new azure.storage.Queue("myqueue", { storageAccountName: storageAccount.name, }); queue.onEvent("MyMessageHandler",

    { callback: async (context, msg) => { doWork(msg); }, }); Serverless Functions as Callbacks
  31. const listener = new awsx.elasticloadbalancingv2.NetworkListener("lb",{ port: 80 }); const service

    = new awsx.ecs.FargateService("nginx", { desiredCount: 2, taskDefinitionArgs: { containers: { nginx: { image: "jonashackt/spring-boot-vuejs", memory: 512, portMappings: [listener], }}}}); Nginx on Fargate with a Load Balancer
  32. Let’s run pulumi up … Type Name Status + pulumi:pulumi:Stack

    pulumi-typescript-aws-fargate-dev created + ├─ awsx:x:ec2:SecurityGroup nginx created + │ └─ aws:ec2:SecurityGroup nginx created + ├─ awsx:x:ec2:Vpc default-vpc-accd9bc5 created + │ ├─ awsx:x:ec2:Subnet default-vpc-accd9bc5-public-1 created + │ └─ awsx:x:ec2:Subnet default-vpc-accd9bc5-public-0 created + ├─ awsx:x:ecs:FargateService spring-boot-vuejs created + │ └─ aws:ecs:Service spring-boot-vuejs created + ├─ awsx:x:ecs:FargateTaskDefinition spring-boot-vuejs created + │ ├─ aws:cloudwatch:LogGroup spring-boot-vuejs created + │ ├─ aws:iam:Role spring-boot-vuejs-execution created + │ ├─ aws:iam:Role spring-boot-vuejs-task created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-execution-9a42f520 created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-task-32be53a2 created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-task-fd1a00e5 created + │ └─ aws:ecs:TaskDefinition spring-boot-vuejs created + ├─ awsx:x:ecs:Cluster default-cluster created + │ ├─ awsx:x:ec2:SecurityGroup default-cluster created + │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule default-cluster-ssh created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-ssh created + │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule default-cluster-containers created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-containers created + │ │ ├─ awsx:x:ec2:EgressSecurityGroupRule default-cluster-egress created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-egress created + │ │ └─ aws:ec2:SecurityGroup default-cluster created + │ └─ aws:ecs:Cluster default-cluster created + └─ aws:lb:ApplicationLoadBalancer nginx created + ├─ awsx:lb:ApplicationTargetGroup nginx created + │ └─ aws:lb:TargetGroup nginx created + ├─ awsx:lb:ApplicationListener nginx created + │ ├─ awsx:x:ec2:IngressSecurityGroupRule nginx-external-0-ingress created + │ │ └─ aws:ec2:SecurityGroupRule nginx-external-0-ingress created + │ ├─ awsx:x:ec2:EgressSecurityGroupRule nginx-external-0-egress created + │ │ └─ aws:ec2:SecurityGroupRule nginx-external-0-egress created + │ └─ aws:lb:Listener nginx created + └─ aws:lb:LoadBalancer nginx created
  33. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope",

    "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Applications
  34. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope",

    "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Applications
  35. Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity)

    Managed Service Managed Service Application Application Application
  36. // Define the application Pod. const appPod = new kx.PodBuilder({

    containers: [{ image: "app:1.0.0", env: { "APP_CONFIG_PATH": "/app/config" }, volumeMounts: [configs.mount("/app/config")], }], }); // Create a Kubernetes Deployment using the previous Pod definition. const deployment = new kx.Deployment("nginx", { spec: appPod.asDeploymentSpec({ replicas: 3 }), }); // Expose the Deployment with a load balancer using a Kubernetes Service. const service = deployment.createService({ type: kx.types.ServiceType.LoadBalancer, }); Kubernetesx Library
  37. const group = require("./index").group; // check: Instances must not have

    SSH open to the Internet. it("must not open port 22 (SSH) to the Internet", done => { pulumi.all([ group.urn, group.ingress ]).apply(([ urn, ingress ]) => { if (ingress.find(rule => rule.fromPort == 22 && rule.cidrBlocks.find(block => block === "0.0.0.0/0"))) { done(new Error(`Illegal SSH port 22 open to the Internet on group ${urn}`)); } else { done(); } }); }); Unit Testing
  38. Policy as Code const policies = new PolicyPack("aws", { policies:

    [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });
  39. Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components,

    modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code