From YAML to TypeScript: Developer’s View on Cloud Automation

Modern cloud applications consist of many moving parts, so they are hard to manage without employing Infrastructure as Code approach. Traditionally, tools like CloudFormation, ARM templates and Terraform employ text-based formats, which tend to be tedious, repetitive and difficult to reuse. What if instead of configuration files you could use TypeScript? See how you can bring your favorite developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. From YAML to TypeScript: Developer’s View on Cloud Automation Ø | November 8 | 2019
2. Why Infrastructure as Code General- purpose Languages
3. On-Premises 2
4. On-premises Load Balancer Web Server Web Server Load Balancer Database Server Database Server
5. LB Web Web DB LB DB LB Web Web DB LB DB Web DB Web + DB
6. Issues with Manual Server Management
7. Configuration management tools: Puppet/Chef
8. Benefits of Infrastructure as Code
9. Cloud Automation 6
10. Unified Management API
11. REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15
12. var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
13. az storage account create \ --location westus \ --name samplesa \ --resource-group myrg \ --sku LRS Command-line Tools
14. Scripting: Challenges
15. Desired State Configuration 8
16. Desired State Configuration
17. Managing Resource Graphs
18. Challenges
19. Desired State Provisioning
20. Markup Languages 12 Yet Another Markup Language
21. Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]
22. Converted to REST calls PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/ {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageaccount?api- version=2016-01-01 { "location": "westus", "properties": { } "sku": { "name": "Standard_LRS" }, "kind": "Storage" }
23. AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled
24. Challenges
25. resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)
26. • • • • • Providers • • • • •
27. Markup Authoring Experience
28. Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL”
29. { "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References
30. { "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals
31. locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops
32. Reusability
33. Testability
34. I think we solved these problems before?
35. IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring
36. What if developers did infrastructure?
37. Pulumi 20
38. General-Purpose Programming Languages
39. const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example
40. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
41. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
42. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
43. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Still, Desired State!
44. How Pulumi works CLI and Engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() Create, update, delete
45. • • • • • Providers • • • • •
46. Benefits of Real Code
47. Auto- completion
48. Compile Errors 48
49. for (const az of aws.getAvailabilityZones().names) { const server = new aws.ec2.Instance(`web-${az}`, { instanceType: "t2.micro", securityGroups: [ group.id ], ami: "ami-0080e4c5bc078760e", availabilityZone: az, }); } Conditionals, loops
50. class AksCluster extends pulumi.ComponentResource { constructor(name: string, args: AksClusterArgs, opts: pulumi.ComponentResourceOptions = {}) { super("examples:aks:AksCluster", name, args, opts); const password = new random.RandomPassword("password"/*, {...}*/); const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/); const adApp = new azuread.Application("aks"/*, {...}*/); const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/); const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/); const subnet = new azure.network.Subnet("subnet"/*, {...}*/); const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/) } } Components
51. AWS Cloud Development Kit (CDK) 31
52. м TypeScript Python Java .NET Compiles to CloudFormation
53. const hello = new lambda.Function(this, 'HelloHandler', { runtime: lambda.Runtime.NODEJS_8_10, code: lambda.Code.asset('lambda'), handler: 'hello.handler' }); new apigw.LambdaRestApi(this, 'Endpoint', { handler: hello }); AWS CDK example
54. Programming Languages for Infrastructure? You sure?
55. Architecture as Code 33
56. Scheduled Tasks
57. const ecsScheduledTask = new ScheduledEc2Task(stack, 'ScheduledTask’, { schedule: events.Schedule.expression('rate(1 minute)’) scheduledEc2TaskImageOptions: { image: ecs.ContainerImage.fromRegistry('amazon/amazon-ecs-sample'), memoryLimitMiB: 256, environment: { name: 'TRIGGER', value: 'CloudWatch Events' }, }, }); Scheduled Tasks
58. Zip Incoming Reports
59. const tpsReports = new aws.s3.Bucket("tpsReports"); const tpsZips = new aws.s3.Bucket("tpsZips"); tpsReports.onObjectCreated("zipTpsReports", (e) => { for (const rec of e.Records || []) { const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ]; const data = await s3.getObject({ Bucket: buck, Key: key })); zip.addFile(key, data.Body); await s3.putObject({ Bucket: tpsZips.bucket.get(), Key: `${key}.zip`, Body: zip.toBuffer(), }); } }); Functions = Callbacks
60. const tpsReports = new aws.s3.Bucket("tpsReports"); const tpsZips = new aws.s3.Bucket("tpsZips"); tpsReports.onObjectCreated("zipTpsReports", (e) => { for (const rec of e.Records || []) { const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ]; const data = await s3.getObject({ Bucket: buck, Key: key })); zip.addFile(key, data.Body); await s3.putObject({ Bucket: tpsZips.bucket.get(), Key: `${key}.zip`, Body: zip.toBuffer(), }); } }); Functions = Callbacks
61. Global Apps (1)
62. Global Apps (2)
63. Global Apps (3)
64. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Apps
65. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Apps
66. Demo! 37
67. Conclusions 37
68. Define Infrastructure as Code Making Cloud Apps?
69. Fine-grained components Better Automation
70. 70 Infrastructure is a part of your applications. Use your developer skills to build it.