Share
Scaling Infrastructure with Pulumi
TechMeetup Conference 2023 - Ostrava (Czech Republic)
Every company goes through a journey of growing their services and the infrastructure that needs to support it. The right tools are needed to support this journey. By leveraging regular programming languages, Pulumi supports you at every step in this growth trajectory.
In this talk, Ringo will sketch how Pulumi supports your team in integrating the infrastructure setup at every step of the way.
Some of the questions which will be answered during the talk:
* why did Pulumi choose regular programming languages?
* how can this still be declarative?
* am I limited to a single programming language for all my infrastructure code?
* how can I reuse my infrastructure code over different teams?
* how can I set up secure & compliant infrastructure?
* what if I want to build a custom developer platform?
This talk will show you solutions ranging from the smallest Pulumi infrastructure program up to a fully integrated self-service platform setup.
Transcript
61 slides
-
TechMeetup Conference 2023 Ostrava 3 November 2023 Ringo De Smet
Mastodon: @[email protected] Twitter/X: @ringods -
2 On-Premises Datacenters (Server-Centric) Lift-and-Shift Migrations to Cloud (Server-Centric) D
ocker AW S Lam bda TODAY Cloud Engineering AW S EC 2 (IaaS) R estful A P Is TIPPING POINT: Legacy tools and processes were not designed for modern applications and cloud services, stifling innovation. G itH ub Web-Scale IT (Server-Centric) Modern Cloud-Native Apps (Service-Centric) D evO ps Infrastructure-as-Code Cloud-as-Software Managing Servers Focus on Building Infrastructure & Operating IT Managing Services Focus on Composing & Consuming Cloud Services Server Config Management M icroservices Kubernetes A zure G C P 2010 2000 2023 The Rise of Modern Cloud Cloud complexity is escalating -
3 Yesterday’s tools and practices don’t scale in the modern
cloud era Legacy Tools Built for the speed and scale of yesterday Modern Tools Built for high speed and massive scale -
4 Getting Started A first full environ- ment Growing your
portfolio Any protection ? Growing your company How does Pulumi support you in your growth? -
6 Multi-Cloud Infrastructure as Code BUILD, DEPLOY, AND MANAGE ANYWHERE
WITH A STANDARD WORKFLOW Infra Providers Cloud Native Clouds Core Features: • Any Cloud, Any Language • State Management • Secrets Management • Guaranteed Preview Plans • CI/CD Integrations • Webhooks • REST API • Automation API • Dashboards and Reports -
7 Pulumi - High Level View • Pulumi Architecture •
Language Host • CLI & Engine • Providers • State Backend •Pulumi Service •Cloud Storage (S3, …) -
8 Pulumi - High Level View • Automate your infrastructure
setup with 1 or more Pulumi projects • A project contains • a program: this is your “template” from which you can create stacks • the stacks: each stack is an “instance” of the program • One project can use info from another project as input -
9 How Pulumi Works - first run Pulumi CLI &
Engine $ pulumi up -
10 How Pulumi Works - first run Language Host Pulumi
CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup -
11 How Pulumi Works - first run Language Host Pulumi
CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG -
12 How Pulumi Works - first run Language Host Pulumi
CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CREATE SG web-sg: … Actual SG info -
13 How Pulumi Works - second run Pulumi CLI &
Engine $ pulumi up web-sg: … -
14 How Pulumi Works - second run Language Host Pulumi
CLI & Engine $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup web-sg: … -
15 How Pulumi Works - second run Language Host Pulumi
CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start aws.ec2. SecurityGroup provider: aws Start CHECK SG web-sg: … -
16 How Pulumi Works - multiple resources Pulumi CLI &
Engine $ pulumi up -
17 How Pulumi Works - multiple resources Language Host Pulumi
CLI & Engine $ pulumi up runtime: nodejs Start SG & Instance -
18 How Pulumi Works - multiple resources Language Host Pulumi
CLI & Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG & CREATE Instance -
20 How Pulumi Works - dependencies Language Host Pulumi CLI
& Engine $ pulumi up runtime: nodejs Start SG & Instance -
21 How Pulumi Works - dependencies Language Host Pulumi CLI
& Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start SG & Instance provider: aws Start CREATE SG CREATE Instance Wait -
23 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod
org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod -
24 Pulumi Stacks • Pulumi.<stack>.yaml • Configuration for an “instance”
of your project • Input values specific to your instance • Namespaces • get/require • Structured configuration -
26 Pulumi Stacks • Stack References • Can use values
from other stacks • Info retrieved from the state backend -
27 Pulumi at Scale org/network/dev org/network/qa org/network/prod org/cluster/dev org/cluster/qa org/cluster/prod
org services svc1-pro d org services svc2-pro d org services svc3-pro d org services svc1-qa org services svc2-qa org services svc3-qa org services svc1-dev org services svc2-dev org services svc3-dev network cluster services dev qa prod vpc-out-sg service-sg clust-sg-rule svc2-service-rule svc2-out-rule -
29 Pulumi - ComponentResource Language Host Pulumi CLI & Engine
Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start Network + children provider: aws Start CREATE … -
30 Pulumi - Multi Language Component Language Host Pulumi CLI
& Engine Pulumi Provider (e.g. aws) $ pulumi up runtime: nodejs Start org.nw.Network provider: aws Start CREATE … MLC (e.g. org) Start Nw resources -
32 Policy as Code What is it? • Enforce security,
compliance and best practices • Use general purpose programming languages* • Write expressive policies • Enforce gated deployments • Meet business requirements -
33 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
34 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
35 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
36 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
37 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
38 Policy as Code import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; new PolicyPack("aws-security-basics", { policies: [{ name: "s3-no-public-read", description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => { if (bucket.acl === "public-read" || bucket.acl === "public-read-write") { reportViolation( "You cannot set public-read or public-read-write on an S3 bucket. "); } }), },{ name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.ec2.SecurityGroup, (sg, args, reportViolation) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { reportViolation("Ingress rules with public internet access are prohibited."); } }), }], }); Security Maintain security across all clouds, all applications and resource types -
39 Policy as Code Data protection import * as aws
from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], }); -
40 Policy as Code Data protection import * as aws
from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], }); -
41 Policy as Code Data protection import * as aws
from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-location", { policies: [ { name: "required-storage-versioning", description: "Versioning is required.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { if (! bucket.versioning) { reportViolation("Bucket versioning must be enabled."); } }), }, ], }); -
42 Policy as Code Cost optimization import * as aws
from "@pulumi/aws"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("aws-cost", { policies: [ { name: "expire-old-logs", description: "Expire old logs.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, _, reportViolation) => { bucket.lifecycleRules?.forEach(rule => { if (! rule.enabled || !rule.expiration || !rule.expiration.days || rule.expiration.days < 45) { reportViolation("Bucket logs must an expiration days < 45."); } }); }), }, ], }); -
43 Policy as Code Reliability import * as k8s from
"@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-reliability", { policies: [ { name: "minimum-replica-count", description: "Checks that Kubernetes Deployments and ReplicaSets have at least three replicas.", enforcementLevel: "mandatory", validateResource: [ validateResourceOfType(k8s.apps.v1.Deployment, (deployment, _, reportViolation) => { if (!deployment.spec || !deployment.spec.replicas || deployment.spec.replicas < 3) { reportViolation("Kubernetes Deployments should have at least three replicas."); } }), validateResourceOfType(k8s.apps.v1.ReplicaSet, (replicaSet, _, reportViolation) => { if (!replicaSet.spec || !replicaSet.spec.replicas || replicaSet.spec.replicas < 3) { reportViolation("Kubernetes ReplicaSet should have at least three replicas."); } }), ], }, ], }); -
44 Policy as Code Best Practices Learn, codify, share and
automatically reapply. import * as k8s from "@pulumi/kubernetes"; import { PolicyPack, validateResourceOfType } from "@pulumi/policy"; const policies = new PolicyPack("k8s-best", { policies: [ { name: "pods-are-prohibited", description: "Checks that Kubernetes Pods are not being used directly.", enforcementLevel: "mandatory", validateResource: validateResourceOfType(k8s.core.v1.Pod, (_, __, reportViolation) => { reportViolation("Kubernetes Pods should not be used directly. " + "Instead, you may want to use a Deployment, ReplicaSet or Job."); }), }, ], }); -
45 Policy Remediations Don’t just issue policy violations, fix them
automatically • Automatically remediate configuration violations at deployment time • Add `transform` method to policy pack and Pulumi IaC will enforce it • Available in the open source IaC with organization-wide enforcement for Pulumi Business Critical customers Examples • Apply tags to resources automatically • Remove public Internet access from firewalls and gateways • Apply encryption and backups to storage New -
46 CrossGuard What is it? • Author & Publish Policy
Packs • Policy Pack versioning • Applies to any runtime (mix and match) • Policy packs and stack grouping -
47 CrossGuard Policy validation • Fetch Policy packs from the
Pulumi Service (SaaS & Self-Hosted) • Check resource properties (graph) • Evaluation before deployment Benefits • Catch problem before they reach production • Educate developers on best practices -
48 CrossGuard Policy Pack configuration • Policy pack version selection
• Per pack and per rule override -
49 Compliance-Ready Policies 100s of new out of the box
policies • PCI DSS, ISO 27001, SOC 2, CIS • All Pulumi Languages and Clouds supported • Build custom policy packs. Choose cloud provider X service X severity X compliance framework New -
52 Pulumi Developer Portal Out of the box golden path
to production for developers • Private organization templates • Simple user interface to browse templates and deploy them through a new project wizard • Deploy via Pulumi CLI or Pulumi Deployments • Integrate with Git for instant project creation • Pulumi ESC for secrets/config • Extensible with Pulumi Cloud REST API New -
53 Pulumi Backstage • Integrate Pulumi into CNCF Backstage Portals
◦ Deploy infrastructure with Pulumi programs ◦ Use private templates from Pulumi Developer Portal • Open Source • Available in Backstage Marketplace https://backstage.io/plugins New -
55 Pulumi ESC Architecture EXTENSIBLE WITH DOZENS OF SOURCES AND
TARGETS ESC Environments (Hierarchies of Config/Secrets) 🔐Native Secrets Third party secrets: (and more…) Sources Native CLI Execution environments: (and more…) Targets # shopping-service-production imports: - aws-production - shopping-service # aws-production config: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service config: stripeKey: value: ***** secret: true dbPassword: provider: workers-kv inputs: Auth Tokens RBAC Identity Managed or OSS Encryption Audit Logs Web UX Policy Versioning Sync (future) New -
56 Pulumi IaC Under the Hood Example # shopping-service-production imports:
- aws-production - shopping-service values: dbPassword: ABCDEF # aws-production values: aws: provider: aws-oidc inputs: roleArn: …arn… sessionName: aws-prod # shopping-service values: stripeKey: value: ***** secret: true dbPassword: provider: 1password inputs: # Pulumi.prod.yaml environment: - shopping-service-production $ esc run acme/shopping-service-production -- aws s3 ls import * as esc from ‘pulumi-esc’; const env = new esc.Environment(...); const config = env.open("shopping-service-production"); env.watch("shopping-service-production", config => …); Consuming from: CLI SDK AWS 1password -
58 Migrate from Terraform to Pulumi HCL $ git checkout
<your-hcl> $ pulumi convert --from terraform --language typescript --out pulumi Blog State $ pulumi stack init $ pulumi import --from terraform ./terraform.tfstate Blog New