Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Terraform 0.10 - HUG

What's new in Terraform 0.10 - HUG

In this talk we cover an introduction to Terraform, including it's primary use cases of Infrastructure as Code, composing multiple providers (such as IaaS, SaaS, and PaaS), and enabling self service of infrastructure. We show how Terraform works and the basics of the workflow. Then we talk about new features of Terraform 0.10, with a focus on the split between core and providers, the Kubernetes provider, and workspaces. Terraform 0.9 is reviewed with a focus on destroy provisioners, remote backends, state locking.

Armon Dadgar

August 16, 2017
Tweet

More Decks by Armon Dadgar

Other Decks in Technology

Transcript

  1. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant

    Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
  2. SEP 18-20, 2017 | AUSTIN, TEXAS One day of training

    followed by two days of talks on the future of infrastructure. Discount Code: HUG20
  3. 6 HashiCorp confidential do not distribute USE CASES Easily combine

    and build complex infrastructure Self-Serve Infrastructure Common Workflow for Any Infrastructure Hybrid Cloud Management Automate Infrastructure Provisioning Infrastructure as Code
  4. 7 HashiCorp confidential do not distribute COLLABORATE on infrastructure as

    code using a version control system (VCS) VALIDATE with a common integrated workflow AUTOMATE public cloud, private cloud, and external services THE IDEAL STATE
  5. Terraform Goals • Unified view of infrastructure • Infrastructure as

    code • Compose multiple tiers (IaaS to PaaS to SaaS) • Safely change/iterate infrastructure over time • One workflow
  6. Terraform Features • Open Source • Infrastructure as Code •

    Resource Providers • Plan and Apply • Collaboration, History [Enterprise]
  7. File resource "google_compute_instance" "server" { name = "server" machine_type =

    "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
  8. File resource "google_compute_instance" "server" { name = "server" machine_type =

    "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
  9. File resource "google_compute_instance" "server" { name = "server" machine_type =

    "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
  10. Infrastructure as Code • Human-readable (HCL), Machine-editable (JSON) • Declarative

    • Text makes it versionable and VCS-friendly • Can be split into multiple files and modules
  11. Resource Providers • Integration point • Expose resources, such as

    "aws_instance", etc. • CRUD API to implement • Core vs. Providers simplifies logic for external contributors
  12. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul

    Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
  13. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul

    Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
  14. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul

    Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
  15. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul

    Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
  16. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul

    Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
  17. File resource "google_compute_instance" "server" { name = "server" machine_type =

    "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
  18. Plan • Plan shows you what will happen • Plans

    can be saved to guarantee what will happen • Plans show reasons for certain actions (such as re-create) • Not equivalent to "noop" due to the ability to save a plan
  19. Terminal $ terraform plan + google_compute_instance.server can_ip_forward: "false" disk.#: "1"

    disk.0.auto_delete: "true" disk.0.image: "ubuntu-1404-trusty-..." machine_type: "g1-small" metadata_fingerprint: "<computed>" name: "server" network_interface.#: "1" network_interface.0.address: "<computed>" network_interface.0.name: "<computed>" network_interface.0.network: "default" self_link: "<computed>" tags_fingerprint: "<computed>" zone: "us-central1-a" ... Plan: 3 to add, 0 to change, 0 to destroy.
  20. Apply • Executes changes in order based on dependencies •

    Parallelizes changes when possible • Handles and recovers transient errors
  21. Terminal $ terraform apply google_compute_instance.server: Creating... can_ip_forward: "" => "false"

    disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.image: "" => "ubuntu-1404-trusty..." machine_type: "" => "g1-small" metadata_fingerprint: "" => "<computed>" name: "" => "server" network_interface.#: "" => "1" network_interface.0.address: "" => "<computed>" network_interface.0.name: "" => "<computed>" network_interface.0.network: "" => "default" self_link: "" => "<computed>" tags_fingerprint: "" => "<computed>" zone: "" => "us-central1-a" ... Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
  22. Apply for Changes • Not only creation, but changes over

    time • Plan will show you what will happen • The `-target` flag can be used for fine-grained change
  23. Terraform 0.10 (August 2, 2017) • Split of Terraform Core

    / Providers • Kubernetes Provider • Workspaces renamed from State Environments
  24. Terraform Parts • Core Responsible for: • Parsing Configuration •

    Resource Graph • Interpolation • State Management • Execution • Providers responsible for: • Resource Definition & Validation • API Integration
  25. Terraform Repository • Everything lived in a single repository •

    Released ~2 weeks • Version Sprawl (10+ releases per major version)
  26. Mono Repo Issues • Terraform Core moves slowly • Weeks

    to implement new features • Terraform providers move fast • 70+ Providers today • 1000+ Resources • Hours to implement new resources
  27. Mono Repo Issues • Hard to triage issues • 2000+

    issues • Core Bugs to Provider Feature Requests • Hard to implement Core features • Hard to scale providers • Impossible to update Provider without updating Core
  28. Divide and Conquer • Grand Split proposed months ago •

    Split Providers into individual repositories • Release Providers on independent cadence • Download Providers as needed • Version constrain Providers • "terraform init”
  29. File provider aws { version = "~> v0.1.3" region =

    "us-west-2" } provider fastly { # Implicit: version = “latest” api_key = “exampleapikey" }
  30. Terminal $ terraform init Initializing provider plugins... - Checking for

    available provider plugins on https:// releases.hashicorp.com... - Downloading plugin for provider "fastly" (0.1.2)... - Downloading plugin for provider "aws" (0.1.4)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.fastly: version = "~> 0.1" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
  31. End Goal • Independent Versioning and Release for Providers /

    Core • Live documentation for multiple versions • Make it easier to add and support Providers • Infrastructure as Code for all the things!
  32. Kubernetes Provider • Terraform manages IaaS, SaaS, and PaaS •

    Kubernetes has sub-resources to manage (Pods, Controllers) • Compose other resources (load balancers, DNS, CDN) with K8S • Common workflow for everything
  33. Kubernetes Provider • Data Source: kubernetes_service • Data Source: kubernetes_storage_class

    • Resource: kubernetes_config_map • Resource: kubernetes_horizontal_pod_autoscalar • Resource: kubernetes_limit_range • Resource: kubernetes_namespace • Resource: kubernetes_persistent_volume • Resource: kubernetes_persistent_volume_claim • Resource: kubernetes_pod • Resource: kubernetes_replication_controller • Resource: kubernetes_resource_quota • Resource: kubernetes_secret • Resource: kubernetes_service • Resource: kubernetes_service_account • Resource: kubernetes_storage_class • …
  34. Using Schedulers with Terraform • Kubernetes • Nomad • Docker

    Swarm • AWS ECS • Manage Application Lifecycle • Compose with other resources
  35. Workspaces • A state namespace • Allows single folder of

    TF config to manage multiple distinct sets of infrastructure resources
  36. Terminal $ terraform workspace list * default $ terraform workspace

    new mitchellh-test Created and switched to workspace "mitchellh-test"! $ terraform workspace list default * mitchellh-test
  37. File resource "aws_instance" "example" { count = “${terraform.workspace == "default"

    ? 5 : 1}" tags { Name = "web - ${terraform.workspace}" } # ... other fields }
  38. Terraform 0.9 (March 15, 2017) • Destroy provisioners • Remote

    backends • State locking • Workspaces (formerly State Environments)
  39. Provisioners (Terraform <= 0.8) • Run arbitrary code locally or

    remotely on resource creation • If provisioner fails, resource is tainted and scheduled for recreation on the next apply
  40. File resource "null_resource" "example" { provisioner "local-exec" { command =

    "echo foo" } provisioner "local-exec" { command = "echo destroying" when = "destroy" } }
  41. Terminal $ terraform apply null_resource.example: Creating... null_resource.example: Provisioning with 'local-exec'...

    null_resource.example (local-exec): Executing: /bin/sh -c "echo foo" null_resource.example (local-exec): foo null_resource.example: Creation complete (ID: 1965091882910923448) Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
  42. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 1965091882910923448)

    null_resource.example: Destroying... (ID: 1965091882910923448) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "echo destroying" null_resource.example (local-exec): destroying null_resource.example: Destruction complete
  43. Destroy Provisioners • Configured with when = "destroy" • Run

    on resource destroy (not just "terraform destroy") • Failure cancels physical resource destruction by default • Can allow failure with on_failure = "continue"
  44. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 8665586891184105369)

    null_resource.example: Destroying... (ID: 8665586891184105369) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "exit 1" Error applying plan: 1 error(s) occurred: * null_resource.example (destroy): 1 error(s) occurred: * Error running command 'exit 1': exit status 1.
  45. File resource "null_resource" "example" { provisioner "local-exec" { command =

    "exit 1" when = "destroy" on_failure = "continue" } }
  46. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 8665586891184105369)

    null_resource.example: Destroying... (ID: 8665586891184105369) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "exit 1" null_resource.example: Destruction complete Destroy complete! Resources: 1 destroyed.
  47. Destroy Provisioners • Useful for resource cleanup • Can SSH

    into machine (any machine!) prior to destruction • Recommend resource cleanup live as part of the resource itself, but destroy provisioners give you another option
  48. Before Remote Backends (TF <= 0.8) • Awkward "remote config"

    command • Users could accidentally run Terraform without remote init • Configuration only via CLI • Local cache of state stored in .terraform/terraform.tfstate • Changed remote configuration was manual
  49. Terminal $ # TERRAFORM <= 0.8, BEFORE REMOTE BACKENDS $

    terraform remote config \ -backend=S3 \ -backend-config="bucket=<bucket>" \ -backend-config="key=<path to file>" ...
  50. Remote Backends • Subsumes "remote state", enables locking, workspaces, more

    • Configure from tf files, external configuration, or CLI • Detects configuration change • Forces new users of a TF configuration to initialize • One command to init them all: `terraform init`
  51. Terminal $ terraform init Initializing the backend... Successfully configured the

    backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Terraform has been successfully initialized!
  52. Terminal $ # New user, didn't run init $ terraform

    console Backend reinitialization required. Please run "terraform init". Reason: Initial configuration of the requested backend "s3" ...
  53. Terminal $ terraform console Backend reinitialization required. Please run "terraform

    init". Reason: Backend configuration changed for "s3" ...
  54. Terminal $ terraform console Backend reinitialization required. Please run "terraform

    init". Reason: Unsetting the previously set backend "s3" ...
  55. Remote Backends • One command to init: `terraform init` •

    Automatic detection of backend change (set, change, unset) • No state stored locally at all • Always gitignore ".terraform" folder
  56. A Focus on Safety • Common complaint: easy to corrupt

    remote state • Remote backends add new layer of safety: detecting changes, checking "lineage", disallowing writing unsafe state, more.
  57. A New "Init" • Init has existed since Terraform 0.1

    • Used to just setup folder structure for new projects • Now the single source of init, safe to run multiple times • Initializes backend, downloads modules, creates folders • One day: downloads providers, verifies versions, more...
  58. State Locking • For supported backends, Terraform automatically locks state

    on write operations • If unlock fails, error is shown with lock ID to allow a force unlock • Doesn't lock against concurrent reads
  59. File terraform { backend "consul" {} } resource "null_resource" "example"

    { provisioner "local-exec" { command = "sleep 10" } }
  60. Terminal $ terraform apply Error loading state: failed to lock

    state in Consul: Lock Info: ID: 5c0b66d6-018f-59b4-5536-499fec947fb2 Path: foo Operation: OperationTypeApply Who: [email protected] Version: 0.9.1 Created: 2017-04-04 16:16:59.733058195 +0000 UTC Info: $ terraform console >
  61. SEP 18-20, 2017 | AUSTIN, TEXAS One day of training

    followed by two days of talks on the future of infrastructure. Discount Code: HUG20