Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TER301: Terraforming Safely and Collaboratively...

TER301: Terraforming Safely and Collaboratively on AWS with Atlantis

Presented during AWS Summit 2019 Developer Lounge and AWS User Group Meetup May 2019

Steve Teo

April 10, 2019
Tweet

More Decks by Steve Teo

Other Decks in Technology

Transcript

  1. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T TER301: Terraforming AWS safely and collaboratively with Atlantis Steve Teo Director of Cloud Security Engineering Horangi Cyber Security www.linkedin.com/in/steveteo D e v e l o p e r L o u n g e
  2. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Steve ”Potay” Teo • CloudDevSecOps Fanatic • Currently building products at • Progression => UI/UX Designer / Developer => Full Stack Developer => Build & Release Engineer => Infrastructure & Tools Architect • AWS Areas of Interests: AWS Multi-Account Strategy, Cloud Security • Totally uncertified and proud :P
  3. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Communities I serve www.meetup.com/AWS-SG/ www.meetup.com/ Atlassian-User-Group-Singapore/
  4. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  5. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  6. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T AWS User Group Singapore - Monthly
  7. Before we start, a quick poll. How many of you

    know what Terraform is? How many of you are already using Terraform? How many of you are using Terraform in a team setting?
  8. However, if you are starting to use Terraform in a

    team setting, it is definitely going to look like this.
  9. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T What this talk is about • Quick Introduction to Terraform • Challenges around Terraform State Management and Collaboration • How Atlantis solves these challenges safely and collaboratively • Demo What this talk is not about • Every other advanced Terraform topic • Terraform vs CloudFormation • Every Little Atlantis Feature! RTFM
  10. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources
  11. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources aws_instance aws_ec2_transit_gateway aws_db_instance
  12. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Providers • Multi-Provider Orchestration • Abstracts APIs as resources • Context for resources
  13. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Write Desired State • Think of how the world should be • Infrastructure as Code • Versionable • Specify dependencies
  14. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Plan • Preview changes before applying • Reduces mistakes & uncertainty • Map resource dependencies
  15. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Plan • Preview changes before applying • Reduces mistakes & uncertainty • Map resource dependencies
  16. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Terraform Apply • Reach the desired state • Live result log • Safe Orchestration
  17. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T One Simple Workflow
  18. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Desired State vs Current State
  19. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Desired State vs Current State
  20. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T State File
  21. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T State file is Sacred! • Real world mapping of resources • Track metadata of resource and its dependency with respect to other resources • State is exposed - Protect your state file at all cost! • Teams collaborate via state files
  22. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Local Terraform Usage Pros • Simple Issues • No backup of state files • Not even possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow
  23. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Local Terraform Usage Pros • Simple Issues • No backup of state files • Not even possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow
  24. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Solo Development
  25. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend: S3 • Durable Storage • Enable Bucket Versioning • DynamoDB Locking - State can be accessed safely
  26. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend: S3 • Durable Storage • Enable Bucket Versioning • DynamoDB Locking - State can be accessed safely
  27. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Remote Backend Terraform Usage Cons • No backup of state files • Not even really possible to collaborate in a team setting • Different developers have different Terraform environments • Doesn’t integrate into a CI/CD workflow our
  28. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Now only Ops can perform change
  29. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Now only Ops can perform change
  30. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Workflow before Atlantis
  31. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Atlantis • Started in HootSuite by Anubhav Mishra and currently maintained by Luke Kysow • Battle-tested at HootSuite with 100 developers working on 600 Terraform Repositories • Self-hosted • Open-source, supported by Hashicorp • Well written docs • Active maintainer
  32. Atlantis monitors git pull requests and centralizes Terraform Plans and

    Applies and enforces collaboration through pull requests
  33. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Monitors git pull requests
  34. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Monitors git pull requests
  35. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  36. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  37. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies
  38. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Centralizes Terraform Plans and Applies • Issue: Pull requests can become really noisy through multiple plan iterations • Solution: Iterate locally against a Sandbox account or against read only state in s3 bucket
  39. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests • Pull Request Approval Check • Merge-ability Check
  40. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests • Pull Request Approval Check • Merge-ability Check
  41. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Enforces collaboration through pull requests
  42. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Standardized Workflow after Atlantis
  43. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T { Ops } { Dev } { Ops } { Dev } { Dev } Developers Writing Infrastructure Code
  44. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Developers Writing Infrastructure Code { Ops } { Dev } { Ops } { Dev } { Dev }
  45. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  46. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Gotchas I Found - Approval / mergability checks doesn’t really work in solo based scenarios in a team based setting - Works in a very plan / apply centric model. No out of the box support for other terraform state manipulation commands, or even destroy. All this needs to be handled - Security models for working within Atlantis workflows are still being worked out, eg. https://github.com/runatlantis/atlantis/issues/308. Definitely evaluate this very carefully if you are using this for production system within a low-trust team model
  47. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  48. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  49. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T References • www.speakerdeck.com/anubhavmishra/taming-terraform-workflow- using-terraform-modules-and-github-1?slide=67 • www.speakerdeck.com/anubhavmishra/taming-infrastructure- workflow-at-scale-with-terraform • www.medium.com/runatlantis/introducing-atlantis-6570d6de7281 • www.medium.com/runatlantis/putting-the-dev-into-devops-why- your-developers-should-write-terraform-too-d3c079dfc6a8 • https://medium.com/runatlantis/moving-atlantis-to-runatlantis- atlantis-on-github-4efc025bb05f
  50. Thank you! S U M M I T © 2019,

    Amazon Web Services, Inc. or its affiliates. All rights reserved. Steve Teo Director of Cloud Security Engineering Horangi Cyber Security www.linkedin.com/in/steveteo
  51. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.