Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Multi AWS Account Story

A Multi AWS Account Story

It was presented during the AWS Singapore March Meetup

Steve Teo

March 14, 2017
Tweet

More Decks by Steve Teo

Other Decks in Technology

Transcript

  1. Self-Introduction • Been working on AWS for almost 2 years

    • Developer -> Release Engineer -> Cloud DevOps • Organiser of the Singapore Atlassian User Group https://www.meetup.com/Atlassian-User-Group-Singapore/ • LinkedIn: https://www.linkedin.com/in/steve-teo-b7988541/
  2. Background • September 2015 • Dual Roles - Public Cloud

    Engineering SME & Product Development Team • Product Engineering Team ◦ On-Prem moving to the Cloud ◦ Mixed Windows & Linux Workloads, Manual Deployment Process ◦ Charge-back was a very important consideration ◦ “Oh btw, you are the Infra Lead now”
  3. State of the AWS Accounts • Good: Consolidated Billing •

    Good: No need to link anything back to on-prem • Bad: Multiple VPCs with conflicting or too big CIDRs. E.g. 10.0.0.0/16 • Bad: No clear purpose of various AWS accounts • Bad: Undocumented AWS Accounts • Bad: Unclear chargeback / workload purpose • Bad: Abandoned Workloads • Can’t be helped: Vendors had AWS Management IAM Access • Definite Pain Point: Conflicting or too big CIDRs
  4. Research • From One to Many: Evolving VPC ◦ https://www.youtube.com/watch?v=jjk_zZRLXXw

    8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • VPC Peering • Blast Radius • Read passing mention of Intuit having > 1000 AWS Accounts • Action Plan ◦ VPC Segmentation ◦ AWS Account Segregation (Security / Charge-back)
  5. How We Separated AWS Accounts • Separate by Projects ◦

    Purpose & Charge-back • Then Environments (Prod , Non-Prod) ◦ Security • Shared Services (Management), eg. Tools, AD • Consolidated Billing Master Account
  6. Other Ways/Motivations to Separate AWS Accounts • Direct Connect Account

    • Backup Vault Account (For DR) • Logging Account • Security Account • Nature of Apps (Regulated vs Non-Regulated) • Different Departments / Teams / Chargeback • AWS Account Limit / API Rate Limits • Limit Visibility of Workloads • Organisational Changes
  7. New Challenges to Solve • Good: Easy to reason about

    our infrastructure and sleep at night • Day 2 Concerns ◦ Creating and Securing Each AWS Account ◦ VPC Connectivity ◦ Visibility of resources ◦ All Others ▪ DNS ▪ Logs, Monitoring, Compliance & Audit ▪ Backup, Patch Management • Good to invest engineering effort to strategize and automate this!
  8. Creation of AWS Accounts • Every AWS Account requires a

    new email address (painful process for us) • Creation was a manual process. Painful and boring • Snooped around ◦ https://github.com/intuit/aws_account_utils ◦ AWS Internal API to automate AWS Account Creation • Have a naming convention to your account • If I had to do it again ◦ Use AWS Organisations ◦ Use Email Addresses with the + trick, eg. [email protected]
  9. Securing AWS Root Account • Root Account ◦ Multiple Google

    Authenticators with Authy - to prevent SPOF • If I had to do it again ◦ Use AWS Organisations ◦ “An new account that is created by using AWS Organizations does not initially have a password assigned to the root user.”
  10. Securing AWS IAM Accounts • What we saw out there

    ◦ STS AssumeRole from Jump Host Account ◦ AWS Managed AD was not existent yet • Our Approach ◦ AD -> Okta -> SAML Federation to every AWS Account ◦ AD was good for us because of Windows workloads ◦ Okta was good for us to manage complexity with multiple services
  11. DNS • More of a challenge associated with VPC Peering

    as a result of AWS Accounts • We ended up using AD DNS to manage internal DNS across all VPCs, with Route53 Zones for public DNS • Considerations ◦ Route53 doesn’t integrate neatly with resources in different AWS accounts
  12. Logs, Monitoring, Compliance & Audit • Infrastructure Monitoring ◦ Datadog.

    Cloudwatch can’t scale beyond 1 Account • Centralize Logs ◦ Cloudtrail ◦ Config ◦ S3 Logs ◦ ELB Logs ◦ Infrastructure Events ◦ CloudFront ◦ VPC Flow Logs • Leverage on ChatOps - Real-Time monitoring and discussion
  13. Backup, Patch Management • Backup ◦ Use Cloud Protection Management

    (CPM) ◦ However: Inflexible licensing around multiple AWS Accounts • Patch Management ◦ Consider using AWS EC2 Systems Manager ◦ Or leverage on Configuration Management with Packer / Shared AMIs
  14. Other Complexities Found Along the Way • Consolidated Billing and

    Reserved Instances ◦ Wanted selection of reserved instances specifically not against certain accounts • Reserved Instances and Capacity Reservation ◦ RIs in a central account vs RIs in separate accounts ◦ Capacity Reservation only holds if specific account and specific AZs • Resource Duplication ◦ Eg. Multiple NAT Instances - Considering using Proxy Instances • Support for Multiple AWS Accounts? - We were on Enterprise Support, so this did not bite us
  15. Recommended Best Practices • Evolving Architecture ◦ Make the most

    important decision first, eg. VPC Segmentation ◦ “A good architecture allows major decisions to be deferred” • Document or CMDB the following ◦ AWS Accounts ◦ VPCs • Naming Strategy for at least ◦ AWS Accounts • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything • Secure Everything ◦ Use AWS Organisations Service Control Policies
  16. Recommended Best Practices • Use SaaS/Tools that can scale to

    multiple AWS Accounts (eg. Licensing, Automation) • Be aware of the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Be aware of limits which can impact the overall architecture ◦ VPC Peering Limits ◦ Software Licenses • Look at AWS Organisations
  17. Summary • Think about using a multi Account Strategy in

    any AWS Journey; every organisation can benefit from this • Research far and wide, think of scenarios and do POCs • Good to invest engineering effort to strategize and automate the associated challenges
  18. Learning Resources • From One to Many: Evolving VPC ◦

    https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi-account-se curity-strategy/ • http://www.glomex.com/blog/multi-account-handling/