Share
Embed
A Multi AWS Account Story
It was presented during the AWS Singapore March Meetup
Steve Teo
March 14, 2017
Transcript
29 slides
-
Self-Introduction • Been working on AWS for almost 2 years
• Developer -> Release Engineer -> Cloud DevOps • Organiser of the Singapore Atlassian User Group https://www.meetup.com/Atlassian-User-Group-Singapore/ • LinkedIn: https://www.linkedin.com/in/steve-teo-b7988541/ -
Background • September 2015 • Dual Roles - Public Cloud
Engineering SME & Product Development Team • Product Engineering Team ◦ On-Prem moving to the Cloud ◦ Mixed Windows & Linux Workloads, Manual Deployment Process ◦ Charge-back was a very important consideration ◦ “Oh btw, you are the Infra Lead now” -
State of the AWS Accounts • Good: Consolidated Billing •
Good: No need to link anything back to on-prem • Bad: Multiple VPCs with conflicting or too big CIDRs. E.g. 10.0.0.0/16 • Bad: No clear purpose of various AWS accounts • Bad: Undocumented AWS Accounts • Bad: Unclear chargeback / workload purpose • Bad: Abandoned Workloads • Can’t be helped: Vendors had AWS Management IAM Access • Definite Pain Point: Conflicting or too big CIDRs -
Research • From One to Many: Evolving VPC ◦ https://www.youtube.com/watch?v=jjk_zZRLXXw
8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • VPC Peering • Blast Radius • Read passing mention of Intuit having > 1000 AWS Accounts • Action Plan ◦ VPC Segmentation ◦ AWS Account Segregation (Security / Charge-back) -
How We Separated AWS Accounts • Separate by Projects ◦
Purpose & Charge-back • Then Environments (Prod , Non-Prod) ◦ Security • Shared Services (Management), eg. Tools, AD • Consolidated Billing Master Account -
Other Ways/Motivations to Separate AWS Accounts • Direct Connect Account
• Backup Vault Account (For DR) • Logging Account • Security Account • Nature of Apps (Regulated vs Non-Regulated) • Different Departments / Teams / Chargeback • AWS Account Limit / API Rate Limits • Limit Visibility of Workloads • Organisational Changes -
New Challenges to Solve • Good: Easy to reason about
our infrastructure and sleep at night • Day 2 Concerns ◦ Creating and Securing Each AWS Account ◦ VPC Connectivity ◦ Visibility of resources ◦ All Others ▪ DNS ▪ Logs, Monitoring, Compliance & Audit ▪ Backup, Patch Management • Good to invest engineering effort to strategize and automate this! -
Creation of AWS Accounts • Every AWS Account requires a
new email address (painful process for us) • Creation was a manual process. Painful and boring • Snooped around ◦ https://github.com/intuit/aws_account_utils ◦ AWS Internal API to automate AWS Account Creation • Have a naming convention to your account • If I had to do it again ◦ Use AWS Organisations ◦ Use Email Addresses with the + trick, eg. [email protected] -
Securing AWS Root Account • Root Account ◦ Multiple Google
Authenticators with Authy - to prevent SPOF • If I had to do it again ◦ Use AWS Organisations ◦ “An new account that is created by using AWS Organizations does not initially have a password assigned to the root user.” -
Securing AWS IAM Accounts • What we saw out there
◦ STS AssumeRole from Jump Host Account ◦ AWS Managed AD was not existent yet • Our Approach ◦ AD -> Okta -> SAML Federation to every AWS Account ◦ AD was good for us because of Windows workloads ◦ Okta was good for us to manage complexity with multiple services -
DNS • More of a challenge associated with VPC Peering
as a result of AWS Accounts • We ended up using AD DNS to manage internal DNS across all VPCs, with Route53 Zones for public DNS • Considerations ◦ Route53 doesn’t integrate neatly with resources in different AWS accounts -
Logs, Monitoring, Compliance & Audit • Infrastructure Monitoring ◦ Datadog.
Cloudwatch can’t scale beyond 1 Account • Centralize Logs ◦ Cloudtrail ◦ Config ◦ S3 Logs ◦ ELB Logs ◦ Infrastructure Events ◦ CloudFront ◦ VPC Flow Logs • Leverage on ChatOps - Real-Time monitoring and discussion -
Backup, Patch Management • Backup ◦ Use Cloud Protection Management
(CPM) ◦ However: Inflexible licensing around multiple AWS Accounts • Patch Management ◦ Consider using AWS EC2 Systems Manager ◦ Or leverage on Configuration Management with Packer / Shared AMIs -
Other Complexities Found Along the Way • Consolidated Billing and
Reserved Instances ◦ Wanted selection of reserved instances specifically not against certain accounts • Reserved Instances and Capacity Reservation ◦ RIs in a central account vs RIs in separate accounts ◦ Capacity Reservation only holds if specific account and specific AZs • Resource Duplication ◦ Eg. Multiple NAT Instances - Considering using Proxy Instances • Support for Multiple AWS Accounts? - We were on Enterprise Support, so this did not bite us -
Recommended Best Practices • Evolving Architecture ◦ Make the most
important decision first, eg. VPC Segmentation ◦ “A good architecture allows major decisions to be deferred” • Document or CMDB the following ◦ AWS Accounts ◦ VPCs • Naming Strategy for at least ◦ AWS Accounts • Automate AWS Account Configuration - Terraform / CloudFormation • Centralize Everything • Secure Everything ◦ Use AWS Organisations Service Control Policies -
Recommended Best Practices • Use SaaS/Tools that can scale to
multiple AWS Accounts (eg. Licensing, Automation) • Be aware of the nature of AWS resources (eg. Account-Specific, AZ-Specific, Region-Specific, VPC-Specific) • Be aware of limits which can impact the overall architecture ◦ VPC Peering Limits ◦ Software Licenses • Look at AWS Organisations -
Summary • Think about using a multi Account Strategy in
any AWS Journey; every organisation can benefit from this • Research far and wide, think of scenarios and do POCs • Good to invest engineering effort to strategize and automate the associated challenges -
Learning Resources • From One to Many: Evolving VPC ◦
https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31 ◦ Latest Version: https://www.youtube.com/watch?v=3Gv47NASmU4 20:35 • https://aws.amazon.com/answers/account-management/aws-multi-account-se curity-strategy/ • http://www.glomex.com/blog/multi-account-handling/