[2019.06 Meetup] [TALK #1] Stefan Killian - How to Manage Cloud Infrastructure at MAN Truck & Bus

A crucial part of DevOps is to have a platform where you can build your product. A great platform gives you the power and enables you to do more. Therefore, we use a Multi-Account AWS approach. Each Development Team gets one or more AWS accounts. However, to set up and maintain multiple AWS accounts is challenging.

MAN Truck & Bus present how they give their development teams access to AWS, how the Cloud Platform Team is collaborating with the developer teams to onboard and support them. However, the developer team will stay in the full responsibility of their account. How MAN Truck & Bus provisions new AWS Accounts, set security guard rails, and use governance features to ensure audits.

We show how we use many AWS Features like AWS Organizations, IAM, Step Functions, Lambda, Cloudformation, GuardDuty, Config, Cloudtrail, Route53 to provision new and secure AWS accounts. The whole composition is following the principle to automate everything and use continuous integration and continuous deployment where we can.

Stefan Killian is an IT Architect at MAN Truck & Bus SE in Munich, Germany. He covers topics around cloud, AWS, microservices and legacy integration platform.

Transcript

1. DEVOPS MEETUP LISBON HOW TO MANAGE CLOUD INFRASTRUCTURE Stefan Killian Photo by Alex Paganelli on Unsplash MAN | Digital Hub MAN Truck & Bus´s HQ
2. < > ŒŽ 34x + 6x + We drive transportation to the next level by creating a startup within our cooperate MAN Truck & Bus | Community Event | April 2019 | The journey of MAN Digital Hub in Lisbon 2 MAN Digital Hub: Our journey so far Jan 2018 Kick-off Nov 2017 Go! 2020 Jul 2018 Start Operations Aug 2018 New Office 100x Office Opening Oct 2018 Nov 2018 Volkswagen Press Event Today Setup, Growing, Recruiting, Scouting, Entering Scene, … We are currently setting up our Cloud Platform and CI/CD Platform Team!
3. Agenda 1 What do we want to achieve? 2 Team organization and Responsibilities 3 Cloud Platform and Blueprints – What do we provide? 4 Setup and Provisioning – How it is done? 5 Wrap up
4. WHAT DO WE WANT TO ACHIEVE AT MAN TRUCK & BUS? Photo by Andreas Brücker on Unsplash
5. < > ŒŽ What do we want to achieve at MAN Truck and Bus? DevOps in a controlled manner § Principles § Architecture § Security Guard Rails § Reduce common efforts - Do not reinvent the wheel (over and over again) DevEx (Developer Experience) § Enhance our Development/SoftwareDelivery Efforts § Enable our Product Teams (Onboarding, Consulting, Training, Education) Build and Run Cloud Native Application/Products in AWS § Apps which are born in the cloud! Developed and Maintained by MAN Truck & Bus. Our goals 5 Intro Note: This applies not to all application and/or IT organization of MAN Truck & Bus MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure
6. < > ŒŽ Architecture and Principles Principles § Cloud serverless first > Cloud fully managed > Cloud managed (> Custom-Build/Operate on EC2) § Infrastructure as Code (IaC) – every Infrastructure must be defined as IaC and must use AWS CloudFormation § CI/CD – every code artifact (also IaC) should be deployed via pipeline in Gitlab CI § Stateless Application – Apps must be stateless and follow the Twelve-Factor App methodology (Link) § Grant least privileges – give minimal amount of permissions that are required to get job done Cloud native applications only 6 What do we want to achieve at MAN Truck and Bus? MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure
7. TEAM ORGANIZATION AND RESPONSIBILITIES
8. < > ŒŽ How are we organized? Teams § Product Teams § Platform Team Shared Responsibility Model – why we need it? § An AWS Account comes with great freedom and power, but this only comes along with also a greater responsibility of operation (like high availability, backup, restore), security and many other topics § Therefore a clear understanding of the responsibility between Cloud Platform Engineering Team and the Product Team is needed MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 8 Team organization and responsibilities
9. < > ŒŽ 9 Product Team´s Task and Responsibilities Team organization and responsibilities B A Managed Database Microsservices Managed Database Microsservices Product Team 1 Product Team N C D § Are building their application in cloud native approach with containers or FaaS (AWS Lambda) and using cloud service for API Gateways or managed services for data persistence or/and queuing. § Product Team develops and deploys via our CI/CD Platform § You build it, you run it, (you secure it) - Product Team is fully resposible for their product inclusive their infrastructure (Operation) § Cost Control, Monitoring and Planning of AWS spend MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure
10. < > ŒŽ 10 Cloud Platform Engineering Team´s Task and Responsibilities Team organization and responsibilities § Providing a Platform as a product § Cloud Infrastructure with AWS Mulit-Account Setup and Provisioning of new Accounts § CI/CD Platform – (Gitlab CI, Jfrog Artifactory, Sonarqube) § Provide and maintain a common set of blueprints, basic templates and examples § Infrastructure as Code (IaC) - Templates for queuing, persistence, etc § The Cloud Platform Engineering Team offers § Onboarding Support & Consulting § Enablement § Training § Architecture Reviews § Organize community events for collaborating and contributing to the same standards MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure Cloud Platform Engineering Team
11. < > ŒŽ 11 Collaboration between Product Teams and Cloud Platform Engineering Team Team organization and responsibilities B A Managed Database Microsservices Managed Database Microsservices Product Team 1 Product Team N C D Collaboration § Inner-Source with Gitlab CI § Open issues § Request new features via issues § Open merge requests § Encourage Product Team to submit code § Community Events § Microsoft Teams MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure Cloud Platform Engineering Team
12. CLOUD PLATFORM AND BLUEPRINTS
13. < > ŒŽ What do we provide? § Accounts for platform management § Multiple AWS accounts per Product Team § One AWS Account per stage § Sandbox Account for experiments § Management (Mgmt) Account for orchestration over all stages like CI/CD § Examples § Gitlab Runner with Example Pipeline for Cross- Account Deployments § Blueprints § Runtime for containerized microservices § Runtime for serverless microservices Cloud Platform and Blueprints Mgmt Prod Int Dev Sandbox deploy deploy deploy MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure Product - Account structure per bounded context 13 Master Audit IAM Shared Services Platform Account
14. < > ŒŽ Cloud Platform Cloud Platform Engineering´s Responsibility Cloud Platform and Blueprints Hybrid VPC Private Public AWS Customer Account AWS Shared Service Account Corporate Network Private Shared Services Firewall Private Public Private Public Internet Transit Gateway AWS IAM / Audit / Control Accounts GuardDuty Config CloudTrail S3 SES Budgets IAM IAM Roles Route53 Public Hosted Zone and Resolver for MAN internal DNS Domains Route53 MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 14
15. < > ŒŽ Cloud Platform Product Team´s Responsibility Cloud Platform and Blueprints Hybrid VPC AWS Customer Account AWS Shared Service Account Corporate Network Private Shared Services Firewall Internet Transit Gateway Product Team´s Responsibility MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 15
16. < > ŒŽ Private subnet Blueprint for containerized microservices Cloud Platform and Blueprints VPC AWS Account API Gateway Container Registry (ECR) Elastic Container Service (ECS) Network Load Balancing Fargate App-Container Auto Scaling group App-Container CloudFront Public subnet Simple Storage Service (S3) Database / Persistence FrontEnd (SPA) Application Integration MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 16
17. SETUP AND PROVISIONING
18. < > ŒŽ How we manage Multi-Account AWS Environment § AWS Organizations § Organizational Units § Service Control Policies (SCP) § AWS Cloudtrail (via Organizations) § AWS Config § Account Vending Maschine - Create and Update AWS Accounts § Step Functions § Lambda § CloudFormation Automate everything! MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 18 Setup and Provisioning { "Sid": "DenyModifyDefaultTemplates", "Effect": "Deny", "Action": [ "cloudformation:Set*", "cloudformation:Cancel*", "cloudformation:Signal*", "cloudformation:Continue*", "cloudformation:Delete*", "cloudformation:Update*", "cloudformation:Stop*", "cloudformation:Execute*", "cloudformation:Create*" ], "Resource": [ "arn:aws:cloudformation:*:*:stack/gov-*/*", "arn:aws:cloudformation:*:*:stackset/gov-*:*" ], "Condition": { "StringNotLike": { "aws:PrincipalARN": [ "arn:aws:iam::*:role/man/Automation", "arn:aws:iam::*:role/man/OrgAdmin", "arn:aws:iam::*:role/AutomationStackSets" ] } } } Example of SCP
19. < > ŒŽ § IAM § Governance Features § Security § Send Notification Setup and Provisioning MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 19 Account Vending Maschine - general
20. < > ŒŽ § IAM § Governance Features § Security § Send Notification Setup and Provisioning MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 20 Account Vending Maschine - general
21. < > ŒŽ Setup and Provisioning § Budgets § DNS Delegation § VPC / Networking Note: This setup is work in progress MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure 21 Account Vending Maschine - specific
22. < > ŒŽ How we use Continuous Integration and Deployment (CI/CD) § We are dogfooding our CI/CD Platform § We are currently focusing on CI and Source Code Quality § Due to Security Restrictions we are not doing Continuous Deployments What next? § Deploying Lambda for Automation as new version without pointing the alias to the new version § Service Control Policies (SCP) should be checked with a nightly job against the saved code in the source code repository 22 Setup and Provisioning MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure
23. WRAP UP
24. < > ŒŽ Wrap up § CI/CD for Account Automation needs to be improved - currently only focus on quality, CD is work in progress § Product Teams need experience in Cloud => Training and Enablement is very important § Sharing of Information is not easy, e.g. § Usage of Blueprints and Templates § Feedback Loops – What developers need? § Platform Engineering – treat your platform as product – a great platform can enhance the software delivery § Reduce common efforts § Help onboard new teams to the cloud 24 Wrap up MAN Truck & Bus | DevOps Meetup Lisbon | Stefan Killian | June 2019 | How to Manage Cloud Infrastructure
25. THANK YOU! We are hiring for our MAN Digital Hub (Jobs)