Building a CI/CD pipeline on AWS

An overview of why and how to build a CI/CD pipeline using the Code-suite AWS tools

1. BUILDING A CI/CD PIPELINE ON AWS https://milancermak.com @milancermak Photo by Robert Bock on Unsplash
2. WHAT TO BUILD Holy grail A git-push-to-production pipeline, with automated rollbacks, with separated environments, with separated accounts, distributed in multiple regions Start simple Automate deployments for EVERYTHING Photo by Kobby Mendez on Unsplash
3. WHY BOTHER? Speed Focus Peace of mind GOOD FOR BUSINESS Photo by Taha Mazandarani on Unsplash
4. TOOLS OF THE TRADE CloudFormation CodeCommit & GitHub CodeBuild CodeDeploy CodePipeline Photo by Rick Mason on Unsplash
5. CODEPIPELINE
7. ACTION CONFIGURATION Action configuration depends on the action category: source, build, test, deploy, approval, invoke
8. CODEPIPELINE ARTIFACTS Artifact is a fancy word for a zip file Code from the repository Output of CodeBuild … Accessible to actions Stored in an S3 bucket
10. AN EXAMPLE Photo by Victor Garcia on Unsplash
11. PULL CODE FROM REPO
12. Name: FetchSource Actions: - Name: CodeCommit ActionTypeId: Category: Source Owner: AWS Provider: CodeCommit Version: '1' Configuration: RepositoryName: !GetAtt CodeRepository.Name BranchName: master OutputArtifacts: - Name: SourceOutput
13. MAKE THE PIPELINE UPDATE ITSELF
14. Name: UpdatePipeline Actions: - Name: PipelineStack ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM RoleArn: !GetAtt CloudformationRole.Arn StackName: !Ref AWS::StackName TemplatePath: SourceOutput::infrastructure/ pipeline.yml ParameterOverrides: !Sub '{"Service": "$ {Service}"}' InputArtifacts: - Name: SourceOutput
15. PREPARE THE DEPLOYMENT PACKAGE
16. Name: Build Actions: - Name: Functions ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: '1' Configuration: ProjectName: !Ref CodeBuildProject InputArtifacts: - Name: SourceOutput OutputArtifacts: - Name: BuildOutput
17. CODEBUILD PROJECT DECLARATION
18. CodeBuildProject: Type: AWS::CodeBuild::Project Properties: Artifacts: Type: CODEPIPELINE Cache: Location: !Sub '${ArtifactsBucket}/buildcache/ functions' Type: S3 Environment: ComputeType: BUILD_GENERAL1_SMALL EnvironmentVariables: - Name: ARTIFACTS_BUCKET Value: !Ref ArtifactsBucket Type: PLAINTEXT Image: aws/codebuild/python:3.7.1 Type: LINUX_CONTAINER ServiceRole: !GetAtt CodeBuildRole.Arn Source: BuildSpec: infrastructure/buildspec.yml Type: CODEPIPELINE TimeoutInMinutes: 15
19. CODEBUILD TASK DECLARATION
20. BUILDSPEC.YML Collection of build commands in YAML Creates artifacts that are passed further along the pipeline Tips: Print useful progress messages, run verbose programs in quiet mode Use a cache Put “complex” multiline commands in a .sh file Update aws-cli if you need newish services support Debug your build process locally with Docker
21. version: 0.2 env: variables: # ARTIFACTS_BUCKET is available from pipeline.yml PIP_CACHE: pip_cache phases: install: commands: - printenv - pip install -r requirements-dev.txt —cache-dir $PIP_CACHE pre_build: commands: - pytest -vrf tests build: commands: - aws cloudformation package --template-file infrastructure/functions.yml --output-template-file packaged_functions.yml --s3-bucket "${ARTIFACTS_BUCKET}" artifacts: files: - packaged_functions.yml cache: paths: - 'pip_cache/**/*'
22. DEPLOY BETA STACK
23. Name: Deploy Actions: - Name: Beta ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND RoleArn: !GetAtt CloudformationRole.Arn StackName: !Sub '${Service}-beta-functions' TemplatePath: BuildOutput::packaged_functions.yml ParameterOverrides: !Sub '{"Service": "${Service}", "Stage": "beta"}' InputArtifacts: - Name: BuildOutput
24. DEPLOY PRODUCTION STACK
25. Actions: - Name: Prod ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: '1' Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_IAM,CAPABILITY_AUTO_EXPAND RoleArn: !GetAtt CloudformationRole.Arn StackName: !Sub '${Service}-prod-functions' TemplatePath: BuildOutput::packaged_functions.yml ParameterOverrides: !Sub '{"Service": "${Service}", "Stage": “prod"}' InputArtifacts: - Name: BuildOutput
26. LAMBDA CANARY DEPLOYS
27. USING CODEDEPLOY Connect Lambda with CodeDeploy via a CloudWatch Alarm Typically monitor the error rate Easy with AWS SAM, but there’s too much magic Tips: Make sure the alarms works as expected! Skip gradual rollout for non-production stacks Deploy “smaller regions” (i.e. not us-east-1) first
28. Parameters: Stage: Description: Environment stage (deployment phase) Type: String AllowedValues: - beta - prod Conditions: IsBeta: !Equals [!Ref Stage, beta] Resources: HelloWorld: Type: AWS::Serverless::Function Properties: # ... AutoPublishAlias: !Ref Stage DeploymentPreference: Type: !If [IsBeta, AllAtOnce, Canary10Percent5Minutes] Alarms: - !Ref HelloWorldErrorsAlarm
29. HelloWorldErrorsAlarm: Type: AWS::CloudWatch::Alarm Properties: ComparisonOperator: GreaterThanThreshold Dimensions: - Name: Resource Value: !Sub '${HelloWorld}:${Stage}' - Name: FunctionName Value: !Ref HelloWorld - Name: ExecutedVersion Value: !GetAtt HelloWorld.Version.Version EvaluationPeriods: 2 MetricName: Errors Namespace: AWS/Lambda Period: 60 Statistic: Sum Threshold: 0
30. DEMO
31. WHAT’S NEXT? Dynamic feature pipelines Separate accounts per stage Multi-region deployments Pipeline performance metrics Deploying dependent services Photo by Samuel Zeller on Unsplash
32. THANK YOU Photo by Joel & Jasmin Førestbird on Unsplash