Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Best Practices for Safe Deployments on AWS Lamb...

Best Practices for Safe Deployments on AWS Lambda and Amazon API Gateway

AWS Summit, Berlin, February 27th, 2019

Deploying frequently is fundamental to reducing the feedback loop and increasing developer productivity. There are multiple features available in AWS Lambda, Amazon API Gateway, and AWS Serverless Application Model (AWS SAM) that you can use to implement a continuous deployment pipeline with safe deployment strategies, such as canary releases. In this session, we review the possible options applied to different scenarios, such as microservices architectures, chaos engineering, and A/B testing to discover the best practices for your use cases.

Danilo Poccia

February 27, 2019
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Best Practices for Safe Deployments on AWS Lambda and Amazon API Gateway Danilo Poccia Principal Evangelist, Serverless AWS @danilop
  2. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Listen Iterate Experiment Innovation Flywheel Experiments power the engine of rapid innovation
  3. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Infrastructure as code ✓ Make infrastructure changes repeatable and predictable ✓ Release infrastructure changes using the same tools as code changes ✓ Replicate production environment in a staging environment to enable continuous testing
  4. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Infrastructure as code Declarative I tell you what I need I tell you what to do Imperative
  5. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Infrastructure as code best practices ✓ Infrastructure and application in the same source repository For example: AWS CloudFormation HashiCorp Terraform ✓ Deployments include infrastructure updates
  6. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Infrastructure as code for serverless apps For example: AWS Serverless Application Model (SAM) Serverless Framework Lambda Functions DynamoDB Tables S3 Buckets API Gateways
  7. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T SAM template AWSTemplateFormatVersion: '2010-09-09’ Transform: AWS::Serverless-2016-10-31 Resources: GetFunction: Type: AWS::Serverless::Function Properties: Handler: index.get Runtime: nodejs8.10 CodeUri: src/ Policies: - DynamoDBReadPolicy: TableName: !Ref MyTable Events: GetResource: Type: Api Properties: Path: /resource/{resourceId} Method: get MyTable: Type: AWS::Serverless::SimpleTable Just 20 lines to create: • Lambda function • IAM role • API Gateway • DynamoDB table O pen Source
  8. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T SAM CLI https://github.com/awslabs/aws-sam-cli
  9. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T SAM CLI https://github.com/awslabs/aws-sam-cli
  10. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T SAM CLI sam init --name my-function --runtime python cd my-function/ sam build sam package --s3-bucket my-packages-bucket \ --output-template-file packaged.yaml sam deploy --template-file packaged.yaml \ --stack-name my-function-prod sam publish # To the AWS Serverless Application Repository
  11. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T TweetSource: Type: AWS::Serverless::Application Properties: Location: ApplicationId: arn:aws:serverlessrepo:... SemanticVersion: 2.0.0 Parameters: TweetProcessorFunctionName: !Ref MyFunction SearchText: '#serverless -filter:nativeretweets' Nested apps to simplify solving recurring problems Standard Component Custom Business Logic Polling schedule (CloudWatch Events rule) trigger TwitterProcessor SearchCheckpoint TwitterSearchPoller Twitter Search API
  12. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Serverless deployments Code Stack Package Deploy Template
  13. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Serverless deployments with a test environment Feedback Loop Production Stack Deploy Code Test Stack Package Deploy Template
  14. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T CodeDeploy – Lambda deployments Enable in your serverless application template Resources: GetFunction: Type: AWS::Serverless::Function Properties: DeploymentPreference: Type: Canary10Percent10Minutes Alarms: - !Ref ErrorsAlarm Hooks: PreTraffic: !Ref PreTrafficHook Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes AllAtOnce
  15. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function alias “live” v1 Lambda function code 100%
  16. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 100% Run PreTraffic hook against v2 code before it receives traffic v2 code 0%
  17. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 90% Wait for 10 minutes, roll back in case of alarm v2 code 10%
  18. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T CodeDeploy – Lambda canary deployment API Gateway Lambda function weighted alias “live” v1 code 0% Run PostTraffic hook and complete deployment v2 code 100%
  19. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T API Gateway canary stage API Gateway Production stage v1 code v2 code 99.5% 0.5% Canary stage
  20. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T BUSINESS LOGIC LIB B Before BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B BUSINESS LOGIC LIB A LIB B LIB A Use Lambda Layers for shared code that doesn’t change frequently
  21. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC BUSINESS LOGIC LIB A LIB B Use Lambda Layers for shared code that doesn’t change frequently Focus on your business logic and speed up function deployments After
  22. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Project Product v1 v2 v3 Customer needs
  23. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Project Product Reach milestone Customer value Lifecycle costs Cost to reach milestone Backward looking Forward looking
  24. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T “The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right.” Jeff Bezos 2016 Letter to Shareholders “Jeff, what does Day 2 look like?” That’s a question I just got at our most recent all-hands meeting. I’ve been reminding people that it’s Day 1 for a couple of decades. I work in an Amazon building named Day 1, and when I moved buildings, I took the name with me. I spend time thinking about this topic. “Day 2 is stasis. Followed by irrelevance. Followed by excruciating, painful decline. Followed by death. And that is why it is always Day 1.” To be sure, this kind of decline would happen in extreme slow motion. An established company might harvest Day 2 for decades, but the final result would still come. I’m interested in the question, how do you fend off Day 2? What are the techniques and tactics? How do you keep the vitality of Day 1, even inside a large organization? Such a question can’t have a simple answer. There will be many elements, multiple paths, and many traps. I don’t know the whole answer, but I may know bits of it. Here’s a starter pack of essentials for Day 1 defense: customer obsession, a skeptical view of proxies, the eager adoption of external trends, and high-velocity decision making. True Customer Obsession There are many ways to center a business. You can be competitor focused, you can be product focused, you can be technology focused, you can be business model focused, and there are more. But in my view, obsessive customer focus is by far the most protective of Day 1 vitality. Why? There are many advantages to a customer-centric approach, but here’s the big one: customers are always beautifully, wonderfully dissatisfied, even when they report being happy and business is great. Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf. No customer ever asked Amazon to create the Prime membership program, but it sure turns out they wanted it, and I could give you many such examples. Staying in Day 1 requires you to experiment patiently, accept failures, plant seeds, protect saplings, and double down when you see customer delight. A customer-obsessed culture best creates the conditions where all of that can happen. Resist Proxies As companies get larger and more complex, there’s a tendency to manage to proxies. This comes in many shapes and sizes, and it’s dangerous, subtle, and very Day 2. A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right. Gulp. It’s not that rare to hear a junior leader defend a bad outcome with something like, “Well, we followed the process.” A more experienced leader will use it as an opportunity to investigate and improve the process. The process is not the thing. It’s always worth asking, do we own the process or does the process own us? In a Day 2 company, you might find it’s the second. Another example: market research and customer surveys can become proxies for customers – something that’s especially dangerous when you’re inventing and designing products. “Fifty-five percent of beta testers report being satisfied with this feature. That is up from 47% in the first survey.” That’s hard to interpret and could unintentionally mislead. Resist proxies
  25. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Feedback to business Feedback Loop Production Stack Deploy Code Test Stack Package Deploy Template Business
  26. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Product Features Defects Risks Debts Product development Business Customers Security & Compliance Developers & Architects Avoid Overutilization
  27. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  28. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Capital One – Credit Offers API serverless architecture Affiliates www.capitalone.com/ credit-cards/prequalify AWS Cloud Capital One API Gateway VPC Lambda Function Traces Logs Production Support Command Center COAT Credit Offers API Team Lambda Function S3 Bucket TTL Third-Party API
  29. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Capital One – Credit Offers API CI/CD pipeline Continuous Improvement, Continuous Delivery! GitHub LGTM Bot Jenkins AWS SAM S3 Bucket (Versioning) Lambda Function DeploymentType: dev: AllAtOnce qa: AllAtOnce qaw: AllAtOnce prod: Canary10Percent10Minutes prodw: Canary10Percent10Minutes canary5xxGetProductsAlarm: Type: AWS::CloudFormation::Alarm Properties: AlarmActions: - !FindInMap: - params - AdminSNSTopic - !Ref Environment AlarmDescription: 500 error from product listing Lambda. ComparisonOperator: GreatherThanOrEqualTothreshold Period: 300 Statistic: Sum Threshold: 1 EvaluationPeriod: 1
  30. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Capital One – Benefits from taking the API serverless Performance gains From the time the request is received by lambda to the time to send the response back 70% Cost savings By removing EC2, ELB and RDS from our solution 90% Increase in team velocity Reduce investment in team’s time on DevOps and dedicate back to feature development! 30%
  31. S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  32. Thank you! S U M M I T © 2019,

    Amazon Web Services, Inc. or its affiliates. All rights reserved. Danilo Poccia @danilop