Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[2020.05 Meetup] [Talk#1] João Tiago - Load tes...

[2020.05 Meetup] [Talk#1] João Tiago - Load testing UK’s biggest food publisher using AWS & Artillery

See the video at https://youtu.be/jZSgg9pmw2g

DevOps Lisbon

May 18, 2020
Tweet

More Decks by DevOps Lisbon

Other Decks in Technology

Transcript

  1. The Client • Publishing Industry • 15K+ peak active users

    • PHP + JS Stack • Legacy codebase • Lack of good practices
  2. The Engage • 6 months project • Initial Engagement •

    Improve DevOps processes • Build CI/CD pipelines • Help set up Infrastructure as Code
  3. • 4 months into the project • The client was

    going to release a new version of the application. • Dozens of new services. • No assurance it would handle production load The Spinoff
  4. The Problem Release of major version of the app, running

    on a new infrastructure. • Lots of features being implemented • No considerations regarding performance • Teams struggled with ownership
  5. Load testing engine integrated with the CI pipeline Maintainability •

    We won’t be at the client for long. • It as to be easy to understand; Scalability • Has to produce production-like load. Solution Requirements
  6. Tests Runner Why Artillery? • Flexible and lightweight. • Allows

    writing tests using templated YAML. • Access to Artillery's creator.
  7. Project Structure • Includes all of the tests’ dependencies; •

    Delimited ownership; • Easier packaging and shipping. scenarios/example-api ├── local │ ├── user-ids.csv │ └── song-ids.csv ├── preproduction │ ├── user-ids.csv │ └── song-ids.csv ├── overwrites.env # Env var configuration ├── setup.sh # Setup script executed once before the tests ├── teardown.sh # Teardown script ├── helpers.js # Scenario helper functions ├── config.yml # Service endpoints and SLOs configuration └── spec.yml # Scenario description
  8. Project Structure Single Github Repo • Dozens services; • Stored

    infra configuration; • Github Actions: • Build and push Docker image to ECR; • Scenarios synced to S3.
  9. 13 aws s3 cp --recursive s3://$SCENARIOS_S3_BUCKET/$SERVICE/ . source overwrites.env npx

    artillery run --config config.yaml -e $ENVIRONMENT spec.yml
  10. 14 Scaling Load Tests Increased the # of Artillery containers

    Deployed to an ECS Cluster, through a Lambda Function. You’ll get “High CPU”, Bruh
  11. CI Integration CodeBuild Step • Has the service’s config which

    passes to the lambda; • Step locking for free; • Great place for beforeAll & afterAll hooks.
  12. 16

  13. 20 Results 10 RPS +100 RPS Results of running the

    load tests against a single instance container locally on 2Gb RAM 1vCPU
  14. What Was Good • Extremely exciting project to work on;

    • Utilised novel and refreshing tools; • Exposed and helped fix countless critical performance issues before launch; • Multiple teams were able to update their tests independently.
  15. Opportunities For Improvement • We started testing later then we

    should. • We wrote the first set of test for all services. It was tedious . • We're not actually testing the production environment.
  16. Key Takeaways • Start testing as soon as you can.

    • Run the load tests locally. • Make writing load tests part of your development lifecycle. • Make load testing part of your CI pipeline. • The best way to test production, is by testing production.