[2020.05 Meetup] [Talk#1] João Tiago - Load testing UK’s biggest food publisher using AWS & Artillery

See the video at https://youtu.be/jZSgg9pmw2g

1. Load testing UK’s biggest food publisher using AWS & Artillery
2. João Tiago (JT) Software Engineer @ YLD @johnytiago
3. The Client • Publishing Industry • 15K+ peak active users • PHP + JS Stack • Legacy codebase • Lack of good practices
4. The Engage • 6 months project • Initial Engagement • Improve DevOps processes • Build CI/CD pipelines • Help set up Infrastructure as Code
5. • 4 months into the project • The client was going to release a new version of the application. • Dozens of new services. • No assurance it would handle production load The Spinoff
6. The Problem Release of major version of the app, running on a new infrastructure. • Lots of features being implemented • No considerations regarding performance • Teams struggled with ownership
7. Load testing engine integrated with the CI pipeline Maintainability • We won’t be at the client for long. • It as to be easy to understand; Scalability • Has to produce production-like load. Solution Requirements
8. The Solution
9. Tests Runner Why Artillery? • Flexible and lightweight. • Allows writing tests using templated YAML. • Access to Artillery's creator.
10. Tests Runner Why Artillery? • Per-environment configuration • Define SLOs and make tests fail if these are not met.
11. Project Structure • Includes all of the tests’ dependencies; • Delimited ownership; • Easier packaging and shipping. scenarios/example-api ├── local │ ├── user-ids.csv │ └── song-ids.csv ├── preproduction │ ├── user-ids.csv │ └── song-ids.csv ├── overwrites.env # Env var configuration ├── setup.sh # Setup script executed once before the tests ├── teardown.sh # Teardown script ├── helpers.js # Scenario helper functions ├── config.yml # Service endpoints and SLOs configuration └── spec.yml # Scenario description
12. Project Structure Single Github Repo • Dozens services; • Stored infra configuration; • Github Actions: • Build and push Docker image to ECR; • Scenarios synced to S3.
13. 13 aws s3 cp --recursive s3://$SCENARIOS_S3_BUCKET/$SERVICE/ . source overwrites.env npx artillery run --config config.yaml -e $ENVIRONMENT spec.yml
14. 14 Scaling Load Tests Increased the # of Artillery containers Deployed to an ECS Cluster, through a Lambda Function. You’ll get “High CPU”, Bruh
15. CI Integration CodeBuild Step • Has the service’s config which passes to the lambda; • Step locking for free; • Great place for beforeAll & afterAll hooks.
17. Inspiration Reference: https://aws.amazon.com/solutions/implementations/distributed-load-testing-on-aws/
18. 18 Local Setup
19. Results
20. 20 Results 10 RPS +100 RPS Results of running the load tests against a single instance container locally on 2Gb RAM 1vCPU
21. Concluding Remarks
22. What Was Good • Extremely exciting project to work on; • Utilised novel and refreshing tools; • Exposed and helped fix countless critical performance issues before launch; • Multiple teams were able to update their tests independently.
23. Opportunities For Improvement • We started testing later then we should. • We wrote the first set of test for all services. It was tedious . • We're not actually testing the production environment.
24. Key Takeaways • Start testing as soon as you can. • Run the load tests locally. • Make writing load tests part of your development lifecycle. • Make load testing part of your CI pipeline. • The best way to test production, is by testing production.
25. Thank you. Any questions? 25