Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load & Performance Testing the Cloud??!!

Load & Performance Testing the Cloud??!!

At AWS Summit 2016 in Berlin I held a session on why performance testing is still relevant in the cloud.

Abstract:

The Cloud™ is infinite and scalable. Period.
Why is it important to test for performance and scalability characteristics of a cloud-based system? Won't AWS scale for me as long as I can afford it to?
Yes, but… AWS only operates and scales resources. They won't automatically make your system fast, stable and – more importantly – scalable. Performance testing is crucial to understand your system and architecture design and your cloud hosting environment.

Sebastian Cohnen

April 18, 2016
Tweet

More Decks by Sebastian Cohnen

Other Decks in Technology

Transcript

  1. © 2016, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Sebastian Cohnen, Founder & CTO StormForger.com 2016-04-12 Load & Performance Testing the Cloud??!!
  2. EHLO! • 7+ years consulting and development • Focus on

    web performance and architectures • Founder & CTO StormForger.com Sebastian Cohnen (@tisba)
  3. Performance Performance is the ability of a system to fulfill

    a task within defined dimensions. It is a measurement for efficiency and can describe the relative capacity of a system. “1 instance can handle 250 rps with p99 at 50ms…”
  4. Performance …is a measurement for efficiency and can describe the

    relative capacity of a system. “1 instance can handle 250 rps with p99 at 50ms…” Performance & Scalability Scalability …will tell you if and how effectively you can grow capacity by adding resources. “10 instances can handle 2500 rps with p99 at 50ms…”
  5. “In software engineering, performance testing is in general, a testing

    practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload.” – Wikipedia https://en.wikipedia.org/wiki/Software_performance_testing
  6. Performance Testing • A family of non-functional testing methods… •

    which induce a well defined workload • in order to observe the systems behavior • in order to verify and understand its characteristics
  7. Performance Testing Family • Load Testing • Stress Testing •

    Spike Testing • Soak & Endurance Testing • Resilience Testing • Troubleshooting • Configuration Testing • Scalability Testing
  8. The AWS Cloud • IaaS, PaaS, … • APIs and

    automation • On-demand • Cost-effective • Scalable
  9. Scaling Resources ≠ Scaling Applications • Adding resources does not

    necessarily scale your application • e.g. adding more EC2 Instances won't remove the bottleneck at your RDS Elastic Load Balancing Amazon EC2 MySQL DB instance Auto Scaling group
  10. You need to understand… • Your (distributed, μService, …) Application

    • Your Software Architecture • Your Cloud Environment • Used Services and their behavior
  11. Complexity • Complexity has not simply vanished • XaaS &

    Co have moved it “somewhere else” • Impact on performance is still very important
  12. Load Testing • Induce normal/expected workload to a given system

    and observe! • Determine latency, throughput, capacity, … of your system • Ideally: Figure out the capacity per resource (e.g. per EC2 Instance)
  13. Scalability Testing • How does my capacity increase with additional

    resources? • Requirement for Capacity Planning and Cost Estimation
  14. Stress, Spike and Soak Testing • How does your system

    and environment behave under serious stress? (Stress Testing) • What happens when I'm on reddit/hacker news? (Spike Testing) • What happens over longer period of times? (Soak Testing)
  15. Configuration Testing • How does the behavior change, if the

    configuration changes? • Very useful to learn about the impact of environment to your system
  16. Configuration Testing • Instance Types • Auto Scaling • Throughput

    Provisioning • Correct/Optimal Service usage? • OS, Web/App Server Configuration
  17. Availability & Resilience Testing • (Zero-Downtime) Deployments under load •

    Infrastructure changes • Understanding failure scenarios • Testing failover mechanisms • Chaos Engineering
  18. Yes and No! • The testing needs and methods have

    not changed! • BUT: Multiple, scalable performance test environments used to be cumbersome and very expensive • You should utilize automation wherever you can: • Infrastructure, Services, Servers, Code • Test Data(!)
  19. Conclusion • Scaling Resources is not Scaling Applications • Understanding

    is key, and testing is one tool to get there • Complexity has not vanished, it has been shifted • Utilize the cloud to make test setups simpler and achievable! • …and…
  20. Proof of Concept • Design architecture on AWS • Implement

    a prototype • Discuss with AWS Solution Architects • Validate! Run tests, observe, learn and improve!