Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load Testing with over 1 million Users [WebPerf...

Load Testing with over 1 million Users [WebPerfDays BCN]

This is the story on how I came to load test a HTTP-based API system with over 1 million concurrent users for an interactive German TV show.

I held this presentation on the WebPerfDays 2014 in Barcelona.

Sebastian Cohnen

November 21, 2014
Tweet

More Decks by Sebastian Cohnen

Other Decks in Technology

Transcript

  1. The year is 2014… • Still no global IPv6 rollout

    • But we finally have .technology, .domains, .xyz and .guru TLDs
 • TV Shows are getting interactive
  2. Quizduell • alias "QuizReto", "QuizClash", … • Mobile Quiz Game/App

    • >30M players worldwide • >14M in Germany
  3. Behind the Scenes • In-App Web View using AngularJS •

    HTTP & JSON API written in Go • Hosted on Google App Engine
  4. Nothing worked… :-/ • The very first round of “Team

    Germany” failed • Bad press; much speculation
 about hackers, [D]DoS etc.
  5. During the next days… • large scale load testing to

    provide insights • profiling, debugging, refactoring, …
  6. • This is what makes the show “interactive" • API

    polling every 1-10 sec • “server-side DDoS orchestration” (synchronized state polling & you have to answer questions within 15 sec) TV Synchronicity
  7. pre launch load tests:
 up to 85k rps (~250k Users)

    New load tests:
 Up to 330k rps (~1M Users)
  8. Issues • Google DoS Protection • Understand Google App Engine’s

    tuning & scaling knobs • Runtime environment on App Engine is not transparent
  9. The Actual Problem • Customer insisted on last minute changes

    to the backend, mostly real-time statistic related • no time to load test again prior to show premiere
  10. Test Setup • 50 Load Generators (AWS EC2 Ireland) •

    800 cores, 1.5 TB RAM, lot’s of bandwidth • 3.3 TB data moved in over 2B requests • 1M User, 330k rps peak