Ticketea, scaling a highly transactional business

A talk about things that we've done to scale ticketea demanding growing business given by @maraujop at Mad scalability

1. SCALING A HIGHLY TRANSACTIONAL BUSINESS
2. 2 @maraujop @ticketeaeng
3. Who are we? 3 @JavierHdez3 @maraujop @igalarzab @sullymorland @patoroco @imanolcg @RafaRM20 @iamcarlosedo @andrea_mgr @Mc_Arena_pr @javitxudedios @Maquert @gnufede @willyfrog_ @ShideShugo
4. Some numbers, peak hour stats 4 2014 6582 tickets 2015 21287 tickets
5. FRONTEND HISTORY
6. “Microservices are about scaling the number of engineers not the number of requests” Jay Kreps (Co-Creator of kafka)
7. BALANCE YOUR SYSTEM
8. A little bit of history 8
9. frontend Django 9
10. Pros 10 •5 coders worked at the company •Fast to build, easy to test and deploy •Simple stack PROS
11. Cons 11 •Workers blocked on API connections •Errors in backoffice could crash the site •Slow API means nothing loads CONS
13. Afrodita sync system 13
14. Afrodita connection system 14
15. use js
16. 2014 (350 frontends) 100x Django Frontend
17. 2015 (20 frontends) 3.5x Afrodita
18. SYSTEMS
20. CLOUD PROVIDER TICKETEA
21. Why the cloud? 21 •High availability despite of load-spikes •Scale from 3 servers to 400 in minutes •Awesome managed services
22. Why the cloud? 22 •Easier to be fault tolerant •They have great uptime •We build AMIs on deploys and use autoscaling •Preheating
23. BACKEND
24. MY LEGACY CODE IS SLOW
25. Scaling 25 MY LEGACY CODE IS SLOW •Scale In: Bigger machine •Scale Out: More machines behind a load balancer
27. Things you can do 27 •Optimize where your real bottleneck is •Cache more things •Speed up your database usage •Denormalize your database •Execute less code •Speed up your code
28. “The central enemy of reliability is complexity” Andy Bechtolsheim (Co-founder Sun Microsystems)
29. Execute less code 29 •Each time your workers are executing your code they are not able to attend new requests •Move everything not essential to background tasks, specially I/O bound jobs •Watch your timeouts
30. Database 30 •Relational databases are “almost” never the problem •Check your slowlog (create indexes) •Optimize slow or recurrent queries •Use read replicas (careful with replication lag)
32. Database contingency 32
33. Database contingency 33
34. Database contingency 34 •Wrapping everything in a transaction is not cost effective •Deadlocks are tricky to spot •Caching data in this situation can be counter-productive
35. Database contingency 35 •Shard counters •Use another technology, like REDIS (consistency)
36. Stop unnecessary background jobs
37. Find more about us 37 http://www.infoq.com/articles/monolith-to-multilith https://speakerdeck.com/ticketeaeng https://twitter.com/ticketeaeng http://engineering.ticketea.com
38. THANKS