How Gogobot works @ RailsConf Israel 2012

How Gogobot works.

What drives a successful startup behind the scenes, from a technical personal and cultural point of views.

1. ݆ૉ How Gogobot Works? Avi Tzurel Kenso Monday, November 12, 12
2. Who Am I? http://twitter.com/kensodev http://github.com/kensodev http://avi.io http://kensodev.com Monday, November 12, 12
3. I Scale Gogobot Monday, November 12, 12
4. What is Gogobot? Monday, November 12, 12
5. Social Recommendation Engine For Travel Monday, November 12, 12
6. Personalized Recommendations From Friends, not strangers Monday, November 12, 12
7. The Gogobot Architecture Monday, November 12, 12
8. How we make shit work? Monday, November 12, 12
9. CDN + Reverse Proxy Monday, November 12, 12
10. CDN • Edge servers across the world (hundreds) • Caches static assets • CSS, JS, Images • Caches full pages (with smart expire API) Monday, November 12, 12
11. Reverse Proxy • Super fast connection from edges means users get a local experience, with minimal latency • Cache miss? get from load balancer • Logged out traffic almost never hits or affects logged in traffic • scale differently, control differently Monday, November 12, 12
12. CDN + Reverse Proxy Monday, November 12, 12
13. CDN + Reverse Proxy Front End Back End Monday, November 12, 12
14. Front End • Serves user facing content • Communicates with cache layer and the DB • No heavy lifting, respond to the user as fast as possible Monday, November 12, 12
15. Back End • Serves realtime services (Facebook, Twitter) • Heavy lifting thrown at it from the Front End Machines • Hosts workers for the queue service Monday, November 12, 12
16. CDN + Reverse Proxy Front End Back End Monday, November 12, 12
17. Front End Back End Cache Monday, November 12, 12
18. Cache • Memcached cluster • Memcached 1.4+ • 6+ Machines • 4.5 - 15K operations per second • Hosted on Amazon ElastiCache • Solved tons of problems with memcached dying Monday, November 12, 12
19. Front End Back End Cache Monday, November 12, 12
20. Cache MySql MongoDB Redis SOLR Monday, November 12, 12
21. MySql • Master + 2 Slaves • 64G memory for each machine with 400G storage • Hourly backups • Used as the main persistence layer for the site • Reads are from slaves, writes are from master • Logged out traffic will never have access the master Monday, November 12, 12
22. MySql • EBS snapshots are used as backups • Multi region support for Amazon (1a, 1b, 1c) Monday, November 12, 12
23. MongoDB • 9 Shard • 3 Replica sets in each shards • 16HD (100G) raid on each machine • 64G memory for each machine to provide out of memory index for all queries • Used for the Graph Engine + Scoring system Monday, November 12, 12
24. Redis • Used for key+value store • Cache tagging solution on top of memcached • Queue services is hosted on Redis • Master / Slave replica • different redis cluster for different things Monday, November 12, 12
25. Redis • Different Redis clusters for each need • Indexer • Cache tagging • Realtime push with Node.js to the client • When one down, others behave normally Monday, November 12, 12
26. SOLR • Search index • NoSQL (Schema Less) • Master/Slave • Slave on each app machine, single master • Eventual consistency Monday, November 12, 12
27. Numbers • 400m+ graph users • 10K triggers per user in the scoring system • Grew ~170X last 18 months • Announced 1m registered users 4 months ago • hit 2m registered users 2 weeks after • ~70K registrations avg per day Monday, November 12, 12
28. Numbers • HUGE growth in a relatively short time Monday, November 12, 12
29. 99.9% uptime Monday, November 12, 12
30. ~200ms avg server response time Monday, November 12, 12
31. Logged out user gets a page in 30ms Monday, November 12, 12
32. How can you manage all of this? Monday, November 12, 12
33. Not too technical Monday, November 12, 12
34. Embed it in your culture • Developers should support end users • Get Satisfaction for example • Bugs / Problems must be engaged early and often • Be completely and utterly open Monday, November 12, 12
35. Communication • Single and agreed line of communication for everything • Chatroom through the day • Pull requests for code review • Email for updates • SMS notification for urgent stuff Monday, November 12, 12
36. No QA! Monday, November 12, 12
37. TDD Monday, November 12, 12
38. Twitter Driven Deployment Monday, November 12, 12
39. Use UTest for manual testing Daily Monday, November 12, 12
40. Chatroom Monday, November 12, 12
41. Monday, November 12, 12
42. Keep others involved Monday, November 12, 12
43. Ask Questions Monday, November 12, 12
44. AUTOMATE! Monday, November 12, 12
45. Build and Deploy Monday, November 12, 12
46. Monday, November 12, 12
47. the Gbot Monday, November 12, 12
48. gbot deploy production Monday, November 12, 12
49. What can he do? (if you ask him nicely) Or sudo Monday, November 12, 12
50. Gbot • Deploy anything, anywhere, anytime • Tell Jokes • Remind us about bugs • Run custom builds • Alert on server issues • cheer us up Monday, November 12, 12
51. Monday, November 12, 12
52. <keyword> tweet - Returns a link to a tweet about <keyword> Monday, November 12, 12
53. Monitor Monday, November 12, 12
54. Monitor • Monit • CPU, Memory, Server health • God • process lifecycle Monday, November 12, 12
55. Not just Up / Down Monday, November 12, 12
56. Measure Monday, November 12, 12
57. Measure • NewRelic • measure performance of everything • Ruby • MySql • Memcached • External Services Monday, November 12, 12
58. Monday, November 12, 12
59. Monday, November 12, 12
60. Share! Monday, November 12, 12
61. Share internally and externally Monday, November 12, 12
62. Monday, November 12, 12
63. github.com/gogobot/laptop Monday, November 12, 12
64. Chef Recipes Monday, November 12, 12
65. Sharing • Share your ideas • talk about features, future plans • plan together • open source your brain Monday, November 12, 12
66. Thank you! Monday, November 12, 12
67. Questions? Monday, November 12, 12