Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Gogobot works @ RailsConf Israel 2012

Avi Tzurel
November 12, 2012

How Gogobot works @ RailsConf Israel 2012

How Gogobot works.

What drives a successful startup behind the scenes, from a technical personal and cultural point of views.

Avi Tzurel

November 12, 2012
Tweet

More Decks by Avi Tzurel

Other Decks in Programming

Transcript

  1. CDN • Edge servers across the world (hundreds) • Caches

    static assets • CSS, JS, Images • Caches full pages (with smart expire API) Monday, November 12, 12
  2. Reverse Proxy • Super fast connection from edges means users

    get a local experience, with minimal latency • Cache miss? get from load balancer • Logged out traffic almost never hits or affects logged in traffic • scale differently, control differently Monday, November 12, 12
  3. Front End • Serves user facing content • Communicates with

    cache layer and the DB • No heavy lifting, respond to the user as fast as possible Monday, November 12, 12
  4. Back End • Serves realtime services (Facebook, Twitter) • Heavy

    lifting thrown at it from the Front End Machines • Hosts workers for the queue service Monday, November 12, 12
  5. Cache • Memcached cluster • Memcached 1.4+ • 6+ Machines

    • 4.5 - 15K operations per second • Hosted on Amazon ElastiCache • Solved tons of problems with memcached dying Monday, November 12, 12
  6. MySql • Master + 2 Slaves • 64G memory for

    each machine with 400G storage • Hourly backups • Used as the main persistence layer for the site • Reads are from slaves, writes are from master • Logged out traffic will never have access the master Monday, November 12, 12
  7. MySql • EBS snapshots are used as backups • Multi

    region support for Amazon (1a, 1b, 1c) Monday, November 12, 12
  8. MongoDB • 9 Shard • 3 Replica sets in each

    shards • 16HD (100G) raid on each machine • 64G memory for each machine to provide out of memory index for all queries • Used for the Graph Engine + Scoring system Monday, November 12, 12
  9. Redis • Used for key+value store • Cache tagging solution

    on top of memcached • Queue services is hosted on Redis • Master / Slave replica • different redis cluster for different things Monday, November 12, 12
  10. Redis • Different Redis clusters for each need • Indexer

    • Cache tagging • Realtime push with Node.js to the client • When one down, others behave normally Monday, November 12, 12
  11. SOLR • Search index • NoSQL (Schema Less) • Master/Slave

    • Slave on each app machine, single master • Eventual consistency Monday, November 12, 12
  12. Numbers • 400m+ graph users • 10K triggers per user

    in the scoring system • Grew ~170X last 18 months • Announced 1m registered users 4 months ago • hit 2m registered users 2 weeks after • ~70K registrations avg per day Monday, November 12, 12
  13. Embed it in your culture • Developers should support end

    users • Get Satisfaction for example • Bugs / Problems must be engaged early and often • Be completely and utterly open Monday, November 12, 12
  14. Communication • Single and agreed line of communication for everything

    • Chatroom through the day • Pull requests for code review • Email for updates • SMS notification for urgent stuff Monday, November 12, 12
  15. What can he do? (if you ask him nicely) Or

    sudo Monday, November 12, 12
  16. Gbot • Deploy anything, anywhere, anytime • Tell Jokes •

    Remind us about bugs • Run custom builds • Alert on server issues • cheer us up Monday, November 12, 12
  17. <keyword> tweet - Returns a link to a tweet about

    <keyword> Monday, November 12, 12
  18. Monitor • Monit • CPU, Memory, Server health • God

    • process lifecycle Monday, November 12, 12
  19. Measure • NewRelic • measure performance of everything • Ruby

    • MySql • Memcached • External Services Monday, November 12, 12
  20. Sharing • Share your ideas • talk about features, future

    plans • plan together • open source your brain Monday, November 12, 12