Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Healthy Distributed Systems

Building Healthy Distributed Systems

Slides from my "Building Healthy Distributed Systems" talk delivered at Erlang Factor in March 2012 in San Francisco.

Video here - https://vimeo.com/43027087

Mark Phillips

March 29, 2012
Tweet

More Decks by Mark Phillips

Other Decks in Technology

Transcript

  1. “A distributed system consists of multiple autonomous computers that communicate

    through a computer network. The computers interact with each other in order to achieve a common goal.” [1] What is a distributed system? basho Thursday, March 29, 12
  2. Distributed, Scalable, Fault Tolerant Horizontally Scalable; Add commodity hardware to

    get more [throughput | processing | storage]. basho Thursday, March 29, 12
  3. Distributed, Scalable, Fault Tolerant Always Available No Single Point of

    Failure Self-healing basho Thursday, March 29, 12
  4. basho { • Founded in 2007 • Collapsed in 2008

    • “Pivoted” in 2009 • Commercial Sponsors of Riak, an Open Source, NoSQL Database • Sells Closed Source Extensions to Riak in the form licenses Thursday, March 29, 12
  5. 2009 2010 2011 14 60 25 Year on Year Growth

    basho Thursday, March 29, 12
  6. “A distributed [company] consists of multiple autonomous [team members] that

    communicate [and collaborate] through various [channels]. The [team members] interact with each other in order to achieve a common goal.” What is a distributed [company]? basho Thursday, March 29, 12
  7. Hiring where the talent is means we don’t sacrifice great

    hires for location, but it also presents various hurdles when attempting to build culture and community. basho basho Thursday, March 29, 12
  8. 1. Make Basho into a Powerhouse 2. Professional Development 3.

    Employee Happiness 4. Deliver Exceptional Product Common Goals for Basho basho Thursday, March 29, 12
  9. Internal Communication and Collaboration • Real-time Chat (Jabber, Camp Fire)

    • Skype (or some for of video chat) • Yammer • GitHub • AgileZen • Email (sort of) • Documentation basho Thursday, March 29, 12
  10. Good Meetings basho • Quarterly In-person “Summits” • Bi-Monthly, Non-Mandatory

    Company All Hands • Stands up, Scrum Thursday, March 29, 12
  11. Make Documentation Part of Your Culture basho • Inside Jokes

    • Internal Talks • Design Documents • Product Ideas • Product Feedback • New Hire Processes • Everything Else Thursday, March 29, 12
  12. Open Source Your Code. And Use GitHub. basho • Contributes

    Directly to Developer Happiness • Makes Your Company’s Product Better • Great Marketing • Use a Permissive License (http://bit.ly/clJyDO) (http://bit.ly/v3OMEf) “Open Source Almost Everything” “Why Your Company Should Have a Permissive Open Source Policy” Thursday, March 29, 12
  13. Poor Culture Rots a Company from within and Lessens its

    Resiliency basho Thursday, March 29, 12
  14. basho Company Fault Tolerance • New CEO + Massive Growth

    = New Challenges • Our System is Constantly Improving Thursday, March 29, 12
  15. “A distributed [community] consists of multiple autonomous [members] that communicate

    [and collaborate] through various [channels]. The [members] interact with each other in order to achieve a common goal.” What is a distributed [community]? basho Thursday, March 29, 12
  16. Code Contributions and Bug Fixes : basho 180 names in

    our THANKS file 1600 hours contributed from Oct 2010 - Sept 2011 Thursday, March 29, 12
  17. Revenue: basho 75% of new customers in 2011 came from

    the Open Source Community Thursday, March 29, 12
  18. Importance of Community for Community Members basho •Working, Quality Code

    •Recognition and Praise •Desire to Contribute •Jobs (whether they like it or not) •Skills Acquisition Thursday, March 29, 12
  19. Communication and Collaboration in a Distributed [Community] basho •IRC •Mailing

    List •Twitter •Riak Recap •Meetups •Q & A Sites •Blogs •Books •Conferences •Actual Meetings •GitHub •Drinking Thursday, March 29, 12
  20. “A distributed system consists of multiple autonomous computers that communicate

    through a computer network. The computers interact with each other in order to achieve a common goal.” What is a distributed system? basho Thursday, March 29, 12
  21. • a database • a key/value store • distributed •

    fault-tolerant • scalable • Dynamo-inspired • used by startups • used by FORTUNE 100 companies • written (primarily) in Erlang • pronounced “REE-awk” • not the right fit for every project and app basho { Thursday, March 29, 12
  22. basho Common Goals for Voxer’s System 1. Serve and Receive

    App Traffic 2. Perform Queries When Needed 3. Don’t Go Down 4. Scale Out to Meet Demand 5. Low, Consistent Response Times Thursday, March 29, 12
  23. Voxer’s Initial Riak Cluster Stats (Oct 2011) •11 Riak Nodes

    •Modest Data Set Size (100s of Gs) •~20,000 Peak Concurrent Users •~4,000,000 Daily Total Requests Then something happened... basho Thursday, March 29, 12
  24. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data basho Thursday, March 29, 12
  25. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data • >40 Node Cluster to serve app traffic basho Thursday, March 29, 12
  26. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily basho Thursday, March 29, 12
  27. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak basho Thursday, March 29, 12
  28. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month basho Thursday, March 29, 12
  29. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for

    User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month • At one point adding three nodes/day basho Thursday, March 29, 12
  30. basho Voxer’s Fault Tolerance • Have lost a lot of

    nodes in production • TCP Incast Problem [2] • LevelDB merge issues • Lots of other shit went wrong but it’s still running :) Thursday, March 29, 12
  31. “Scalability is the ability of a system, network, or process,

    to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.”[3] basho Thursday, March 29, 12