Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Healthy Distributed Systems

Mark Phillips
January 27, 2012

Building Healthy Distributed Systems

Distributed systems aren't limited to the world of computing. Companies and communities, for example, are also distributed networks of resources. In this talk, we'll take a high-level look at computers, companies, and communities as distributed systems and examine why maintaining their health is crucial if they are to scale.

Mark Phillips

January 27, 2012
Tweet

More Decks by Mark Phillips

Other Decks in Technology

Transcript

  1. “A distributed system consists of multiple autonomous computers that communicate

    through a computer network. The computers interact with each other in order to achieve a common goal.” [1] What is a distributed system? basho Friday, January 27, 12
  2. Distributed, Scalable, Fault Tolerant Horizontally Scalable; Add commodity hardware to

    get more [throughput | processing | storage]. basho Friday, January 27, 12
  3. Distributed, Scalable, Fault Tolerant Always Available No Single Point of

    Failure Self-healing basho Friday, January 27, 12
  4. basho { • Founded in 2007 • Collapsed in 2008

    • “Pivoted” in 2009 • Commercial Sponsors of Riak, an Open Source, NoSQL Database • Sells Closed Source Extensions to Riak in the form licenses Friday, January 27, 12
  5. 2009 2010 2011 14 60 25 Year on Year Growth

    basho Friday, January 27, 12
  6. “A distributed [company] consists of multiple autonomous [team members] that

    communicate [and collaborate] through various [channels]. The [team members] interact with each other in order to achieve a common goal.” What is a distributed [company]? basho Friday, January 27, 12
  7. Hiring where the talent is means we don’t sacrifice great

    hires for location, but it also presents various hurdles when attempting to build culture and community. basho basho Friday, January 27, 12
  8. 1. Make Basho into a Powerhouse 2. Professional Development 3.

    Employee Happiness 4. Deliver Exceptional Product Common Goals for Basho basho Friday, January 27, 12
  9. Internal Communication and Collaboration • Real-time Chat (Jabber, Camp Fire)

    • Skype (or some for of video chat) • Yammer • GitHub • AgileZen • Email (sort of) • Documentation basho Friday, January 27, 12
  10. Good Meetings basho • Quarterly In-person “Summits” • Bi-Monthly, Non-Mandatory

    Company All Hands • Stands up, Scrum Friday, January 27, 12
  11. Make Documentation Part of Your Culture basho • Inside Jokes

    • Internal Talks • Design Documents • Product Ideas • Product Feedback • New Hire Processes • Everything Else Friday, January 27, 12
  12. Open Source Your Code. And Use GitHub. basho • Contributes

    Directly to Developer Happiness • Makes Your Company’s Product Better • Great Marketing • Use a Permissive License (http://bit.ly/clJyDO) (http://bit.ly/v3OMEf) “Open Source Almost Everything” “Why Your Company Should Have a Permissive Open Source Policy” Friday, January 27, 12
  13. Poor Culture Rots a Company from within and Lessens its

    Resiliency basho Friday, January 27, 12
  14. basho Company Fault Tolerance • New CEO + Massive Growth

    = New Challenges • Our System is Constantly Improving Friday, January 27, 12
  15. “A distributed [community] consists of multiple autonomous [members] that communicate

    [and collaborate] through various [channels]. The [members] interact with each other in order to achieve a common goal.” What is a distributed [community]? basho Friday, January 27, 12
  16. Code Contributions and Bug Fixes : basho 176 names in

    our THANKS file 1600 hours contributed from Oct 2010 - Sept 2011 Friday, January 27, 12
  17. Revenue: basho 75% of new customers in 2011 came from

    the Open Source Community Friday, January 27, 12
  18. Importance of Community for Community Members basho •Working, Quality Code

    •Recognition and Praise •Desire to Contribute •Jobs (whether they like it or not) •Skills Acquisition Friday, January 27, 12
  19. Communication and Collaboration in a Distributed [Community] basho •IRC •Mailing

    List •Twitter •Riak Recap •Meetups •Q & A Sites •Blogs •Books •Conferences •Actual Meetings •GitHub •Drinking Friday, January 27, 12
  20. “A distributed system consists of multiple autonomous computers that communicate

    through a computer network. The computers interact with each other in order to achieve a common goal.” What is a distributed system? basho Friday, January 27, 12
  21. • a database • a key/value store • distributed •

    fault-tolerant • scalable • Dynamo-inspired • used by startups • used by FORTUNE 100 companies • written (primarily) in Erlang • pronounced “REE-awk” • not the right fit for every project and app basho { Friday, January 27, 12
  22. basho Common Goals for Voxer’s System 1. Serve and Receive

    App Traffic 2. Perform Queries When Needed 3. Don’t Go Down 4. Scale Out to Meet Demand 5. Low, Consistent Response Times Friday, January 27, 12
  23. Voxer’s Initial Riak Cluster Stats (Oct 2011) •11 Riak Nodes

    •Modest Data Set Size (100s of Gs) •~20,000 Peak Concurrent Users •~4,000,000 Daily Total Requests Then something happened... basho Friday, January 27, 12
  24. Voxer’s Current Riak Cluster Stats • 41 Node Cluster for

    User Data • 37 Node Cluster to serve app traffic • ~350G/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month • At one point adding three nodes/day basho Friday, January 27, 12
  25. basho Voxer’s Fault Tolerance • Have lost a lot of

    nodes in production • TCP Incast Problem [2] • LevelDB merge issues • Lots of other shit went wrong but it’s still running :) Friday, January 27, 12
  26. “Scalability is the ability of a system, network, or process,

    to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.”[3] basho Friday, January 27, 12