Riak Lunch'n'learn June 2013

A short talk about Riak, Basho & Dynamo, Presented at a Techcube Lunch'n'learn.

Transcript

1. Sam Elliott - Basho Techcube - 2013/06
2. Riak What is Riak? Riak is an open source distributed, key-value database. Riak is architected for Availability, Fault-Tolerance, Operational Simplicity, and Scalability.
3. Basho & Riak • Basho originally wrote Riak • Riak is now Open-Source • Developers employed by Basho • Basho sell Enterprise Features & Support
4. Dynamo Dynamo: Amazon’s Highly Available Key-value Store This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazons core services use to provide an always-on experience.
5. CAP Theorem • Consistency • Availability • Partition-tolerance
6. CAP Theorem • Consistency • Availability • Partition-tolerance
7. CAP Theorem • Consistency • Availability • Partition-tolerance
8. CAP Theorem • Consistency • Availability • Partition-tolerance Eventual Consistency A value will eventually see all its updates, once all partition repairs are complete.
9. Consistent Hashing 0 2160
10. Consistent Hashing 0 2160 hash(<<Bucket>>,<<Key>>)
11. Consistent Hashing 0 2160 hash(<<Bucket>>,<<Key>>) riak1.example.com
12. Consistent Hashing 0 2160 hash(<<Bucket>>,<<Key>>) riak1.example.com
13. Consistent Hashing 0 2160 riak1.example.com
14. Replicas & Quorums 0 2160 hash(<<Bucket>>,<<Key>>) read(<<Bucket>>, <<Key>>)
15. Replicas & Quorums 0 2160 hash(<<Bucket>>,<<Key>>) read(<<Bucket>>, <<Key>>, N=3)
16. Replicas & Quorums 0 2160 hash(<<Bucket>>,<<Key>>) read(<<Bucket>>, <<Key>>, N=3)
17. Replicas & Quorums 0 2160 hash(<<Bucket>>,<<Key>>) read(<<Bucket>>, <<Key>>, N=3, R=2)
18. Replicas & Quorums 0 2160 hash(<<Bucket>>,<<Key>>) read(<<Bucket>>, <<Key>>, N=3, R=2)
19. Quorum Options n r w Also: Primary & Durable
20. Nodes & Virtual Nodes riak1.example.com riak2.example.com riak3.example.com riak4.example.com
21. Nodes & Virtual Nodes riak1.example.com riak2.example.com riak3.example.com riak4.example.com
22. Nodes & Virtual Nodes riak1.example.com 11 riak2.example.com 11 riak3.example.com 10 riak4.example.com 0
23. Nodes & Virtual Nodes riak1.example.com 8 riak2.example.com 8 riak3.example.com 8 riak4.example.com 8
24. Nodes & Virtual Nodes riak1.example.com riak2.example.com riak3.example.com riak4.example.com
25. Extras Not only a Key-Value store. We also have: • MapReduce (Erlang & JS) • Links & Link-walking • Secondary Indexes • Fulltext Search (& More Soon) • CRDTs
26. Extras Not only a Key-Value store. We also have: • MapReduce (Erlang & JS) • Links & Link-walking • Secondary Indexes • Fulltext Search (& More Soon) • CRDTs
27. MapReduce
28. Riak Search A hand-rolled Solr-like Search Engine, built into Riak
29. Riak Search - Mk II (Coming Soon) Yokozuna Integrate Solr itself with a Riak Cluster: use Riak for distribution and Solr for searching. https://github.com/basho/yokozuna
30. Concurrent Updates 1
31. Concurrent Updates 1 2
32. Concurrent Updates 1 2 3a
33. Concurrent Updates 1 2 3a 3b
34. Concurrent Updates 1 2 3a 3b 4
35. Concurrent Updates 1 2 3a 3b 4
36. Concurrent Updates 1 2 3a 3b 4
37. An Approach One Approach to Eventual Consistency: • Use allow_mult=true (not the default) • Teach your application how to resolve conflicts • Fetch all siblings • Replace siblings with merged value
38. Another Approach • Each “actor” maintains their own per-actor data • Upon fetch, merged into a single object In General: • Store the data by “actor” • Teach your application how to resolve that structure into an object
39. Another Approach • Each “actor” maintains their own per-actor data • Upon fetch, merged into a single object In General: • Store the data by “actor” • Teach your application how to resolve that structure into an object • CRDTs: Convergent Replicated Data Types
40. CRDTs Convergent Replicated Data Types Data Structures that Riak knows how to merge, preserving all operations. • Counters (PN-Counter, ...) • Sets (OR-Set, ...) • Graphs • TreeDoc (Collaborative Doc) • Maps (field name → CRDT)
41. Riak EE Enterprise Edition Riak with: • Multi Data Centre Replication • Better Statistics
42. Riak CS Riak CS An S3-compatible Multi-tenant Storage System
43. Conclusions • Distributed databases aren’t trivial • Riak doesn’t hide anything • CRDTs are where it’s at for EC
44. Any Questions? sam.elliott@basho.com