Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Your Data Safely with Riak
Search
Alex Moore
January 10, 2014
Programming
3
140
Scaling Your Data Safely with Riak
CodeMash 2014 Presentation.
Alex Moore
January 10, 2014
Tweet
Share
Other Decks in Programming
See All in Programming
NSOutlineView何もわからん:( 前編 / I Don't Understand About NSOutlineView :( Pt. 1
usagimaru
0
340
ECS Service Connectのこれまでのアップデートと今後のRoadmapを見てみる
tkikuc
2
250
Jakarta EE meets AI
ivargrimstad
0
250
3 Effective Rules for Using Signals in Angular
manfredsteyer
PRO
0
120
EMになってからチームの成果を最大化するために取り組んだこと/ Maximize team performance as EM
nashiusagi
0
100
Remix on Hono on Cloudflare Workers
yusukebe
1
300
Contemporary Test Cases
maaretp
0
140
3 Effective Rules for Using Signals in Angular
manfredsteyer
PRO
1
100
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
9
3.3k
TypeScript Graph でコードレビューの心理的障壁を乗り越える
ysk8hori
3
1.2k
レガシーシステムにどう立ち向かうか 複雑さと理想と現実/vs-legacy
suzukihoge
14
2.3k
CSC509 Lecture 11
javiergs
PRO
0
180
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
32
1.5k
Producing Creativity
orderedlist
PRO
341
39k
Done Done
chrislema
181
16k
4 Signs Your Business is Dying
shpigford
180
21k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
28
2k
What's in a price? How to price your products and services
michaelherold
243
12k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Intergalactic Javascript Robots from Outer Space
tanoku
269
27k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
KATA
mclloyd
29
14k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Transcript
with Riak @alexmoore Scaling Data Safely
Internet 1 (2) (3)
Internet 503 Timeout 1 (2) (3)
Internet 1 (2) 3
Internet 1 (2) 3
Internet (2) 3 2 3 1
Internet 4 2 3 (2) 1 3 3
Internet CDN 2 6 7 2 4 3 1 3
3
Internet CDN 5 6 2 4 3 2 1 3
3 7 3
Internet CDN 6 7 2 3 2 1 3 3
Internet CDN P S S 3 6 7 2 5
3 2 1 4 4 3 3 7
Internet CDN 6 8 2 3 2 1 3 3
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic Who’s on first?
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic What the hell have you built? Who’s on first?
Enter Riak
Riak is a distributed, fault-tolerant, highly available database
None
Distributed
Distributed a cluster of nodes each node has an identical
role
Fault- tolerant
Fault- Tolerant symmetrical no Single Point of Failure
Highly Available
Highly Available replicas & redundancy
Scalable
Scalable add or remove nodes linear scalability
Scalable add or remove nodes linear scalability
Multiple Platforms
Multiple Platforms
Multiple Platforms
Predictable Performance
None
Internet CDN 5 6 2 3 2 1 3 3
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
1 2 3 4 5
6 7 8 9 10 11 12 Building A Cluster # All Nodes $ riak start ! # Nodes 2..n.each $ riak-‐admin join
[email protected]
! # Any Node $ riak-‐admin cluster plan $ riak-‐admin cluster commit !
None
None
Architecture
Riak is the ops-friendly database
Riak is distributed
Consistent Hashing hash(“bucket/key”)
hash ring
tokenize it
node 0 node 1 node 2
node 0 node 1 node 2 hash(key)
node 0 node 1 node 2 Replicas are stored to
the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2
node 0 node 1 node 2 node 3 + Scaling
out
Quorum requests N R W PR/PW DW
get(“users/clay-davis”)
get(“users/clay-davis”) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak get(“users/clay-davis”) Coordinating node Cluster
6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) obj
Read Repair (Anti-Entropy)
replica replica replica
replica replica replica X
replica replica replica replica replica replica
Active Anti-Entropy (self healing clusters)
real-time updates persistent non-blocking disk-based
merkle tree to track changes coordinated at the vnode level
runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
= hashes marked “dirty”
None
None
None
None
= keys to read-repair
Using Riak
Riak is the ops-friendly database
Riak is friendly to Developers, too!
{“key”: “value”}
key value key value key value key value key value
key value key value key value key value Keys are namespaced into Buckets
Buckets aren’t “real”
✘ ✘ ✘ Buckets aren’t “real”
Metadata Bucket Properties
Aitch Tee Tee Pee GET /buckets/bucket/keys/key PUT /buckets/bucket/keys/key DEL /buckets/bucket/keys/key
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Starting $ gem install riak-‐client ! book = { isbn: '1111979723', title: 'Moby Dick', author: 'Herman Melville', body: 'Call me Ishmael. Some years ago...', copies_owned: 3 }
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Create # Starting Client client = Riak::Client.new(protocol: 'pbc', pb_port: 8087, host: '10.0.0.1') ! books_bucket = client.bucket('books') ! robj_moby = books_bucket.new(book[:isbn]) robj_moby.data = book robj_moby.store ! !
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Read fetched_book = books_bucket.get('1111979723') ! puts fetched_book.raw_data ! => {"isbn":"1111979723","title":"Moby Dick”, "author":"Herman Melville", "body":"Call me Ishmael. Some years ago...","copies_owned":3} ! fetched_book.data['isbn'] ! => "1111979723"
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Delete fetched_book.delete ! # or ! books_bucket.delete('1111979723')
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - 2I ! fetched_book.indexes['author_bin'] = [] fetched_book.indexes['author_bin'] << fetched_book.data['author'] ! fetched_book.store ! books_bucket.get_index('author_bin','Herman Melville') ! => ["1111979723"]
etc...
JUST ONE MORE THING
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
in riak.conf:
search = off in riak.conf:
search = off search = on in riak.conf:
1 2 3 4 5
6 7 8 9 10 11 12 client.create_search_index(“tweets_idx”) ! bucket = Riak::Bucket.new(client, "tweets") bucket.props = {"search_index" => "tweets_idx"} ! # load some tweets into riak… ! results = client.search("tweets", "text:who") ! puts results[‘docs'].first
1 2 3 4 5
6 7 8 9 10 11 12 pp results[‘docs'].first ! {"score"=>"2.50804900000000019489e+00", "_yz_rb"=>"tweets", "_yz_rk"=>"421672897250226176", ...}
More resources: docs.basho.com “Taste of Riak”
Multi-Datacenter Replication (E.G. - Nuke proofing)
Full Sync
Real-Time
None
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Q & A Alex Moore - Client Services Engineer, Basho
@alexmoore
[email protected]
http://alexmoore.io