Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scaling Your Data Safely with Riak
Search
Alex Moore
January 10, 2014
Programming
3
140
Scaling Your Data Safely with Riak
CodeMash 2014 Presentation.
Alex Moore
January 10, 2014
Tweet
Share
Other Decks in Programming
See All in Programming
17年周年のWebアプリケーションにTanStack Queryを導入する / Implementing TanStack Query in a 17th Anniversary Web Application
saitolume
0
250
Security_for_introducing_eBPF
kentatada
0
110
range over funcの使い道と非同期N+1リゾルバーの夢 / about a range over func
mackee
0
110
Monixと常駐プログラムの勘どころ / Scalaわいわい勉強会 #4
stoneream
0
280
Webエンジニア主体のモバイルチームの 生産性を高く保つためにやったこと
igreenwood
0
340
生成AIでGitHubソースコード取得して仕様書を作成
shukob
0
480
MCP with Cloudflare Workers
yusukebe
2
220
Effective Signals in Angular 19+: Rules and Helpers
manfredsteyer
PRO
0
110
なまけものオバケたち -PHP 8.4 に入った新機能の紹介-
tanakahisateru
1
120
快速入門可觀測性
blueswen
0
380
「Chatwork」Android版アプリを 支える単体テストの現在
okuzawats
0
180
ある日突然あなたが管理しているサーバーにDDoSが来たらどうなるでしょう?知ってるようで何も知らなかったDDoS攻撃と対策 #phpcon.2024
akase244
1
150
Featured
See All Featured
KATA
mclloyd
29
14k
Producing Creativity
orderedlist
PRO
341
39k
Practical Orchestrator
shlominoach
186
10k
4 Signs Your Business is Dying
shpigford
181
21k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
GraphQLの誤解/rethinking-graphql
sonatard
67
10k
Bash Introduction
62gerente
608
210k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
2
290
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.3k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
The Language of Interfaces
destraynor
154
24k
Docker and Python
trallard
42
3.1k
Transcript
with Riak @alexmoore Scaling Data Safely
Internet 1 (2) (3)
Internet 503 Timeout 1 (2) (3)
Internet 1 (2) 3
Internet 1 (2) 3
Internet (2) 3 2 3 1
Internet 4 2 3 (2) 1 3 3
Internet CDN 2 6 7 2 4 3 1 3
3
Internet CDN 5 6 2 4 3 2 1 3
3 7 3
Internet CDN 6 7 2 3 2 1 3 3
Internet CDN P S S 3 6 7 2 5
3 2 1 4 4 3 3 7
Internet CDN 6 8 2 3 2 1 3 3
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic Who’s on first?
Internet CDN 6 8 2 3 2 1 3 3
3? 4? q User Session Widgets Lolcats Stuff 8? magic What the hell have you built? Who’s on first?
Enter Riak
Riak is a distributed, fault-tolerant, highly available database
None
Distributed
Distributed a cluster of nodes each node has an identical
role
Fault- tolerant
Fault- Tolerant symmetrical no Single Point of Failure
Highly Available
Highly Available replicas & redundancy
Scalable
Scalable add or remove nodes linear scalability
Scalable add or remove nodes linear scalability
Multiple Platforms
Multiple Platforms
Multiple Platforms
Predictable Performance
None
Internet CDN 5 6 2 3 2 1 3 3
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
1 2 3 4 5
6 7 8 9 10 11 12 Building A Cluster # All Nodes $ riak start ! # Nodes 2..n.each $ riak-‐admin join
[email protected]
! # Any Node $ riak-‐admin cluster plan $ riak-‐admin cluster commit !
None
None
Architecture
Riak is the ops-friendly database
Riak is distributed
Consistent Hashing hash(“bucket/key”)
hash ring
tokenize it
node 0 node 1 node 2
node 0 node 1 node 2 hash(key)
node 0 node 1 node 2 Replicas are stored to
the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2 hash(“companies/GE”) Replicas are stored
to the N - 1 contiguous partitions
node 0 node 1 node 2
node 0 node 1 node 2 node 3 + Scaling
out
Quorum requests N R W PR/PW DW
get(“users/clay-davis”)
get(“users/clay-davis”) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12
get(“users/clay-davis”) Get Handler (FSM) client Riak hash(“users/clay-davis”) == 10, 11,
12 Coordinating node Cluster 6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak get(“users/clay-davis”) Coordinating node Cluster
6 7 8 9 10 11 12 13 14 15 16 The Ring
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2
get(“users/clay-davis”) Get Handler (FSM) client Riak Coordinating node Cluster 6
7 8 9 10 11 12 13 14 15 16 The Ring R=2 obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) Get Handler (FSM) client Riak R=2 obj obj
get(“users/clay-davis”) obj
Read Repair (Anti-Entropy)
replica replica replica
replica replica replica X
replica replica replica replica replica replica
Active Anti-Entropy (self healing clusters)
real-time updates persistent non-blocking disk-based
merkle tree to track changes coordinated at the vnode level
runs as a background process exchange with neighbor vnodes for inconsistencies resolution semantics: trigger read-repair
= hashes marked “dirty”
None
None
None
None
= keys to read-repair
Using Riak
Riak is the ops-friendly database
Riak is friendly to Developers, too!
{“key”: “value”}
key value key value key value key value key value
key value key value key value key value Keys are namespaced into Buckets
Buckets aren’t “real”
✘ ✘ ✘ Buckets aren’t “real”
Metadata Bucket Properties
Aitch Tee Tee Pee GET /buckets/bucket/keys/key PUT /buckets/bucket/keys/key DEL /buckets/bucket/keys/key
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Starting $ gem install riak-‐client ! book = { isbn: '1111979723', title: 'Moby Dick', author: 'Herman Melville', body: 'Call me Ishmael. Some years ago...', copies_owned: 3 }
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Create # Starting Client client = Riak::Client.new(protocol: 'pbc', pb_port: 8087, host: '10.0.0.1') ! books_bucket = client.bucket('books') ! robj_moby = books_bucket.new(book[:isbn]) robj_moby.data = book robj_moby.store ! !
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Read fetched_book = books_bucket.get('1111979723') ! puts fetched_book.raw_data ! => {"isbn":"1111979723","title":"Moby Dick”, "author":"Herman Melville", "body":"Call me Ishmael. Some years ago...","copies_owned":3} ! fetched_book.data['isbn'] ! => "1111979723"
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - Delete fetched_book.delete ! # or ! books_bucket.delete('1111979723')
1 2 3 4 5
6 7 8 9 10 11 12 Working With Ruby - 2I ! fetched_book.indexes['author_bin'] = [] fetched_book.indexes['author_bin'] << fetched_book.data['author'] ! fetched_book.store ! books_bucket.get_index('author_bin','Herman Melville') ! => ["1111979723"]
etc...
JUST ONE MORE THING
Internet CDN 5 6 2 3 2 1 3 3
3 3 4 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4
in riak.conf:
search = off in riak.conf:
search = off search = on in riak.conf:
1 2 3 4 5
6 7 8 9 10 11 12 client.create_search_index(“tweets_idx”) ! bucket = Riak::Bucket.new(client, "tweets") bucket.props = {"search_index" => "tweets_idx"} ! # load some tweets into riak… ! results = client.search("tweets", "text:who") ! puts results[‘docs'].first
1 2 3 4 5
6 7 8 9 10 11 12 pp results[‘docs'].first ! {"score"=>"2.50804900000000019489e+00", "_yz_rb"=>"tweets", "_yz_rk"=>"421672897250226176", ...}
More resources: docs.basho.com “Taste of Riak”
Multi-Datacenter Replication (E.G. - Nuke proofing)
Full Sync
Real-Time
None
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
put(“users/jimmy”)
Internet CDN 5 6 2 3 2 1 3 3
3 4
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Internet CDN 5 6 2 3 2 1 3 3
3 4 ?
Q & A Alex Moore - Client Services Engineer, Basho
@alexmoore
[email protected]
http://alexmoore.io