Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
A Riak Query Tale
Search
Mathias Meyer
February 01, 2012
Programming
5
1k
A Riak Query Tale
An introduction to the abundance of ways you can get data out of Riak.
Mathias Meyer
February 01, 2012
Tweet
Share
More Decks by Mathias Meyer
See All by Mathias Meyer
Building and Scaling an Distributed and Inclusive Team
roidrage
0
1.2k
cooking infrastructure with chef
roidrage
4
220
The Message Queue is Dead, Long Live the Message Queue
roidrage
4
680
riak-js
roidrage
1
270
designing for concurrency with riak
roidrage
11
1.8k
metrics, monitoring, logging
roidrage
82
14k
design for cloud - jax 2012
roidrage
2
290
Don't Use NoSQL
roidrage
10
1k
Designing Applications for Amazon Web Services (GOTO Aarhus)
roidrage
6
350
Other Decks in Programming
See All in Programming
GitHub Copilot Workspace で我々のアプリ開発がどう変わるのか?
shuyakinjo
0
720
So you want to write a User-Defined Function for Flink? @ Current 2024 - Austin, TX
hpgrahsl
0
110
Shinjuku.rb#95:心の技術書紹介
free_world21
1
120
上手に付き合うコンポーネントテスト
quramy
3
1.1k
Crafting Cross-Platform Adventures: Building a Game Engine with Kotlin Multiplatform
dwursteisen
0
330
API Platform for Laravel
dunglas
1
1.1k
サーバーレスで負荷試験!Step Functions + Lambdaを使ったk6の分散実行
shuntakahashi
6
1.6k
#TROCCOUG dbt Cloud+TROCCO
troccoug
0
140
Google Sign-inの移行から始めるCredential Manager活用
clockvoid
0
560
Flutterアプリを生成AIで生成する勘所
rizumita
0
210
Kotlin 2.0が与えるAndroid開発の進化
masayukisuda
2
770
How to Break into Reading Open Source
kaspth
2
230
Featured
See All Featured
How GitHub (no longer) Works
holman
310
140k
Being A Developer After 40
akosma
84
590k
We Have a Design System, Now What?
morganepeng
48
7.1k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
663
120k
Principles of Awesome APIs and How to Build Them.
keavy
125
17k
A Modern Web Designer's Workflow
chriscoyier
692
190k
Making the Leap to Tech Lead
cromwellryan
129
8.8k
What's in a price? How to price your products and services
michaelherold
242
11k
Rails Girls Zürich Keynote
gr2m
93
13k
Designing Experiences People Love
moore
138
23k
Robots, Beer and Maslow
schacon
PRO
157
8.2k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.6k
Transcript
A Riak Query Tale Mathias Meyer, @roidrage NoSQL Cologne
http://riakhandbook.com
Riak Distributed Database Fault-Tolerant Content-Agnostic Scalable on Demand
Querying Data
Key-Value $ curl localhost:8098/riak/users/roidrage
Links $ curl -‐v localhost:8098/riak/users/roidrage < HTTP/1.1 200 OK <
Link: </riak/users/klimpong>; riaktag="friend"
Links $ curl .../riak/users/roidrage/users,friend,_/
Listing Keys $ curl .../riak/users?keys=true
Don’t do that!
Streaming Keys $ curl .../riak/users?keys=stream
Avoid that!
Loads all the keys.
MapReduce
MapReduce Transform (Map) Aggregate (Reduce)
Warning: JavaScript
MapReduce riak.add("users"). map("Riak.mapValues").
run()
MapReduce var nameLength = function(value) { var doc
= Riak.mapValues(value)[0]; return [doc.length]; }
MapReduce riak.add("users"). map(nameLength).
run()
MapReduce riak.add("users"). map(nameLength).
reduce("Riak.reduceSum"). run()
MapReduce var average = function(values) { var avg
= values.reduce(function(n, sum) { return sum += n; }, 0); return [(avg / values.length)]; }
MapReduce riak.add("users"). map(nameLength).
reduce(average). run()
MapReduce riak.add("users"). map(nameLength).
reduce(average). run() Uh-Oh!
MapReduce riak.add(["users", "roidrage"]). map(nameLength).
reduce(average). run() Better!
JavaScript M/R Breaks with Millions of Objects Uses External Libraries
Serializes Data for JavaScript
Warning: Erlang
MapReduce riak.add('tweets'). map({language: 'erlang',
module: 'riak_kv_mapreduce', function: 'map_object_value'}).run()
MapReduce $ riak attach > {ok, C} = riak:local_client().
MapReduce C:mapred([{<<"users">>, <<"roidrage">>}], [{map, {modfun, riak_kv_mapreduce, map_object_value}, none, false}, {reduce,
{modfun, riak_kv_mapreduce, reduce_count_inputs}, none, true}]).
MapReduce ExtractFirstName1 = fun(RObject, _, _) -‐>
Value = riak_object:get_value(RObject), [FirstName, _] = re:split(Value, " "), [FirstName] end.
MapReduce C:mapred([{<<"users">>, <<"roidrage">>}],
[{map, {qfun, ExtractFirstName}, none, true}]).
Erlang M/R Much more efficient than JavaScript No serialization No
ad-hoc functions through HTTP
Key-Filters Reduce MapReduce input Based on key matches
Key-Filters riak.add({bucket: 'users', key_filters: [["matches", "^roid"]]})
Key-Filters riak.add({bucket: 'users', key_filters: [["to_upper"],
["matches", "^ROID"]]})
Key-Filters riak.add({bucket: 'users', key_filters: [["to_upper"],
["to_lower"], ["matches", "^roid"]]})
Key-Filters riak.add({bucket: 'users', key_filters: [["to_upper"],
["ends_with", "RAGE"]]})
Key-Filters riak.add({bucket: 'users', key_filters:
[["and", [["string_to_int"], ["less_than", 10]], [["string_to_int"], ["greater_than", 5]]]]})
Don't use key filters.
Riak 2i Sorted Secondary Indexes Simple Reverse Lookups Maintained Manually
Requires LevelDB
Riak 2i curl -‐X PUT .../riak/users/roidrage -‐d @-‐ \
-‐H "Content-‐Type: text/plain" \ -‐H "X-‐Riak-‐Index-‐firstname_bin: mathias" \ -‐H "X-‐Riak-‐Index-‐lastname_bin: meyer"
Riak 2i X-‐Riak-‐Index-‐firstname_bin: Mathias X-‐Riak-‐Index-‐lastname_bin: Meyer
Riak 2i X-‐Riak-‐Index-‐firstname_bin: Mathias X-‐Riak-‐Index-‐lastname_bin: Meyer X-‐Riak-‐Index-‐age_int: 34
Riak 2i X-‐Riak-‐Index-‐firstname_bin: Mathias X-‐Riak-‐Index-‐lastname_bin: Meyer X-‐Riak-‐Index-‐age_int: 34 X-‐Riak-‐Index-‐topics_bin: nosql,cloud,operations
Riak 2i # Match $ curl .../buckets/users/index/firstname_bin/Mathias
Riak 2i # Range $ curl .../buckets/users/index/firstname_bin/Mathias/Till
Riak 2i # Key $ curl .../buckets/users/index/$key/roidrage
Ordered Keys! (sort of)
MapReduce riak.add({bucket: 'users',
index: 'lastname_bin', key: 'mathias'}). map('Riak.mapValuesJson').run()
Riak 2i No Multi-Index Queries Requires Extra Work in the
App Returns only keys Document-partitioned
Riak Search Full-Text Search Solr-ish Interface Integrates with Riak
Riak Search curl -‐X PUT localhost:8098/riak/users -‐d @-‐ \
-‐H "Content-‐Type: application/json" {"props":{"precommit": [{"mod":"riak_search_kv_hook","fun":"precommit"} ]}}
Indexing Riak Objects curl -‐X PUT .../riak/users/roidrage \
-‐d "Mathias Meyer" -‐H "Content-‐Type: text/plain"
Solr-ish Interface curl .../solr/users/select?q=value:Mathias
Riak Search value:Mathias OR value:Till value:Mathias AND value:Meyer value:Mat* value:[Mathias
TO Till]
MapReduce riak.addSearch("users", "value:Mathias"). map("Riak.mapValues").run()
Riak Search Full text search of structured data Term-partitioned Efficient
for one term queries Multiple Interfaces No Anti-Entropy
When?
Key Listings Never! Almost
MapReduce Analytical Queries Fixed Dataset
Key Filters Never!
Riak 2i Simple Lookups and Range Queries Unbounded Queries Full
Fault-Tolerance
Riak Search Larger documents Full indexing Flexible queries Low frequency
terms
Questions?