Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Modeling for Scale with Riak Data Types (Q...

Data Modeling for Scale with Riak Data Types (QCon NY)

Eventually consistent databases like Riak give you horizontal scale, high availability and predictable latency, but can make data modeling more difficult. Riak 2.0 breaks from this pattern by introducing rich data types that are familiar to developers but handle eventual consistency for you.

In this talk, we'll briefly discuss the theory behind these new datatypes and then walk through some example applications that use them in popular languages.

Sean Cribbs

June 11, 2014
Tweet

More Decks by Sean Cribbs

Other Decks in Technology

Transcript

  1. Data Modeling for Scale with Riak Data Types Sean Cribbs

    @seancribbs #riak #datatypes QCon NYC 2014
  2. Consistency? 1 2 3 B A No clear winner! Throw

    one out? Keep both? Cassandra
  3. Consistency? 1 2 3 B A No clear winner! Throw

    one out? Keep both? Cassandra Riak
  4. Semantic Resolution • Your app knows the domain - use

    business rules to resolve • Amazon Dynamo’s shopping cart
  5. Semantic Resolution • Your app knows the domain - use

    business rules to resolve • Amazon Dynamo’s shopping cart “Ad hoc approaches have proven brittle and error-prone”
  6. How CRDTs Work • A partially-ordered set of values •

    A merge function • An identity value • In!ation operations
  7. How CRDTs Work • A partially-ordered set of values •

    A merge function • An identity value • In!ation operations What CRDTs Enable • Consistency without coordination • Fluent, rich interaction with data
  8. This research is supported in part by European FP7 project

    609 551 SyncFree http://syncfree.lip6.fr/ (2013--2016).
  9. Data Modeling for Riak • Identify needs for both read

    and write • Design around key as index • Denormalize relationships if possible • Weigh data size against coherence
  10. Riak Data Types increment decrement Counter :: int add* remove

    Set :: { bytes } remove update* Map :: bytes → DT
  11. Riak Data Types increment decrement Counter :: int add* remove

    Set :: { bytes } remove update* Map :: bytes → DT
  12. Riak Data Types increment decrement Counter :: int add* remove

    Set :: { bytes } assign Register :: bytes remove update* Map :: bytes → DT
  13. Riak Data Types increment decrement Counter :: int add* remove

    Set :: { bytes } enable* disable Flag :: boolean assign Register :: bytes remove update* Map :: bytes → DT
  14. Ad Network • Impressions - when someone sees an ad

    • Click-through - when someone clicks on an ad • Hourly rollups ad-metrics/<campaign>/<type>-<hour>
  15. $ riak-admin bucket-type create ad-metrics \ '{"props":{"datatype":"counter"}}' ad-metrics created $

    riak-admin bucket-type activate ad-metrics ad-metrics has been activated $ riak-admin bucket-type list ad-metrics (active) Ad Network
  16. from riak import RiakClient from rogersads import RIAK_CONFIG from time

    import strftime client = RiakClient(**RIAK_CONFIG) metrics = client.bucket_type('ad-metrics') def record_metric(campaign, metric_type): key = metric_type + strftime('-%Y%m%d-%H') counter = metrics.bucket(campaign).new(key) counter.increment() counter.store() Ad Network
  17. • RSVPs - guest lists • Connections - friends lists

    per-user • Likes - expressing interest PartyOn
  18. $ riak-admin bucket-type create partyon-sets \ '{"props":{"datatype":"set"}}' partyon-sets created $

    riak-admin bucket-type activate partyon-sets partyon-sets has been activated $ riak-admin bucket-type list partyon-sets (active) PartyOn
  19. from riak.datatypes import Set sets = client.bucket_type('partyon-sets') rsvps = sets.bucket('rsvps')

    friends = sets.bucket('friends') likes = sets.bucket('likes') PartyOn
  20. def rsvp_get(event): return rsvps.get(event) # Returns a Set def rsvp_add(event,

    user): guests = rsvps.new(event) guests.add(user) guests.store(return_body=True) return guests.context def rsvp_remove(event, user, context): guests = Set(rsvps, event, context=context) guests.remove(user) guests.store() PartyOn
  21. GameNet • User pro"les - demographic data users/profiles/<userid> • Achievements

    - trophies per game users/trophies/<userid> • Game state - progress and stats users/<gameid>/<userid>
  22. $ riak-admin bucket-type create users \ '{"props":{"datatype":"map"}}' users created $

    riak-admin bucket-type activate users users has been activated $ riak-admin bucket-type list users (active) GameNet
  23. users = client.bucket_type('users') def update_profile(user, fields): profile = users.bucket('profiles').get(user) for

    field in fields: if field in USER_FLAGS: if fields[field]: profile.flags[field].enable() else: profile.flags[field].disable() else: value = fields[field] profile.registers[field].assign(value) profile.store() GameNet
  24. def add_trophy(user, game, trophy): trophies = users.bucket('trophies').get(user) trophies.sets[game].add(trophy) trophies.store() def

    get_trophies(user, game): trophies = users.bucket('trophies').get(user) return trophies.sets[game].value GameNet
  25. def build_structure(user, game, structure, gold, wood, stone): gamestate = users.bucket(game).get(user)

    gamestate.sets['structures'].add(structure) gamestate.counters['gold'].decrement(gold) gamestate.counters['wood'].decrement(wood) gamestate.counters['stone'].decrement(stone) gamestate.store(return_body=True) return gamestate.value GameNet
  26. Bene"ts • Richer interactions, familiar types • Write mutations, not

    state • No merge function to write • Same reliability and predictability of vanilla Riak
  27. Caveats • Value size still matters • Updates not idempotent

    (yet) • Cross-key atomicity not possible (yet)
  28. Future • Riak 2.0 due out this summer - betas

    available now! • Richer querying, lighter storage requirements, more types