Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Scalable Event Service with Cassandr...

Building a Scalable Event Service with Cassandra: Design to Code

Cassandra Summit Europe 2014

Tareq Abedrabbo

December 04, 2014
Tweet

More Decks by Tareq Abedrabbo

Other Decks in Technology

Transcript

  1. Building a Scalable Event Service with Cassandra Design to Code

    David Borsos & Tareq Abedrabbo Cassandra Summit Europe 2014
  2. This talk is about… ‣ What we built ‣ Why

    we built it ‣ How we built it
  3. ✓ Capture millions of platform and business events ✓ Trigger

    downstream processes asynchronously ✓ Customise standard processes in a non-intrusive way ✓ Provide a system-wide transaction log ✓ Analytics ✓ Auditing ✓ System testing
  4. However… - Ambiguous requirements - New paradigm and emerging architecture

    - We need to look at the problem as a whole - We need to avoid building useless features - We need to avoid accumulating technical debt
  5. • A simple event is an opaque value, typically a

    time series item • meter reading • A structured event can have an arbitrarily complex structure that can evolve over time • user registration event
  6. Evolution of Event • Payload and meta-data as simple collections

    of key/value • The type is persisted with each event • to make events readable • to avoid managing schemas
  7. An event store should be - Simple request/response paradigm with

    clear guarantees - Accessible, ideally even from legacy services - Ability to query for events
  8. • Store an event • POST /api/events/ • Read an

    event • GET /api/events/{eventId}
  9. Anatomy of an Event { "type" : "SOME.EVENT.TYPE", "source" :

    "some-component:instance", "metadata" : { "anyMetaKey" : "someMetaValue", "tags" : [ "tag1", "tag2" ] }, "payload" : { "anyKey1" : "someValue", "anyKey2" : 3 } }
  10. The Event Table Key Event id1 timestamp type payload …

    123 X <<blob>> … id2 timestamp type payload … 456 Y <<blob>> …
  11. • Query events • GET /api/events?{parameters} • {queryString} can consist

    of the following fields: • start, end, startOffset, limit, tag, type, order
  12. Querying One denormalised table for each query Query Key id1

    ts11 id1 ts12 id2 ts21 id2 ts22 p1 b1 v11 p2 b2 v12 v21 p2 b2 v22
  13. • Cons: • Denormalise for each query, again and again

    • Higher disk usage • Disk space is cheap, but not free • Write latency is affected • Time-bucketed indexes can create hot spots (hot shards)
  14. • Same service contract • Indices are updated synchronously or

    asynchronously • Basic client guarantee: if a POST is successful the event has been persisted “sufficiently” • Events can be published to a message broker
  15. • Pro • Decoupling • client are unaware of the

    implementation details • Intuitive ReSTful interface • Disk consumption is more reasonable • Easily extensible • Pub/sub
  16. • Cons • Not primarily optimised for latency • Still

    sufficiently performant for our use-cases • More complex service code • Needs to execute multiple CQL queries in sequence • Cluster hotspots can still occur, in theory
  17. Indices • Ascending and descending time buckets for each query

    type • Index value references an event stored in the main table by its id
  18. Indices events_by_type_asc ( tbucket text, type text, eventid timeuuid, primary

    key ((type, tbucket), eventid)) with clustering order by (eventid asc); events_by_type_desc ( tbucket text, type text, eventid timeuuid, primary key ((type, tbucket), eventid)) with clustering order by (eventid desc); Ascending and descending time buckets for each query type
  19. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 type1 bucket2

    id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  20. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ type1 bucket1 ▲ query

    range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  21. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  22. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  23. Pagination GET /api/events?start=141..&type=X&limit=5 type time Ὂ ◀ query range ▶︎

    type1 bucket1 ▲ query range ▼ type1 bucket2 id1 id2 type1 bucket3 id3 id4 type1 bucket4 id5 id6 id7 id8 type1 bucket5
  24. GET /api/events?start=141..&type=X&limit=5 { “events” : [ { “id” : “uuid1”,

    “type” : “X”, “metadata” : { … }, “payload” : { … } }, { … } ], “continuation” : “/api/events?continueFrom=uuid7&type=X&limit=5” }
  25. Performance Characteristics • 70 to 85 million events per day

    • Client latency increases moderately with increased parallel load (40ms to 60ms, +10ms on the client) • Current behaviour exceeds by far current target volumes
  26. Lessons learnt • Scalability is not only about raw performance

    and latency • Experiment • Simplify • Understand Thrift, use CQL
  27. • Data model improvements: User Defined Types • DateTiered compaction

    • Analytics with Spark • Add other data views