Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Introduction to Druid

Druid
November 11, 2013

An Introduction to Druid

The history and motivations behind why we built Druid and our plans for the future of the project.

Druid

November 11, 2013
Tweet

More Decks by Druid

Other Decks in Technology

Transcript

  1. 2013 ‣ Queries that were cached • fast ‣ Queries

    against aggregate tables • fast to acceptable ‣ Queries against base fact table • generally unacceptable I. RDBMS - THE RESULTS
  2. 2013 I. RDBMS - PERFORMANCE Naive benchmark scan rate ~5.5M

    rows / second / core 1 day of summarized aggregates 60M+ rows 1 query over 1 week, 16 cores ~5 seconds Page load with 20 queries over a week of data long time
  3. 2013 ‣ Pre-aggregate all dimensional combinations ‣ Store results in

    a NoSQL store II. NOSQL - THE SETUP ts gender age revenue 1 M 18 $0.15 1 F 25 $1.03 1 F 18 $0.01 Key Value 1 revenue=$1.19 1,M revenue=$0.15 1,F revenue=$1.04 1,18 revenue=$0.16 1,25 revenue=$1.03 1,M,18 revenue=$0.15 1,F,18 revenue=$0.01 1,F,25 revenue=$1.03
  4. 2013 ‣ Queries were fast • range scan on primary

    key ‣ Inflexible • not aggregated, not available ‣ Not continuously updated • aggregate first, then display ‣ Processing scales exponentially II. NOSQL - THE RESULTS
  5. 2013 ‣ Dimensional combinations => exponential increase ‣ Tried limiting

    dimensional depth • still expands exponentially ‣ Example: ~500k records • 11 dimensions, 5-deep • 4.5 hours on a 15-node Hadoop cluster • 14 dimensions, 5-deep • 9 hours on a 25-node Hadoop cluster II. NOSQL - PERFORMANCE
  6. 2013 ‣ Problem with RDBMS: scans are slow ‣ Problem

    with NoSQL: computationally intractable WHAT WE LEARNED
  7. 2013 ‣ Problem with RDBMS: scans are slow ‣ Problem

    with NoSQL: computationally intractable ! ! ! ! ! ‣ Tackling the RDBMS issue seems easier WHAT WE LEARNED
  8. “ ” “REAL-TIME” IS THE NEW “CLOUD” THE NEW ADJECTIVE

    THAT EVERYONE WANTS BUT NO ONE KNOWS WHAT IT MEANS
  9. 2013 WHAT IT MEANS TO US RATE OF QUERY RETURN

    QUERY TIME < 5S ! INGESTION LATENCY
 EVENT OCCURRENCE TO VISIBILITY < 10S
  10. 2013 ‣ Ingest Data streams ‣ Query data as soon

    as it is ingested ‣ Buffer data in memory ‣ Periodically “hand off” data REAL-TIME NODES
  11. 2013 ‣ Main workhorses of a Druid cluster ‣ Load

    historical data (most data in a cluster is historical data) ‣ Respond to queries HISTORICAL NODES
  12. 2013 ARCHITECTURE Query API Historical Nodes Realtime Nodes Query API

    Broker Nodes Query API Query Rewrite Scatter/Gather a Hand Off Data
  13. 2013 ‣ Knows which nodes hold what data ‣ Query

    scatter/gather (send requests to nodes and merge results) ‣ Caching BROKER NODES
  14. 2013 DATA! timestamp page language city country ... added deleted!

    2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
  15. 2013 COLUMN COMPRESSION · DICTIONARIES ‣ Create ids • Justin

    Bieber -> 0, Ke$ha -> 1 ‣ Store • page -> [0 0 0 1 1 1] • language -> [0 0 0 0 0 0] timestamp page language city country ... added deleted! 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
  16. 2013 BITMAP INDICES ‣ Justin Bieber -> [0, 1, 2]

    -> [111000] ‣ Ke$ha -> [3, 4, 5] -> [000111] timestamp page language city country ... added deleted! 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
  17. 2013 FAST AND FLEXIBLE QUERIES JUSTIN BIEBER [1, 1, 0,

    0] KE$HA [0, 0, 1, 1] JUSTIN BIEBER OR KE$HA [1, 1, 1, 1] row page 0 Justin(Bieber 1 Justin(Bieber 2 Ke$ha 3 Ke$ha
  18. 2013 BITMAP INDEX COMPRESSION ‣ CONCISE compression • Boolean operations

    directly on compressed indices • Less memory => faster scan rates ! ‣ More details • http://ricerca.mat.uniroma3.it/users/colanton/concise.html
  19. 2013 REPLICATION ROLLING DEPLOYMENTS + RESTARTS GROW = START PROCESSES

    SHRINK = KILL PROCESSES 2 YEARS · NO DOWNTIME FOR SOFTWARE UPDATE AVAILABILITY
  20. 2013 ASK ABOUT OUR PLUGINS ‣ Extensible architecture • Build

    your own modules and extend Druid • Add your own complex metrics (cardinality estimation, approximate histograms and quantiles, approximate top-K algorithms, etc.) • Add your own data ingestion modules (Hadoop, Storm, etc.) • Add your own proprietary modules ‣ Build your own node types from existing modules ‣ Available in Druid 0.6
  21. 2013 DRUID AS A PLATFORM Batch Ingestion (Hadoop) Streaming Ingestion

    (Storm) Druid Approximate Algorithms Machine Learning (?) Visualizations
  22. 2013 THE COMMUNITY ‣ Open sourced in Oct. 2012 ‣

    Growing Community • ~30 contributors (not all publicly listed) from many different companies • In production at multiple companies, we’re hoping for more! • Support through community forums and IRC • We love contributions!
  23. 2013 REALTIME INGESTION 10 – 100K RECORDS / SECOND /

    NODE ! REAL-WORLD CLUSTER
 150K EVENTS/S (7B EVENTS/DAY)
 500MB/S (2 TB/H) DRUID BENCHMARKS
  24. 2013 SCAN SPEED ~33M ROWS / SECOND / CORE INDEX

    COMPRESSION CONCISE SETS 70% SMALLER THAN BINARY ARRAYS “OR” OPERATION WITH 1M DIMENSION VALUES IN ~5S DRUID BENCHMARKS
  25. 2013 TAKE-AWAYS ‣ When Druid? • Interactive, fast exploration of

    large amounts of data • You need analytics (not key value store) • You want to do your analysis on data as it’s happening (realtime) • Your store needs to be always-on • You need extensibility and flexibility
  26. 2013 TAKE-AWAYS ‣ Not Druid? • The amount of data

    you have can easily be handled by MYSQL • You’re querying for individual entries or doing lookups (not analytics) • You need to constantly do updates and deletes • Batch is good enough • Downtime is no big deal