Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Real-time Analytics with Open Source Technologies

Druid
July 30, 2014

Real-time Analytics with Open Source Technologies

Druid

July 30, 2014
Tweet

More Decks by Druid

Other Decks in Technology

Transcript

  1. REAL-TIME ANALYTICS WITH OPEN SOURCE TECHNOLOGIES KAFKA · HADOOP ·

    STORM · DRUID FANGJIN YANG · GIAN MERLINO SOFTWARE ENGINEERS @ METAMARKETS
  2. PROBLEM DEALING WITH EVENT DATA MOTIVATION EVOLUTION OF A “REAL-TIME”

    STACK ARCHITECTURE THE “RAD”-STACK NEXT STEPS TRY IT OUT FOR YOURSELF OVERVIEW
  3. 2013 THE PROBLEM ‣ Arbitrary and interactive exploration ‣ Multi-tenancy:

    thousands of concurrent users ‣ Recency matters! Alert on major changes ‣ Scalability & Availability
  4. 2013 A SOLUTION ‣ Load all your data into Hadoop.

    Query it. Done! ‣ Good job guys, let’s go home
  5. 2013 PROBLEMS OF THE NAIVE SOLUTION ‣ MapReduce can handle

    almost every distributed computing problem ‣ MapReduce over your raw data is flexible but slow ‣ Hadoop is not optimized for query latency ‣ To optimize queries, we need a query layer
  6. 2013 MAKE QUERIES FASTER ‣ What types of queries to

    optimize for? • Revenue over time broken down by demographic • Top publishers by clicks over the last month • Number of unique visitors broken down by any dimension • Not dumping the entire dataset • Not examining individual events
  7. 2013 DRUID ‣ Druid project started in 2011 ‣ Open

    sourced in Oct. 2012 ‣ Growing Community • ~40 contributors from many different organizations ‣ Designed for low latency ingestion and aggregation • Optimized for the types of queries we were trying to make
  8. 2013 RAW DATA timestamp publisher advertiser gender country click price!

    2011-01-01T01:01:35Z bieberfever.com google.com Male USA 0 0.65! 2011-01-01T01:03:63Z bieberfever.com google.com Male USA 0 0.62! 2011-01-01T01:04:51Z bieberfever.com google.com Male USA 1 0.45! ...! 2011-01-01T01:00:00Z ultratrimfast.com google.com Female UK 0 0.87! 2011-01-01T02:00:00Z ultratrimfast.com google.com Female UK 0 0.99! 2011-01-01T02:00:00Z ultratrimfast.com google.com Female UK 1 1.53
  9. 2013 ROLLUP DATA timestamp publisher advertiser gender country impressions clicks

    revenue! 2011-01-01T01:00:00Z ultratrimfast.com google.com Male USA 1800 25 15.70! 2011-01-01T01:00:00Z bieberfever.com google.com Male USA 2912 42 29.18! 2011-01-01T02:00:00Z ultratrimfast.com google.com Male UK 1953 17 17.31! 2011-01-01T02:00:00Z bieberfever.com google.com Male UK 3194 170 34.01 ‣ Truncate timestamps ‣ GroupBy over string columns (dimensions) ‣ Aggregate numeric columns (metrics)
  10. 2013 PARTITION DATA timestamp publisher advertiser gender country impressions clicks

    revenue! ! ! ! 2011-01-01T01:00:00Z ultratrimfast.com google.com Male USA 1800 25 15.70! 2011-01-01T01:00:00Z bieberfever.com google.com Male USA 2912 42 29.18! ! ! ! ! 2011-01-01T02:00:00Z ultratrimfast.com google.com Male UK 1953 17 17.31! 2011-01-01T02:00:00Z bieberfever.com google.com Male UK 3194 170 34.01! ! ‣ Shard data by time ‣ Immutable chunks of data called “segments” ! ! Segment 2011-01-01T02/2011-01-01T03 ! ! Segment 2011-01-01T01/2011-01-01T02
  11. 2013 IMMUTABLE SEGMENTS ‣ Fundamental storage unit in Druid ‣

    Read consistency ‣ One thread scans one segment ‣ Multiple threads can access same underlying data ‣ Segment sizes -> computation completes in ms ‣ Simplifies distribution & replication
  12. 2013 COLUMN ORIENTATION timestamp publisher advertiser gender country impressions clicks

    revenue! ! ! ! 2011-01-01T01:00:00Z ultratrimfast.com google.com Male USA 1800 25 15.70! 2011-01-01T01:00:00Z bieberfever.com google.com Male USA 2912 42 29.18! ! ‣ Scan/load only what you need ‣ Compression! ‣ Indexes!
  13. 2013 ARCHITECTURE (EARLY DAYS) Broker Node Historical Node Historical Node

    Historical Node Broker Node Queries Hadoop Data
  14. 2013 MORE PROBLEMS ‣ We’ve solved the query problem •

    Druid gave us arbitrary data exploration & fast queries ‣ What about data freshness? • Batch loading is slow! • We need “real-time” • Alerts, operational monitoring, etc.
  15. 2013 THE STORY SO FAR ‣ Clients uploaded data to

    S3 ‣ We used Hadoop + Pig to clean it up, transform, and join it ‣ We loaded the result into Druid ‣ Typical turnaround time: 2–8 hours
  16. 2013 INGESTION DELAYS ‣ Let’s build a streaming version of

    this data pipeline ‣ Three obstacles ‣ Acquiring raw data ‣ Processing the data ‣ Loading processed data into Druid
  17. 2013 FAST DELIVERY WITH KAFKA ‣ High throughput event delivery

    service ‣ Straightforward, reliable design ‣ Buffers incoming data to give consumers time to process it ‣ We can place an HTTP API in front of this
  18. 2013 FAST PROCESSING WITH STORM ‣ Storm is a stream

    processor— one event at a time ‣ We can already process our data using Hadoop MapReduce ‣ Let’s translate that to streams ‣ “Load” operations stream data from Kafka ‣ “Map” operations are already stream-friendly ‣ “Reduce” operations can be windowed with partitioned state
  19. 2013 FAST LOADING WITH DRUID ‣ We have an indexing

    system ‣ We have a serving system that runs queries on data ‣ We can serve queries while building indexes! ‣ Real-time indexing workers do this
  20. 2013 FAST LOADING WITH DRUID Druid Realtime Workers Immediate Druid

    Historical Cluster Periodic Druid Query Broker Kafka Brokers Storm Workers
  21. 2013 WHAT WE GAINED ‣ Druid queries reflect new events

    within seconds ‣ Systems are fully decoupled ‣ Brief processing delays during maintenance ‣ Because we need to restart Storm topologies ‣ But query performance and availability are not affected
  22. 2013 WHAT WE GAVE UP ‣ Stream processing isn’t perfect

    ‣ Difficult to handle corrections of existing data ‣ Windows may be too small for fully accurate operations ‣ Hadoop was actually good at these things
  23. 2013 THE RETURN OF HADOOP ‣ An open-source “lambda architecture”

    ‣ Batch re-processing runs for all data older than a few hours ‣ Batch segments replace real-time segments in Druid ‣ Query broker merges results from both systems “Fixed up,” immutable, historical data –by Hadoop Realtime data –by Storm & Realtime Druid
  24. 2013 THE STACK Event Streams Insight Kafka Hadoop Druid Storm

    ‣ Real-time ‣ Only on-time data ‣ Some hours later ‣ All data
  25. 2013 CORNERSTONES ‣ Druid - druid.io - @druidio ‣ Storm

    - storm.incubator.apache.org - @stormprocessor ‣ Hadoop - hadoop.apache.org ‣ Kafka - kafka.apache.org - @apachekafka
  26. 2013 GET RADICAL ‣ Queries answered quickly, on fresh data

    ‣ Kafka provides fast, reliable event transport ‣ Storm and Hadoop clean and prepare data for Druid ‣ Druid handles queries and manages the serving layer ‣ “Real-time Analytics Data Stack” ‣ …a.k.a. RAD Stack ‣ https://metamarkets.com/2014/building-a-data-pipeline/