Apache Druid 2030 (Darin Briskman, Imply) | RTA Summit 2023

First created in 2011, Apache® Druid is now in its second decade of empowering real-time analytics. How does Druid provide subsecond queries at Petabyte scale, high concurrency, and combined stream and batch analytics? What have we learned from the first decade of Druid, and what is changing? What will Druid look like in 2030?

Transcript

1. Druid 2030 The Origin and Evolution of Real-Time Analytics Applications Darin Briskman darin.briskman@imply.io
2. ©2023, Imply A few things before we start 2 I’m a member of the Druid Community, giving my own opinions and thoughts. I do not speak for the Community, nor for Imply Data. Thanks to the RTA organizers for inviting me and Imply to participate in today’s events. Pinot and Druid are open source cousins, and we’re all working together to advance real-time analytics for everyone.
3. ©2023, Imply About me and Imply 3 Imply was founded by the creators of Apache Druid® to help developers get to successful projects faster
4. ©2023, Imply Druid today is very different from how it began 4
5. ©2023, Imply In the Beginning • Needed fast processing and rollups for AdTech • Requirement: ingest a billion rows in under 1 minute & query them in under 1 second • Tried and failed with RDBMS: Greenplum, Postgres, MySQL, and InfoBright • Tried and failed with NoSQL • Decided to create a new database. How hard could it be? • Announced Druid on 30 Apr 2011
6. ©2023, Imply Evolution and Open Source Rapid iteration as Druid drove the AdTech Business After getting interest from Netflix and others, released Druid as Open Source in 2012. Moved to Apache license in 2015. Promoted to Apache Software Foundation top-level in 2017. Used by over 1900 organizations.
7. ©2023, Imply Global and vibrant Community Companies using Druid Active Contributors YoY Increase in Community Activity Community Members 1,900+ 150% 14,000+ 500+
8. ©2023, Imply Druid Today: a Real-Time Analytics Database Sub-second queries at any scale Interactive analytics on TB-PBs of data High concurrency at the lowest cost 100s to 1000s QPS via a highly efficient engine Real-time and historical insights True stream ingestion for Kafka and Kinesis Plus, non-stop reliability with automated fault tolerance and continuous backup 1 2 3 For analytics applications that require:
9. ©2023, Imply The right database for analytics apps makes a difference OR Without Druid With Druid
10. ©2023, Imply 10 Applications Analytics Applications Druid is built for the intersection of analytics and applications. Apache Druid
11. ©2023, Imply Real-Time Analytics Applications Real-time Analytics Database Real-Time Analytics Requires a Real-Time Database 11 Analytics Data Warehouses Applications Transactional Databases Read-optimized TB-PBs of Data High Cardinality Sub-Sec Response High Concurrency Real-time Data BI Reporting Monthly Reporting Static Dashboards ACID Compliance Small Data Write-optimized BI Reporting Monthly Reporting Static Dashboards ACID Compliance Small Data Write-optimized ✓ ✓ ✓ ✓ ✓ ✓
12. ©2023, Imply Examples of real-time analytics applications 12 Operational Visibility at Scale Rapid Data Exploration Customer-facing Analytics Real-time Decisioning ICE Security Ops Platform Citrix Analytics Service Salesforce Edge Intelligence Reddit Real-Time Ads Powered by
13. ©2023, Imply What makes Druid different 13 Storage Segmentation Maturity Scale Community Success
14. ©2023, Imply Focus on maturity 14 Data Warehousing
15. ©2023, Imply And many more! Focus on community success Retail Financial Gaming Networking/Energy Technology Security Ad Tech Media
16. ©2023, Imply Coming in May: Druid 26.0 16 Schema auto-discovery Get both high performance & flexibility More ANSI SQL Compatibility Unnest & Arrays Shuffle Joins Simpler Data Ingestion Plus additional features including Sessionization, Interpolation, and Advanced Dictionary Compression
17. ©2023, Imply With Druid you now get both high performance & flexibility 17 Strong Data Types High Performance Data Type Discovery Ease of Use like schemaless databases N ew
18. ©2023, Imply 18 Voice Assistant Sends data about each request plus periodic status updates IoT Use Case Streaming Pipeline Analytics Database Data Analyst
19. ©2023, Imply 19 Voice Assistant Temperature & Humidity sensors enabled IoT Use Case Temp & Humidity Data Streaming Pipeline Analytics Database Data Analyst Adding Humidity & Temperature Data No Broken Schemas New columns for temperature & Humidity are automatically discovered and added with the right data type to the Druid table Auto-discover and add New Data Field & Type
20. ©2023, Imply Joins Prepare Incoming Data for Fast Analytics in Druid 20 Partition 1 Partition 2 Partition n Partition 1 Partition 2 Partition n Table 1 Table 2 Table 3
21. ©2023, Imply Pre-joining Data was Necessary to Bring Datasets Together 21 Fact Table A Fact Table A Ingest Store Query Third Party Tools Real-time Analytics
22. ©2023, Imply Ingestion Becomes Simpler, Easier, and Less Expensive 22 Fact Table A Fact Table A Ingest Store Query Druid Can Now Join Datasets at Ingestion
23. ©2023, Imply 23 Unnest “I want to UNNEST this repeated record into its own little temporary table.” Array “I wish I could do a SQL join without getting duplicate rows back.” Extend Standard SQL Features Keeping up with Evolving ANSI SQL Standards
24. ©2023, Imply 24 What is the average basket size with and without groceries? And how is it trending over the last 18 months? Billions transactions/month Standard Query Take too long + Isn’t necessary
25. ©2023, Imply 25 Statistics teaches us, when there is too much data, sample Sample GROUP BY Query a subset of data Druid ensures it is statistically valid Billions transactions/month What is the average basket size with and without groceries? And how is it trending over the last 18 months?
26. ©2023, Imply 26 What was the impact on sales from an approaching hurricane? Automatically figure out time boundaries of a data set Find day parts & start/end of impactful events Time weighted averages and interpolation
27. ©2023, Imply Advanced String Dictionary Compression 27 100 TBs 70 TBs Saving up to 30% on string storage w/ ZERO impact on performance Efficiency of querying numbers while retaining all the flexibility of the human language
28. ©2023, Imply Coming Later this Year: Async Queries + Cold Storage 28
29. ©2023, Imply Streams Everywhere 29
30. ©2023, Imply More Streaming = More Real-Time Data 30
31. ©2023, Imply Next Generation Infrastructure 31 2023-24 Memory Pooling Effective 50%+ reduction in costs of database infrastructure 2025 - 26 Data Processing Accelerators Co-processors that improve database performance per CPU by 3x - 20x
32. ©2023, Imply Cloud Maturity Today: ~45% of global IT infrastructure on the Cloud End of 2027: ~85% of global IT infrastructure on the Cloud Streaming Data and Real-Time Analytics Applications are the default on the Cloud Managed services become even more dominant as a delivery model 32
33. ©2023, Imply RTA and Machine Learning AI/ML becomes just another piece of the analytic toolset, like regressions and sketches. By 2025, all data-center class CPUs include packaged GPUs for machine learning inference acceleration. Active GANs optimize segmentation and query optimization. LLMs automate data gathering and (ironically) data quality 33
34. ©2023, Imply Real-Time Analytics Applications in 2030 34 EB scale and beyond Still subsecond response Ubiquitous streaming Most queries still SQL
35. ©2023, Imply Real-Time Analytics Applications in 2030 Open Source wins Druid is one of the projects that lead Real-Time Analytics databases (no single winner) Streaming and Analytics delivered as managed services Mix of edge and centralized computing 35 + ???
36. ©2023, Imply 36 Questions? … maybe Answers
37. ©2023, Imply 37 Darin Briskman Director of Technology darin.briskman@imply.io Join the Druid Community! https://druid.apache.org Druid Architecture and Concepts https://imply.io/druid-architecture-concepts/ Building Real-Time Analytics Applications https://bit.ly/40AIlB6 Try Polaris, the Druid DBaaS https://imply.io/polaris