Introduction To Apache Drill - Big Data Bellevue 20131023

Transcript

1. Introduction to Apache Drill Big Data Bellevue Meetup @tnachen Timothy Chen
2. Motivation Key Facts Architecture Overview
3. About me
9. I Open Source
22. Use Case: Marketing Campaign Jane, a marketing analyst Determine target segments Data from different sources
23. Use Case: Crime Detection • Online purchases • Fraud, billing, etc. • Batch-generated overview • Modes – Explorative – Alerts
24. Requirements • Support for different data sources • Support for different query interfaces • Low-latency/real-time • Ad-hoc queries • Scalable, reliable
26. Google’s Dremel http://research.google.com/pubs/pub36632.html Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis, Proc. of the 36th Int'l Conf on Very Large Data Bases (2010), pp. 330-339 Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. … “ “ Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. …
27. Google’s Dremel multi-level execution trees columnar data layout
28. Google’s Dremel nested data + schema column-striped representation map nested data to tables
29. Google’s Dremel experiments: datasets & query performance
30. Apache Drill–key facts Inspired by Google’s Dremel Standard SQL 2003 support Plug-able data sources Nested data is a first-class citizen Schema is optional Community driven, open, 100’s involved
33. High-level Architecture
34. Principled Query Execution Source query—what we want to do (analyst friendly) Logical Plan— what we want to do (language agnostic, computer friendly) Physical Plan—how we want to do it (the best way we can tell) Execution Plan—where we want to do it
35. Principled Query Execution Sourc e Query Parser Logica l Plan Optimizer Physical Plan Execution SQL 2003 DrQL MongoQL DSL scanner API Topology CF etc. query: [ { @id: "log", op: "sequence", do: [ { op: "scan", source: “logs” }, { op: "filter", condition: "x > 3” }, parser API
36. Wire-level Architecture Each node: Drillbit - maximize data locality Co-ordination, query planning, execution, etc, are distributed Any node can act as endpoint for a query—foreman Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node
37. Wire-level Architecture Curator/Zookeeper for ephemeral cluster membership info Distributed cache (Hazelcast) for metadata, locality information, etc. Curator/Zk Distributed Cache Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node Distributed Cache Distributed Cache Distributed Cache
38. Wire-level Architecture Originating Drillbit acts as foreman: manages query execution, scheduling, locality information, etc. Streaming data communication avoiding SerDe Curator/Zk Distributed Cache Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node Storage Process Drillbit node Distributed Cache Distributed Cache Distributed Cache
39. Wire-level Architecture Foreman turns into root of the multi-level execution tree, leafs activate their storage engine interface. node node node Curator/Zk
41. On the shoulders of giants … Jackson for JSON SerDe for metadata Typesafe HOCON for configuration and module management Netty4 as core RPC engine, protobuf for communication Vanilla Java, LArray and Netty ByteBuf for off-heap large data structures Hazelcast for distributed cache Netflix Curator on top of Zookeeper for service registry Optiq for SQL parsing and cost optimization Parquet (http://parquet.io)/ ORC Janino for expression compilation ASM for ByteCode manipulation Yammer Metrics for metrics Guava extensively Carrot HPC for primitive collections
42. Key features Full SQL – ANSI SQL 2003 Nested Data as first class citizen Optional Schema Extensibility Points …
43. Extensibility Points Source query  parser API Custom operators, UDF  logical plan Serving tree, CF, topology  physical plan/optimizer Data sources &formats  scanner API Sourc e Query Parser Logica l Plan Optimizer Physical Plan Execution
44. User Interfaces API—DrillClient Encapsulates endpoint discovery Supports logical and physical plan submission, query cancellation, query status Supports streaming return results JDBC driver, converting JDBC into DrillClient communication. REST proxy for DrillClient
45. User Interfaces
46. Let’s get our hands dirty…
47. Demo Install Preparation Usage $ wget http://people.apache.org/~jacques/apache-drill-1.0.0- m1.rc3/apache-drill-1.0.0-m1-binary-release.tar.gz $ tar -zxf apache-drill-1.0.0-m1-binary-release.tar.gz $ export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_11.jdk/Content s/Home $ export DRILL_LOG_DIR=$PWD $ ./bin/drillbit.sh start $ ./bin/sqlline -u jdbc:drill:schema=parquet-local -n admin -p admin
48. Useful Resources Getting Started guide https://github.com/vrtx/incubator- drill/blob/getting_started/docs/getting_started.rst Demo HowTo https://cwiki.apache.org/confluence/display/DRILL/ Demo+HowTo How to build/install Apache Drill on Ubuntu 13.04 http://www.confusedcoders.com/bigdata/apache- drill/how-to-build-apache-drill-on-ubuntu-13-04
49. Be a part of it!
50. Status Heavy development by multiple organizations (MapR, Pentaho, Microsoft, Thoughtworks, XingCloud, etc.) Currently more than 100k LOC Alpha available via http://people.apache.org/~jacques/apache-drill- 1.0.0-m1.rc3/
51. Kudos to … Julian Hyde, Pentaho Lisen Mu, XingCloud Tim Chen, Microsoft Chris Merrick, RJMetrics David Alves, UT Austin Sree Vaadi, SSS Srihari Srinivasan, ThoughtWorks Alexandre Beche, CERN Jason Altekruse, MapR http://incubator.apache.org/drill/team.html • Ben Becker, MapR • Jacques Nadeau, MapR • Ted Dunning, MapR • Keys Botzum, MapR • Jason Frantz • Ellen Friedman • Chris Wensel, Concurrent • Gera Shegalov, Oracle • Ryan Rawson, Ohm Data
52. Contributing Contributions appreciated—not only code drops … Test data & test queries Use case scenarios (textual/SQL queries) Documentation
53. Engage! Follow @ApacheDrill on Twitter Sign up at mailing lists (user | dev) http://incubator.apache.org/drill/mailing-lists.html Standing G+ hangouts every Tuesday at 18:00 CET http://j.mp/apache-drill-hangouts Keep an eye on http://drill-user.org/
54. Twitter: @tnachen Email: tnachen@gmail.com