Build & Query Secure S3 Data Lakes with Presto and Apache Ranger

Looking at Presto and Apache Ranger to build a data lakehouse.

Transcript

1. Feb 28 2023 HANDS-ON VIRTUAL LAB Build & Query Secure S3 Data Lakes with Presto and Apache Ranger. 1
2. WELCOME TO THE AHANA HANDS-ON VIRTUAL LAB Your Lab Guide: Rohan Pednekar
3. Over the next 90 minutes you will: • Explore and understand Presto • Get a walk-through of creating a new cluster using Ahana • Query data sitting in S3 using Presto • Convert data to different Formats Using Presto • Get a walk-through of enabling Apache Ranger for Presto • Run SQL query to explore Ranger policies • Run federated queries/joins across multiple sources combining data in S3 and RDS/MYSQL Objective for Today 3
4. Agenda 1) Understand the Technology (40 mins) a) What is Presto? b) What is Ahana Cloud? c) Apache Ranger Integration 2) Getting your hands dirty ( 40 mins) 3) Summary and Close Out (10 mins) 4
5. Understanding The Technology Presto
6. What is Presto? • Open source, distributed MPP SQL query engine • Query in Place • Federated Querying • ANSI SQL Compliant • Designed ground up for fast analytic queries against data of any size • Originally developed at Facebook • Proven on petabytes of data • SQL-On-Anything • Federated pluggable architecture to support many connector • Opensource, hosted under Linux Foundation • https://github.com/prestodb 6
7. Presto Overview 7 Presto Cluster Coordinator Worker Worker Worker Worker
8. Presto – It’s Exploding Presto is De-Facto SQL Engine https://db-engines.com/en/ranking_trend/relational+dbms Spark SQL vs. Presto “As one of our earliest members, Ahana has been strong supporters of the Presto Foundation since its launch in 2019.”
9. Presto Users
10. Presto Use Cases Data Lakehouse analytics Reporting & dashboarding Interactive ad hoc querying Transformation using SQL (ETL) Federated querying across data sources 10
11. Presto Architecture
12. What makes Presto different? Scalable Architecture Pluggable Connectors Performance 12
13. Scalable Architecture • Two roles - coordinator and worker • Easy scale up and scale down • Scale up to 1000 workers • Validated at web scale companies New Worker New Worker Worker Worker Worker Coordinator Data Source Presto Cluster 13 Coordinator
14. Scalable Architecture Parser/analyzer Worker Worker Worker Metadata API Planner Scheduler Data Location API Data Shuffle Data Shuffle Presto Connector Presto Coordinator BI Tools/Notebooks/Clients Presto CLI Looker JDBC Superset ... Tableau Jupyter Result Sets SQL Any Database, Data Stream, or Storage HDFS Object Stores (S3) MySQL ElasticSearch Kafka ... Presto Connector Presto Connector 14
15. Pluggable Presto Connectors 15 https://prestodb.io/docs/current/connector.html
16. Presto Connector Data Model • Connector: Driver for a data source. • Example: HDFS, AWS S3, Cassandra, MySQL, SQL Server, Kafka • Catalog: Contains schemas from a data source specified by the connector • Schemas: Namespace to organize tables. • Tables: Set of unordered rows organized into columns with types. 16
17. Presto Hive Connector – Data File Types • Supported File Types • ORC • Parquet • Avro • RCFile • CSV • No data ingestion/duplication/movement needed • Query data in-place • SequenceFile • JSON • Text 17
18. Introducing Ahana Cloud Fully-Managed Presto Service
19. Ahana Cloud for Presto Managed Service • Enables data platform engineers in minutes vs. days • Fully integrated & pre-configured • No ETL, in-place analytics
20. Ahana Cloud for Presto Ahana Console (Control Plane) CLUSTER ORCHESTRATION CONSOLIDATED LOGGING SECURITY & ACCESS BILLING & SUPPORT In-VPC Presto Clusters (Compute Plane) AD HOC CLUSTER 1 TEST CLUSTER 2 PROD CLUSTER N Glue S3 RDS Elasticsearch Ahana Cloud Account Ahana console oversees and manages every Presto cluster Customer Cloud Account In-VPC orchestration of Presto clusters, where metadata, monitoring, and data sources reside
21. Ahana Cloud – Reference Architecture • Distributed SQL engine with proven scalability • Interactive ANSI SQL queries • Query data where it lives with Federated Connectors (no ETL) • High concurrency • Separation of compute and storage • Cost Management Features 21
22. Apache Ranger with Presto
23. Why Apache Ranger • An Open-Source Authorization Solution • Cloud Agnostic • Fine-grained access control • Audit support • Secured with SSL • Easy to configure with Ahana Cloud 23
24. Ranger Plugin Architecture • Extended to reuse Hive Ranger Plugin • Centralized, fine-grained access control with column-level, row-level policies across all clusters • Supports centralized auditing of user access • Secured with SSL Support • Simplified integration with Ahana Cloud • Enable, monitor and manage comprehensive data security across data lake for the user-triggered Hive or Glue Catalog Queries 24
25. Fine Grained Access Control 25
26. Getting Your Hands Dirty https://bit.ly/lakehouselab
27. Wrapping Up Conclusion and Things to Try
28. Conclusion In this hands-on workshop you have: 1. About Presto and Ahana Cloud 2. How to effortlessly created and managed Presto clusters 3. Run fast SQL federated queries combining datasets from S3 and MySQL 4. Run presto queries with Apache Ranger enabled 28
29. 1. Scale Up and Scale Down your cluster 2. Try queries with more/less workers; how does performance change? 3. Fine grained access control with Lake Formation 4. Table formats for transactions, schema evolution, table versioning with Apache Hudi, Apache Iceberg, Delta Lake Connector Things to try later... 29
30. Next Steps for You... • Ahana Cloud is available on the AWS Marketplace • Sign-up for a 14-day free trial here: https://ahana.io/sign-up • Community Edition - Free forever service 30
31. How to get involved with Presto Join the Slack channel! prestodb.slack.com Write a blog for prestodb.io! prestodb.io/blog Join the virtual meetup group & present! meetup.com/prestodb Contribute to the project! github.com/prestodb 31
32. Questions? 32
33. Thank you! Stay Up-to-Date with Ahana Website: https://ahana.io/ Blogs: https://ahana.io/blog/ Twitter: @ahanaio 33