YugaByte DB - A Planet-Scale Database for Low Latency Transactional Apps

Amey Banarse and Karthik Ranganathan introduce and demo YugaByte DB, a large scale DB, highlighting distributed transactions with global consistency.

It is well understood that microservices are multi-cloud deployable in an active-active mode. The challenge has always been the data tier powering the apps which has remained as a high-latency, hard-to-scale master-slave RDBMS for a long time. Multi-master deployments as well as the use of a separate NoSQL DB for multi-region data distribution are simply band-aids to this problem and do not deliver the desired business outcomes. YugaByte DB, a planet-scale, highly performant, ACID compliant and transactional Database is a startup founded by the engineers who were instrumental in creating Cassandra inside FaceBook. Karthik from YugaByte and Amey from Pivotal are presenting YugaByte DB running across multiple datacenters with a live demo of a large scale ecommerce app powered by Spring Framework. The talk will also highlight the distributed transactions capabilities with global consistency and tunable reads as well as ease of use for developers using familiar Spring Data Cassandra and Redis modules.

https://www.youtube.com/watch?v=0neiV7Qbwgg

https://www.infoq.com/presentations/yugabytedb/

Transcript

1. YugaByte DB - A Planet-Scale DB for Low Latency Transactional Apps Amey Banarse Advisory Platform Architect, Pivotal @ameybanarse Karthik Ranganathan Founder & CTO, Yugabyte Inc @karthikr
2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Introductions 2  Amey Banarse Advisory Platform Architect, Pivotal University of Pennsylvania @ameybanarse Karthik Ranganathan Co-Founder and CTO, YugaByte DB Nutanix ♦ Facebook ♦ Microsoft University of Texas, Austin @karthikr
3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is YugaByte DB? 3  NoSQL + SQL Cloud Native
4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 4  A transactional, high-performance database for building planet-scale cloud services.
5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 5 YugaByte Story starts with...
6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Facebook in 2007 6 
7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Facebook in 2008-2009 7  How to scale to a billion users? Also: how to survive the week?
8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What happens at 1 Billion users? 8  Dozens of Petabytes Billions of IOPS Scale out frequently Rolling upgrades – zero downtime!
9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Transformation of Facebook 9  How to achieve scale Few datacenters traditional servers Monolithic applications and services Geo-distributed DC’s Containers for applications Microservices based design All about developer agility
10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The database tier was difficult to transform 1 0  Application Tier (Stateless Microservices) Custom Data Platform Transactional, Performant, Global No general platform for the enterprise
11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11 So what...?
12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Your Challenges 1 2  Your approach to solving these? • Millions of IOPS (not billions) • 1000s of apps (not 10s of apps) • Want standard open APIs (not custom)
13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How Pivotal Transforms Customers Changes that are already underway 2 corporate datacenters traditional servers monolithic apps on proprietary app servers Multi-region and Multi-cloud Multiple VM’s & Containers Microservices on OSS Frameworks
14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Transformation for Data Tier ? legacy databases Open Source, Cloud-native Databases
15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Use Existing DBs for Transformation? Not Portable Not Portable Open Source Not Portable Open Source Open Source High Performance, Transactional, Planet-Scale High Performance, Transactional, Planet-Scale High Performance, Transactional, Planet-Scale High Performance, Transactional, Planet-Scale
16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Transformation for Data Tier Legacy Databases Open Source, Cloud-native Databases +
17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ YugaByte DB capabilities TRANSACTIONAL PLANET-SCALE HIGH PERFORMANCE Distributed ACID Transactions Transactional Key-to-Document Storage Engine Low Latency, Tunable Reads High Throughput OPEN SOURCE Apache 2.0 Popular APIs Extended PostgreSQL, Apache Cassandra and Redis Auto Sharding & Rebalancing Global Data Distribution CLOUD-NATIVE Natively runs in Containers, self service Self-Healing, Fault-Tolerant
18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Core DB Features SQL Strong consistency Secondary indexes ACID transactions Expressive query language NoSQL Tunable read latency Write optimized for large data sets Data expiry with TTL Scale out and fault tolerant
19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Best of Cloud-Native Meets Open Source ACID Transactions Globally Consistent SQL API only Not Transactional Multi-Model High Performance Not Globally Consistent Lower Performance
20. 20 YugaByte Confidential © 2018 All rights reserved. YugaByte DB Architecture • YB-Master and YB-TServer services • Raft consensus • DocDB - optimized RocksDB storage • Global Transactions • Pluggable Query Layer - CQL, Redis
21. 21 YugaByte Confidential © 2018 All rights reserved. Ecosystem Integration tablet 1’ tablet2-leade r tablet1-leade r … … tablet2-followe r tablet3-followe r tablet1-followe r tablet1-followe r • Spring Data • Kafka • Spark • Presto • Janus Graph DB tablet 1’ tablet 1’ Raft Replication DocDB Storage node1 node2 node3 Global Transaction Manager Automated Sharding & Load Balancing Cassandra REDIS BET A
22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2 2 On Demand YugaByte DB on PKS Demo http://bit.ly/yugabyte-on-pks
23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Real World Retail Marketplace • Browse and Buy millions of products • Determine Ratings and Review effectiveness • Order and checkout products • ….. Many more. Last but not the least, a highly performant, rich and interactive UX 2 3
24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dataset used for the demo • 20 million+ Products • 100 million reviews + ratings • …. and assume newer Reviews are written to YugaByte DB in real-time 2 4
25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 2 5 Let’s take a look at the UI and work our way back to the underlying architecture http://kronos-react-ui.apps.sebastopol.cf-app.com/
26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 2 6 Http Requests Retail App
27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 2 7 Http Requests Spring Boot Microservices Web App
28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 2 8 Http Requests Spring Boot Microservices Web App YugaByte DB on PKS Transactions, High Consistency, Secondary Indexes
29. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 2 9 AWS S3 Batch Load Http Requests Spring Boot Microservices Web App Spring Cloud Data Flow Transactions, High Consistency, Secondary Indexes YugaByte DB on PKS High Throughput Data Pipelines
30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 3 0 AWS S3 Batch Load . Http Requests Spring Boot Microservices Web App Spring Cloud Data Flow Transactions, High Consistency, Secondary Indexes YugaByte DB on PKS Spark on PKS High Throughput Data Pipelines
31. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Retail Marketplace Architecture 3 1 BOSH AWS S3 Pivotal Container Service Batch Load . Http Requests Spring Boot Microservices Web App Spring Cloud Data Flow Pivotal Application Service Transactions, High Consistency, Secondary Indexes YugaByte DB on PKS Spark on PKS High Throughput Data Pipelines
32. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 3 2 Dissecting the Retail App
33. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dissecting the Retail App 3 3 Secondary Indexes for category listings
34. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Secondary Indexes Distributed Indexes for Efficient Queries 3 4 CREATE TABLE product_rankings ( asin text, category text, sales_rank int, ... PRIMARY KEY (asin, category) ); CREATE INDEX top_products_in_category ON product_rankings (category, sales_rank, asin); Partition by product ID for efficient lookups Secondary index to list top products in a category
35. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Secondary Indexes Distributed Indexes for Efficient Queries 3 5 SELECT * FROM product_rankings WHERE asin = '0684841363'; SELECT * FROM product_rankings WHERE category = 'Books' LIMIT 10 OFFSET 20; Query by ID for product summary data Query by index to list top products in a category by sales rank (best sellers)
36. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dissecting the Retail App 3 6 Global ACID Transactions using Cassandra API
37. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Global Transactions Multi-Row/Multi-Shard Operations At Scale 3 7 CREATE TABLE orders ( order_id text PRIMARY KEY, user_id uuid, order_details jsonb, ... ) WITH transactions = {'enabled': 'true'}; Update inventory and orders tables atomically. Simple to enable global transactions on any table Use JSON type for flexible schema objects.
38. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Global Transactions Multi-Row/Multi-Shard Operations At Scale 3 8 BEGIN TRANSACTION UPDATE product_inventory SET quantity = quantity - 2 WHERE asin = '0684841363'; INSERT INTO orders (order_id, user_id, order_details, … ) VALUES ('<order-id>', '<user-id>', '{id: "0684841363", quantity: 2, … }', … ); END TRANSACTION; Decrease inventory count for products fulfilled. Query and index by JSON attributes if needed! Add to orders table for products purchased.
39. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dissecting the Retail App 3 9 Dynamic sets modeled as Redis SORTED SETs Show me the top 2 book with the most reviews
40. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Dissecting the Retail App 4 0 Pagination Support
41. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ UI Pagination and Offsets 41 SELECT * FROM cronos.products WHERE category=’Music’ LIMIT 10 OFFSET 20; Example command for displaying page 3
42. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Native JSON Modeling product attributes & order history 4 2
43. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Auto Data Expiry with TTL Implement Discounts and Sales Offers on products 4 3 Query the key right away Query the key after 10 seconds Write a key with a 10 second expiry
44. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spark Integration • Analytics and Aggregate Queries • ML workloads - Recommendations and Rankings • Uses pySpark, OSS Spark-Cassandra connectors 4 4 Source Tables Derived Tables Enrichment / pre-aggregation Batch Aggregates Pivotal Container Service
45. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Real-time + Batch Data Ingestion into YugaByteDB • High throughput ingestion pipeline • Scalable Data Microservices 4 5 Scalable Data Pipelines Data Processor S3 Source Cassandra Sink Read From Write To AWS S3
46. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ YugaWare Console Day 2 Automation and Operations • Integrated Prometheus for monitoring • Alerts & Health Checks • Backup/Restore from Object Stores(ex.S3) & NFS • Rolling Upgrades 4 6
47. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 4 7 Real World Use-Cases
48. 48 YugaByte Confidential © 2018 All rights reserved. 1. MySQL master-slave replication 2. Cassandra cross-DC queue for cache updates 3. Per-DC Couchbase for caching Current State Use Case #1 – Global User Identity Login, change password, view profile
49. 49 YugaByte Confidential © 2018 All rights reserved. With YugaByte DB Use Case #1 – Global User Identity Login, change password, view profile Unified platform Zero data loss even on region failures Add new regions with ease 1-click Deployment of Primary Cluster and Read Replicas Read Replicas
50. 50 YugaByte Confidential © 2018 All rights reserved. Redis cluster for low latency reads Fragile (manually sharded & load balanced) Expensive (entire dataset in memory) On-premises only, need hybrid/public cloud scaling and distribution DB Current State Use Case #2 - Financial Data Service
51. 51 YugaByte Confidential © 2018 All rights reserved. With YugaByte DB Use Case #2 - Financial Data Service Higher release velocity Cost-efficient storage Faster cloud migration 1-click Deploy of Redis as a Distributed Database
52. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Use Case 3: Forensic Tools for Blockchain Industry 5 2  “YugaByte DB helped CipherTrace meet the high bar its customers set for availability, scalability, and data consistency while we simultaneously introduced new features. Because YugaByte DB is Cassandra wire compatible, it was a simple process for us to quickly migrate our application from DataStax Enterprise and start delivering these benefits to our customers.” — Steve Ryan, Chief Operating Officer, CipherTrace
53. 53 YugaByte Confidential © 2018 All rights reserved. Spark Integration for AI/ML Realtime analytics on top of transactional data without ETL YCQL 1 2 3
54. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Looking Forward... • Multi-site PKS capabilities, collaborating with K8s SIG • Spring Data YugaByte module incubation • Distributed PostgreSQL support • Jepsen Performance Testing and Node density benchmarks 5 4