How to Run Distributed SQL Databases on Kubernetes

Presented by : Amey Banarse, VP of Product @ Yugabyte

Kubernetes has hit a home run for stateless workloads, but can it do the same for stateful services such as distributed databases? Before we can answer that question, we need to understand the challenges of running stateful workloads on, well anything. In this talk, we will first look at which stateful workloads, specifically databases, are ideal for running inside Kubernetes. Secondly, we will explore the various concerns around running databases in Kubernetes for production environments, such as:

- The production-readiness of Kubernetes for stateful workloads in general
- The pros and cons of the various deployment architectures
- How much performance may be lost when performing IO inside containers
- The failure characteristics of a distributed database inside containers

In this session, we will demonstrate what Kubernetes brings to the table for stateful workloads and what database servers must provide to fit the Kubernetes model. This talk will also highlight some of the modern databases that take full advantage of Kubernetes and offer a peek into what’s possible if stateful services can meet Kubernetes halfway. We will go into the details of deployment choices, how the different cloud-vendor managed container offerings differ in what they offer, as well as compare performance and failure characteristics of a Kubernetes-based deployment with an equivalent VM-based deployment.

Transcript

1. © 2020 - All Rights Reserved 1 YugabyteDB – Distributed SQL Database on Kubernetes Amey Banarse VP of Product, Yugabyte, Inc. Taylor Mull Senior Data Engineer, Yugabyte, Inc.
2. © 2020 - All Rights Reserved Introduction – Amey 2 2 VP of Product, Yugabyte, Inc. Pivotal • FINRA • NYSE University of Pennsylvania (UPenn) @ameybanarse about.me/amey Amey Banarse
3. © 2020 - All Rights Reserved Introduction – Taylor 3 3 Senior Data Engineer, Yugabyte, Inc. DataStax • Charter University of Colorado at Boulder Taylor Mull
4. © 2020 - All Rights Reserved Kubernetes Is Massively Popular in Fortune 500s 4 • Walmart – Edge Computing KubeCon 2019 https://www.youtube.com/watch?v=sfPFrvDvdlk • Target – Data @ Edge https://tech.target.com/2018/08/08/running-cassandra-in-kubernetes -across-1800-stores.html • eBay – Platform Modernization https://www.ebayinc.com/stories/news/ebay-builds-own-servers-intends -to-open-source/ 4
5. © 2020 - All Rights Reserved The State of Kubernetes 2020 5 5 • Substantial Kubernetes growth in Large Enterprises • Clear evidence of production use in enterprise environments • On-premises is still the most common deployment method • Though there are pain points, most developers and executives alike feel Kubernetes is worth it VMware The State of Kubernetes 2020 report https://tanzu.vmware.com/content/ebooks/the-state-of-kubernetes-2020 https://containerjournal.com/topics/container-ecosystems/vmware-releases-state-of-kubernetes-2020-report/
6. © 2020 - All Rights Reserved Data on K8s Ecosystem Is Evolving Rapidly 6 6
7. © 2020 - All Rights Reserved 7 Why Data Services on K8s? Containerized data workloads running on Kubernetes offer several advantages over traditional VM / bare metal based data workloads including but not limited to: • Robust automation framework can be embedded inside CRDs (Custom Resource Definitions) or commonly referred as ‘K8s Operator’ • Simple and selective instant upgrades • Better cluster resource utilization • Portability between cloud and on-premises • Self Service Experience and seamlessly Scale on demand during peak traffic
8. © 2020 - All Rights Reserved 8 A Brief History of Yugabyte Part of Facebook’s cloud native DB evolution • Yugabyte team dealt with this growth first hand • Massive geo-distributed deployment given global users • Worked with world-class infra team to solve these issues Builders of multiple popular databases +1 Trillion ops/day +100 Petabytes data set sizes Yugabyte founding team ran Facebook’s public cloud scale DBaaS
9. © 2020 - All Rights Reserved 9 Transactional, distributed SQL database designed for resilience and scale 100% open source, PostgreSQL compatible, enterprise-grade RDBMS …..built to run across all your cloud environments
10. © 2020 - All Rights Reserved Enabling Business Outcomes 10 • Real-time APIs for financial services data • 14 Billion requests per day with data from over 100 world-wide exchanges • Serves major financial tech and services firms from Betterment to Schwab • Retail personalization platform serving 600+ retailers like Walmart and Nike • Designed to be Multi-cloud/Multi-AZ and tuned to handle Black Friday • System of Record driving shopping list service • Designed to be Multi-Cloud on GCP and Azure • Scaling to 42 states, and 9m shoppers
11. © 2020 - All Rights Reserved YugabyteDB – The Promise of Distributed SQL 11 100% Open Source Multi/ Hybrid cloud /K8s Geo Distribution Resilience & HA Low Latency PostgreSQL compatible SQL Horizontal Scalability
12. © 2020 - All Rights Reserved Designing the Perfect Distributed SQL Database 12 PostgreSQL is more popular than MongoDB Aurora much more popular than Spanner Amazon Aurora Google Spanner A highly available MySQL and PostgreSQL-compatible relational database service Not scalable but HA All RDBMS features PostgreSQL & MySQL The first horizontally scalable, strongly consistent, relational database service Scalable and HA Missing RDBMS features New SQL syntax bit.ly/distributed-sql-deconstructed
13. © 2020 - All Rights Reserved Designed for Cloud Native Microservices 13 Sharding & Load Balancing Raft Consensus Replication Distributed Transaction Manager & MVCC Document Storage Layer Custom RocksDB Storage Engine DocDB Distributed Document Store Yugabyte Query Layer YCQL YSQL PostgreSQL Google Spanner YugabyteDB SQL Ecosystem ✓ Massively adopted ✘ New SQL flavor ✓ Reuse PostgreSQL RDBMS Features ✓ Advanced Complex ✘ Basic cloud-native ✓ Advanced Complex and cloud-native Highly Available ✘ ✓ ✓ Horizontal Scale ✘ ✓ ✓ Distributed Txns ✘ ✓ ✓ Data Replication Async Sync Sync + Async
14. © 2020 - All Rights Reserved All Nodes Are Identical 14 … … YugabyteDB Query Layer YugabyteDB Query Layer YugabyteDB Query Layer DocDB Storage Layer DocDB Storage Layer DocDB Storage Layer Can connect to ANY node Add/remove nodes anytime YugabyteDB Node YugabyteDB Node YugabyteDB Node Microservices platform
15. © 2020 - All Rights Reserved YugabyteDB Deployed as StatefulSets 15 node2 node1 node4 node3 yb-master StatefulSet yugabytedb yb-master-1 pod yugabytedb yb-master-0 pod yugabytedb yb-master-2 pod yb-tserver StatefulSet tablet 1’ yugabytedb yb-tserver-1 pod tablet 1’ yugabytedb yb-tserver-0 pod tablet 1’ yugabytedb yb-tserver-3 pod tablet 1’ yugabytedb yb-tserver-2 pod … Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume Local/Remote Persistent Volume yb-masters Headless Service yb-tservers Headless Service App Clients Admin Clients
16. © 2020 - All Rights Reserved Under the Hood – 3 Node Cluster 16 DocDB Storage Engine Purpose-built for ever-growing data, extended from RocksDB yb-master1 yb-master3 yb-master2 YB-Master Manage shard metadata & coordinate cluster-wide ops node1 node3 node2 Global Transaction Manager Tracks ACID txns across multi-row ops, incl. clock skew mgmt. Raft Consensus Replication Highly resilient, used for both data replication & leader election tablet 1’ tablet 1’ yb-tserver1 yb-tserver2 yb-tserver3 tablet 1’ tablet2-leader tablet3-leader tablet1-leader YB-TServer Stores/serves data in/from tablets (shards) tablet1-follower tablet1-follower tablet3-follower tablet2-follower tablet3-follower tablet2-follower … … …
17. © 2020 - All Rights Reserved 17 Deployment Topologies 1. Single Region, Multi-Zone Availability Zone 1 Availability Zone 2 Availability Zone 3 Consistent Across Zones No WAN Latency But No Region-Level Failover/Repair 2. Single Cloud, Multi-Region Region 1 Region 2 Region 3 Consistent Across Regions Cross-Region WAN Latency with Auto Region-Level Failover/Repair 3. Multi-Cloud, Multi-Region Cloud 1 Cloud 2 Cloud 3 Consistent Across Clouds Cross-Cloud WAN Latency with Auto Cloud-Level Failover/Repair
18. © 2020 - All Rights Reserved Master Cluster 1 in Region 1 Consistent Across Zones No Cross-Region Latency for Both Writes & Reads App Connects to Master Cluster in Region 2 on Failure Master Cluster 2 in Region 2 Consistent Across Zones No Cross-Region Latency for Both Writes & Reads App Connects to Master Cluster in Region 1 on Failure Bidirectional Async Replication Availability Zone 2 Availability Zone 3 Availability Zone 2 Availability Zone 3 Availability Zone 1 Availability Zone 1 Multi-Master Deployments w/ xCluster Replication 18
19. © 2020 - All Rights Reserved Deploying Yugabyte on Multi-Region K8s • Scalable and highly available data tier • Business continuity • Geo-partitioning and data compliance
20. © 2020 - All Rights Reserved 20 YugabyteDB on K8s Multi-Region Requirements • Pod to pod communication over TCP ports using RPC calls across n K8s clusters • Global DNS Resolution system ◦ Across all the K8s clusters so that pods in one cluster can connect to pods in other clusters • Ability to create load balancers in each region/DB • RBAC: ClusterRole and ClusterRoleBinding • Reference: Deploy YugabyteDB on multi cluster GKE https://docs.yugabyte.com/latest/deploy/kubernetes/multi-cluster/gke/helm-chart/
21. © 2020 - All Rights Reserved 21 YugabyteDB on K8s Demo - Single YB Universe deployed on with 3 GKE clusters
22. © 2020 - All Rights Reserved YugabyteDB Universe on 3 GKE clusters Deployment: 3 GKE clusters Each with 3 x N1 Standard 8 nodes 3 pods in each cluster using 4 cores Cores: 4 cores per pod Memory: 7.5 GB per pod Disk: ~ 500 GB total for universe
23. © 2020 - All Rights Reserved 23 Yugabyte Platform Demo
24. © 2020 - All Rights Reserved 24 Ensuring High Performance LOCAL STORAGE Since v1.10 REMOTE STORAGE Lower latency, Higher throughput Recommended for workloads that do their own replication Pre-provision outside of K8s Use SSDs for latency-sensitive apps Higher latency, Lower throughput Recommended for workloads do not perform any replication on their own Provision dynamically in K8s Use alongside local storage for cost-efficient tiering Most used
25. © 2020 - All Rights Reserved 25 Configuring Data Resilience POD ANTI-AFFINITY MULTI-ZONE/REGIONAL/MULTI-REGION POD SCHEDULING Pods of the same type should not be scheduled on the same node Keeps impact of node failures to absolute minimum Multi-Zone – Tolerate zone failures for K8s worker nodes Regional – Tolerate zone failures for both K8s worker and master nodes Multi-Region / Multi-Cluster – Requires network discovery between multi cluster
26. © 2020 - All Rights Reserved 26 BACKUP & RESTORE Backups and restores are a database level construct YugabyteDB can perform distributed snapshot and copy to a target for a backup Restore the backup into an existing cluster or a new cluster with a different number of TServers ROLLING UPGRADES Supports two upgradeStrategies: onDelete (default) and rollingUpgrade Pick rolling upgrade strategy for DBs that support zero downtime upgrades such as YugabyteDB New instance of the pod spawned with same network id and storage HANDLING FAILURES Pod failure handled by K8s automatically Node failure has to be handled manually by adding a new slave node to K8s cluster Local storage failure has to be handled manually by mounting new local volume to K8s Automating Day 2 Operations
27. © 2020 - All Rights Reserved 27 https://github.com/yugabyte/yugabyte-platform-operator Based on Custom Controllers that have direct access to lower level K8S API Excellent fit for stateful apps requiring human operational knowledge to correctly scale, reconfigure and upgrade while simultaneously ensuring high performance and data resilience Complementary to Helm for packaging Extending StatefulSets with Operators CPU usage in the yb-tserver StatefulSet Scale pods CPU > 80% for 1min and max_threshold not exceeded
28. © 2020 - All Rights Reserved 28 Yugabyte Platform Demo
29. © 2020 - All Rights Reserved Tablet Server-4 Live Demo: Cluster Scale Up Confidential Tablet Server-1 t3 t1 t2 t4 Tablet Server-2 t3 t1 t2 t4 Tablet Server-3 t3 t1 t2 t4 1 Before expansion, all 3 nodes taking traffic 2 New node just added. No traffic to new node yet Tablet Server-1 t3 t1 t2 t4 Tablet Server-2 t3 t1 t2 t4 Tablet Server-3 t3 t1 t2 t4 Tablet Server-4 3 New node received t2 from current Leader, which is guaranteed to have consistent copy. A simple file transfer and with just that one Tablet, it is ready to take traffic. Expansion is still in progress Tablet Server-1 t3 t1 t4 Tablet Server-2 t3 t1 t2 t4 Tablet Server-3 t3 t1 t4 Tablet Server-4 t2 4 Zero-downtime cluster expansion and much faster because of Raft and strong consistency t2 Tablet Server-1 t3 t1 t4 Tablet Server-2 t3 t1 t2 t4 Tablet Server-3 t1 t4 t2 t2 t3 t2
30. © 2020 - All Rights Reserved 30 Terminate a YB Node ➜ ~ kubectl delete pods yb-tserver-1 -n yb-dev-yb-webinar-us-west1-c ➜ ~ kubectl get pods -n yb-dev-yb-webinar-us-west1-c
31. © 2020 - All Rights Reserved 31 A Classic Enterprise App Scenario
32. © 2020 - All Rights Reserved Yugastore – E-Commerce app : A Real-World Demo 32 Deployed on https://github.com/yugabyte/yugastore-java
33. © 2020 - All Rights Reserved Yugastore – Kronos Marketplace 33
34. © 2020 - All Rights Reserved Classic Enterprise Microservices Architecture 34 CART MICROSERVICE PRODUCT MICROSERVICE API Gateway CHECKOUT MICROSERVICE UIU Yugabyte Cluster YSQL YCQL YCQL UI APP REST .
35. © 2020 - All Rights Reserved Istio Traffic Management for Microservices 35 CART MICROSERVICE PRODUCT MICROSERVICE API Gateway CHECKOUT MICROSERVICE UIU UI APP Galley Citadel Pilot Istio Edge Proxy Istio Control Plane Istio Service Discovery Istio Edge Gateway Istio Route Configuration using Envoy Proxy
36. © 2020 - All Rights Reserved A Platform Built for a New Way of Thinking 36 ➔ Event + Microservice first design ➔ Team autonomy with platform efficiency ➔ 100% Cloud Native operating model on K8s ➔ Turnkey multi-cloud ➔ Full Spring Data support
37. © 2020 - All Rights Reserved We’re a fast growing project Growth in 1 Year Clusters ▲ 12x Slack ▲ 12x GitHub Stars ▲ 7x We 💛 stars! Give us one: github.com/YugaByte/yugabyte-db Join our community: yugabyte.com/slack
38. © 2020 - All Rights Reserved 38 Thank You Join us on Slack: yugabyte.com/slack Star us on GitHub: github.com/yugabyte/yugabyte-db