Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2023 DoK Day: State of State in Kubernetes

Michelle Au
November 07, 2023

2023 DoK Day: State of State in Kubernetes

We discuss Kubernetes feature and best practices for running stateful workloads.

Michelle Au

November 07, 2023
Tweet

More Decks by Michelle Au

Other Decks in Technology

Transcript

  1. Kaslin Fields GKE & OSS K8s Developer Advocate Google Photo

    Michelle Au GKE & OSS K8s Software Engineer Google Intros November 6, 2023 | Chicago, Illinois
  2. Categorizing Workloads in Kubernetes • Deployments ◦ Long-running workloads, state

    is shared across replicas • DaemonSets ◦ Workloads that run on each node in the cluster • Jobs ◦ A workload that needs to run to completion • CronJobs ◦ Workloads that need to run to completion on a time-based schedule • StatefulSets ◦ Volume per replica, more sticky/persistent identity
  3. StatefulSet Manages the deployment and scaling of a set of

    Pods, and provides guarantees about the ordering and uniqueness of these Pods. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of its Pods. Useful for workloads that require: • Stable, unique network identifiers. • Stable, persistent storage. • Ordered, graceful deployment and scaling. • Ordered, automated rolling updates.
  4. What kinds of workloads count as stateful? • Pre-container style

    architectures ◦ Wordpress (Usually Deployment) • Game Servers ◦ https://github.com/saulmaldonado/ago nes-minecraft (CRD) • Things that deal intricately with data ◦ Databases (Usually StatefulSet/CRD) • AI/ML ◦ Training datasets, models, checkpoints (Usually Jobs)
  5. What are the challenges stateful workloads face? • Maintaining a

    consistent identity ◦ Often for connection to other services • High & Consistent Availability ◦ Upgrades must be handled gracefully and carefully ◦ This needs to be up and ready before that ◦ Stateful workloads often have complex start and end processes
  6. What are we doing to address the challenges of Stateful

    workloads? Lifecycle and Day 2 Management • StatefulSet ◦ Ie. PVC deletion policies (beta) • Custom Resources ◦ Custom Resource Definitions ◦ Operators (How Kubernetes runs CRDs)
  7. What are we doing to address the challenges of Stateful

    workloads? Persistent Volumes • Container Storage Interface (CSI) Ecosystem ◦ Over 100 drivers! (Out of tree!) • Dynamic provisioning, resizing • Snapshots, cloning, custom data sources (beta)
  8. Addressing challenges cont’d: Upgrades & Disruption • Fault tolerance ◦

    Pod topology spreading • Workload isolation for critical workloads ◦ Node Affinity, Taints/Tolerations ◦ Pod Priority and Preemption ◦ Pod Resources and QoS
  9. Addressing challenges cont’d: Upgrades & Disruption • Managing Pod eviction

    ◦ Pod Disruption Budgets ◦ Pod readiness probes ◦ Graceful termination, pre-stop hooks • Not doing upgrades is not an option! DO YOUR UPGRADES!
  10. Future / Upcoming k8s and DoK features k8s 1.29 alpha

    features: Modify volumes - use cases like updating IOPS/throughput Beyond: STS volume expansion Group volume snapshots Cross-namespace snapshots (and other data sources) Declarative node maintenance Topology-aware disruptions DoK community developments: Operator feature matrix Security hardening guide
  11. Best Practices for Stateful Workloads on Kubernetes • Use the

    aforementioned features! • Blue/green strategies for upgrades • Chaos testing • Take regular backups ◦ Backups of the data ◦ Backups of the config • Actually test your recovery procedures! • CI/CD best practices apply • General Kubernetes best practices around security and networking apply
  12. Key Takeaways Stateful is more than just databases Kubernetes sees

    a workload as stateful if something cares about its state in some form (not just data!) Kubernetes provides primitives for app lifecycle, storage, scheduling, and graceful disruption management. Look for these types of features for your stateful needs! A good quality operator can simplify and manage complex day 2 workflows Design your application with modern best practices