Day 2 with Stateful Applications - Implementing a Data Management Strategy

As teams start to onboard mission-critical applications such as databases and NoSQL systems into production, there’s a need to address critical Day 2 concerns. Dealing with regulatory requirements, user error, ransomware and cluster upgrades - requires safeguarding of both data and state. This deck describes the challenges associated with implementing a robust data management strategy with Kasten's K10 platform and Rancher's RKE.

It demonstrates not just how to easily implement backup and disaster recovery but also hybrid and multi-cloud application mobility. In particular, it discusses:

- Why legacy data management tools do not work for cloud-native applications
- Using Kasten’s K10 data management platform to simplify complexity for ops teams
- How to implement an effective backup and disaster recovery strategy with Kubernetes
- How to migrate applications across clusters, regions, and clouds

Transcript

1. Day 2 with Stateful Applications - Implementing a Data Management Strategy
2. agenda what we’ll cover page 02 01Where is the Data? Kubernetes Design Patterns, State Explosion, Growth of Storage 03Getting It Right Implementing Data Protection in Kubernetes, Things to Watch Out For 02Cloud-Native Data Management What does this mean? Why do you need it? What are some common misconceptions? 04Demos and Q&A Demos with Kasten K10 and RKE on Azure followed by Q&A.
3. meet our presenter page 03 Niraj Tolia, PhD CEO and Co-Founder Kasten Interests Building lasting technology and teams with a focus on infrastructure and distributed systems. Research Experience Industry Experience
4. meet our presenter page 04 Research Experience Industry Experience Common Theme: Scalable Storage Systems and Data Management Niraj Tolia, PhD CEO and Co-Founder Kasten Interests Building lasting technology and teams with a focus on infrastructure and distributed systems.
5. There is no such thing as a stateless application!
6. kubernetes stateful applications wide variety of patterns page 06 Application uses data services outside of Kubernetes Data services in Kubernetes – separate from Application Application includes data services – all in Kubernetes
7. data management strategy focus on protection and mobility page 07 Infrastructure or Hardware Failure Accidental or Malicious Data Loss Compliance and Auditing Application Misconfiguration Systems in place to recover applications and data if things go bad
8. data management strategy key elements page 08 Schedule and Policies Application-defined backup and retention policies with flexible schedules Automation Support for end-to-end automation Recovery Testing Support for non-destructive recovery and DR tests Security and Encryption Authentication, Authorization, Data always encrypted at rest and in-flight
9. data management strategy key elements page 09 Schedule and Policies Application-defined backup and retention policies with flexible schedules Automation Support for end-to-end automation Recovery Testing Support for non-destructive recovery and DR tests Security and Encryption Authentication, Authorization, Data always encrypted at rest and in-flight Operate At Scale Multi-Cloud, Multi-Cluster, Multi-Team, Multi-App
10. cloud-native data management
11. vms vs. kubernetes fundamental platform differences page 011 Infra and App Changes • Dynamic autoscaling • Frequent rescheduling • No IP/DNS stability and lack of external visibility • Constant application changes and “repaving” • State and services explosion VMs vs. Kubernetes Strong impedance mismatch between solutions built for VMs vs. Cloud-Native Platforms User Changes • Application-oriented platforms • Developers owning full stack & infra-as-code • Ops role change focusing more on self-service • Requirement for cloud- native APIs + integration See https://blog.kasten.io/posts/why-vm-based-data-management-doesnt-work/ for more info
12. vms vs. kubernetes postgresql recovery example Shutdown PostgreSQL Restore DB files + WALs Run PostgreSQL recovery Start PostgreSQL ... ENTRYPOINT ["docker-entrypoint.sh"] EXPOSE 5432 CMD ["postgres"] Scale Down PostgreSQL Create Recovery Pod or Job Restore DB files + WALs Run PostgreSQL recovery Shutdown Recovery Pod Scale Up PostgreSQL Recovery Playbook for PostgreSQL in a VM Recovery Playbook for PostgreSQL on Kubernetes Use container image with Postgres + Tools Run custom commands Attach PostgreSQL volumes (PVCs) Pod will restart on PG shutdown page 012
13. x-ray into a stateful application state explosion page 013 Real Production Application Number Component Type 106 Pods 92 Secrets 87 Services 76 Deployments 70 Certificates 36 ConfigMaps 16 Persistent Volumes 55 Other Components 538 Total
14. infra-centric data management approaches scale poorly and leave organizations exposed page 014 Storage-level infrastructure will take care of it Let me put together a “quick” script Data-store snapshots Weak consistency Complex restore procedure Limited stack options Tailored to application Often tied to infrastructure More complex than expected Difficult to maintain
15. infra-centric data management approaches scale poorly and leave organizations exposed page 015 My storage overlay does backups & migration 2X management complexity Performance cost for overlays Lowest common denominator No fault isolation
16. Kasten K10: Data Protection and Mobility for Kubernetes Backup & Recovery Disaster Recovery Application Mobility Multi & Hybrid Cloud Polyglot Persistence
17. kasten approach: focus on complete application kubernetes resources and persistent state page 017 Perform complete application capture Consistent data and application resources capture Perform coordinated operations Proper sequencing of resource and data operations Abstract underlying infrastructure Seamless support for storage and data services Applications as the operational unit
18. Ops Focused Simplifies compliance management Enables policy-based automation Provides global visibility unique platform approach: application-centric data management page 018 Dev Friendly Preserves developer control Allows extensibility Supports complex applications Data management must start at the app layer (but also envelop infrastructure) Introducing
19. kasten K10 platform app-centric operations at scale page 019 Enterprise Platform Operator Focused Workflows & Scheduling Policy-driven Automation Compliance Monitoring Application Discovery
20. kanister framework extensibility for data management workflows page 020 Open-Source Framework Developer Focused Application-specific data operation blueprints Distributed eventually consistent systems Complex custom applications Application-specific data management recipes Capture workflows as Kubernetes CRs Workflow orchestration functions Kubernetes execution and resource manipulation Data capture and manipulation primitives Block, file, and object store building blocks Extend existing or author own blueprints Collection of blueprints for common services Cassandra
21. K10 infrastructure integration giving customers choice page 021 Storage Systems Public Cloud Storage Kubernetes Public Cloud Solutions On-Premises Distributions On-Premises Storage * Object Storage Data Services Other Data Stores MySQL Elastic MongoDB Postgres AWS EBS/EFS Google Persistent Disk IBM IKS Azure Disk IBM Block and File AWS S3 Google Cloud Storage Minio Azure Blob S3 Compatible Cassandra Local Storage
22. kubernetes integration north and south bound page 022 Native Kubernetes API • Integrates with Kubernetes API server and provides Kubernetes-native API for third-party integration. • Seamless integration with Kubernetes authentication + RBAC Application and Component Discovery • Dynamic discovery of all application components • Handle applications with polyglot data services Complete Application Capture • Capture entire application stack including Kubernetes resources Kubernetes Orchestration • Ordering of operations across Kubernetes controllers • Container idiosyncrasies Custom Workflows • Provide hooks and tooling that allow operators to define custom workflows without getting into Kubernetes specifics (e.g., cloning volumes, extracting a pod’s data) Ecosystem Integration • Cloud-Native (CRD-based) APIs (e.g., use kubectl) • Prometheus/Grafana Integration • Logging Integration
24. K10 + rancher feature demo Shows application-specific dynamic policy creation with compliance, scheduling, visibility, and auto-discovery backup Illustrates how we generate restore points and restore entire application stacks by repaving infrastructure restore Demonstrates cloning different application stacks across clouds and non- federated clusters migrate
25. Application Blueprint K10 architecture a high-level overview page 025 Virtual or Physical Infrastructure Container Orchestration Platform Brownfield Greenfield Brownfield (enhanced) K10-Protected Applications Application Blueprint Greenfield (enhanced) 3 1 Uses Kubernetes API to discover applications and underlying components and perform lifecycle operations. Orchestrator APIs 1 Optional application-centric hooks can be invoked by easy-to-use blueprints defined in the Kanister framework. Application Framework 3 No proprietary storage layer. Minimal integration with infrastructure specific APIs for the following: • Block storage provider - Snapshot functionality, snapshot and block copy • Object/file provider - S3-compatible object store or other file storage like NFS for artifacts Infrastructure APIs 2 2 3
26. cross-infrastructure application migration data portability workflow page 026 Volume Snapshot Portable Conversion App + Volume Rehydration Object Store Source Infrastructure (e.g., a public cloud) Destination Infrastructure (e.g., on-premises infra) • K10 supports migration between on-premises and public clouds • Data is deduplicated for efficient transfer • Data and metadata are always encrypted at rest and in-flight • Customers can provide encryption keys • Intermediate object store can be placed anywhere App Snapshot
27. Try it For Free! Fully-Featured Starter Edition! https://kasten.io/product
28. Kasten: Cloud-Native Data Management Backup & Recovery Disaster Recovery Application Mobility Multi & Hybrid Cloud Polyglot Persistence 01Application Centric Preferred approach given distributed and varied nature of data stores. 04Cloud-Native Designed to run within and integrate with Kubernetes on any public or private cloud 02Extensible Framework Easily customizable to handle any data service or logic. Developers retain choice. 03Separation of Concerns Clean split between policy and mechanism for alignment and definition of responsibilities.