Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Day 2 with Stateful Applications - Implementing...

Kasten
September 17, 2019

Day 2 with Stateful Applications - Implementing a Data Management Strategy

As teams start to onboard mission-critical applications such as databases and NoSQL systems into production, there’s a need to address critical Day 2 concerns. Dealing with regulatory requirements, user error, ransomware and cluster upgrades - requires safeguarding of both data and state. This deck describes the challenges associated with implementing a robust data management strategy with Kasten's K10 platform and Rancher's RKE.

It demonstrates not just how to easily implement backup and disaster recovery but also hybrid and multi-cloud application mobility. In particular, it discusses:

- Why legacy data management tools do not work for cloud-native applications
- Using Kasten’s K10 data management platform to simplify complexity for ops teams
- How to implement an effective backup and disaster recovery strategy with Kubernetes
- How to migrate applications across clusters, regions, and clouds

Kasten

September 17, 2019
Tweet

More Decks by Kasten

Other Decks in Technology

Transcript

  1. agenda what we’ll cover page 02 01Where is the Data?

    Kubernetes Design Patterns, State Explosion, Growth of Storage 03Getting It Right Implementing Data Protection in Kubernetes, Things to Watch Out For 02Cloud-Native Data Management What does this mean? Why do you need it? What are some common misconceptions? 04Demos and Q&A Demos with Kasten K10 and RKE on Azure followed by Q&A.
  2. meet our presenter page 03 Niraj Tolia, PhD CEO and

    Co-Founder Kasten Interests Building lasting technology and teams with a focus on infrastructure and distributed systems. Research Experience Industry Experience
  3. meet our presenter page 04 Research Experience Industry Experience Common

    Theme: Scalable Storage Systems and Data Management Niraj Tolia, PhD CEO and Co-Founder Kasten Interests Building lasting technology and teams with a focus on infrastructure and distributed systems.
  4. kubernetes stateful applications wide variety of patterns page 06 Application

    uses data services outside of Kubernetes Data services in Kubernetes – separate from Application Application includes data services – all in Kubernetes
  5. data management strategy focus on protection and mobility page 07

    Infrastructure or Hardware Failure Accidental or Malicious Data Loss Compliance and Auditing Application Misconfiguration Systems in place to recover applications and data if things go bad
  6. data management strategy key elements page 08 Schedule and Policies

    Application-defined backup and retention policies with flexible schedules Automation Support for end-to-end automation Recovery Testing Support for non-destructive recovery and DR tests Security and Encryption Authentication, Authorization, Data always encrypted at rest and in-flight
  7. data management strategy key elements page 09 Schedule and Policies

    Application-defined backup and retention policies with flexible schedules Automation Support for end-to-end automation Recovery Testing Support for non-destructive recovery and DR tests Security and Encryption Authentication, Authorization, Data always encrypted at rest and in-flight Operate At Scale Multi-Cloud, Multi-Cluster, Multi-Team, Multi-App
  8. vms vs. kubernetes fundamental platform differences page 011 Infra and

    App Changes • Dynamic autoscaling • Frequent rescheduling • No IP/DNS stability and lack of external visibility • Constant application changes and “repaving” • State and services explosion VMs vs. Kubernetes Strong impedance mismatch between solutions built for VMs vs. Cloud-Native Platforms User Changes • Application-oriented platforms • Developers owning full stack & infra-as-code • Ops role change focusing more on self-service • Requirement for cloud- native APIs + integration See https://blog.kasten.io/posts/why-vm-based-data-management-doesnt-work/ for more info
  9. vms vs. kubernetes postgresql recovery example Shutdown PostgreSQL Restore DB

    files + WALs Run PostgreSQL recovery Start PostgreSQL ... ENTRYPOINT ["docker-entrypoint.sh"] EXPOSE 5432 CMD ["postgres"] Scale Down PostgreSQL Create Recovery Pod or Job Restore DB files + WALs Run PostgreSQL recovery Shutdown Recovery Pod Scale Up PostgreSQL Recovery Playbook for PostgreSQL in a VM Recovery Playbook for PostgreSQL on Kubernetes Use container image with Postgres + Tools Run custom commands Attach PostgreSQL volumes (PVCs) Pod will restart on PG shutdown page 012
  10. x-ray into a stateful application state explosion page 013 Real

    Production Application Number Component Type 106 Pods 92 Secrets 87 Services 76 Deployments 70 Certificates 36 ConfigMaps 16 Persistent Volumes 55 Other Components 538 Total
  11. infra-centric data management approaches scale poorly and leave organizations exposed

    page 014 Storage-level infrastructure will take care of it Let me put together a “quick” script Data-store snapshots Weak consistency Complex restore procedure Limited stack options Tailored to application Often tied to infrastructure More complex than expected Difficult to maintain
  12. infra-centric data management approaches scale poorly and leave organizations exposed

    page 015 My storage overlay does backups & migration 2X management complexity Performance cost for overlays Lowest common denominator No fault isolation
  13. Kasten K10: Data Protection and Mobility for Kubernetes Backup &

    Recovery Disaster Recovery Application Mobility Multi & Hybrid Cloud Polyglot Persistence
  14. kasten approach: focus on complete application kubernetes resources and persistent

    state page 017 Perform complete application capture Consistent data and application resources capture Perform coordinated operations Proper sequencing of resource and data operations Abstract underlying infrastructure Seamless support for storage and data services Applications as the operational unit
  15. Ops Focused Simplifies compliance management Enables policy-based automation Provides global

    visibility unique platform approach: application-centric data management page 018 Dev Friendly Preserves developer control Allows extensibility Supports complex applications Data management must start at the app layer (but also envelop infrastructure) Introducing
  16. kasten K10 platform app-centric operations at scale page 019 Enterprise

    Platform Operator Focused Workflows & Scheduling Policy-driven Automation Compliance Monitoring Application Discovery
  17. kanister framework extensibility for data management workflows page 020 Open-Source

    Framework Developer Focused Application-specific data operation blueprints Distributed eventually consistent systems Complex custom applications Application-specific data management recipes Capture workflows as Kubernetes CRs Workflow orchestration functions Kubernetes execution and resource manipulation Data capture and manipulation primitives Block, file, and object store building blocks Extend existing or author own blueprints Collection of blueprints for common services Cassandra
  18. K10 infrastructure integration giving customers choice page 021 Storage Systems

    Public Cloud Storage Kubernetes Public Cloud Solutions On-Premises Distributions On-Premises Storage * Object Storage Data Services Other Data Stores MySQL Elastic MongoDB Postgres AWS EBS/EFS Google Persistent Disk IBM IKS Azure Disk IBM Block and File AWS S3 Google Cloud Storage Minio Azure Blob S3 Compatible Cassandra Local Storage
  19. kubernetes integration north and south bound page 022 Native Kubernetes

    API • Integrates with Kubernetes API server and provides Kubernetes-native API for third-party integration. • Seamless integration with Kubernetes authentication + RBAC Application and Component Discovery • Dynamic discovery of all application components • Handle applications with polyglot data services Complete Application Capture • Capture entire application stack including Kubernetes resources Kubernetes Orchestration • Ordering of operations across Kubernetes controllers • Container idiosyncrasies Custom Workflows • Provide hooks and tooling that allow operators to define custom workflows without getting into Kubernetes specifics (e.g., cloning volumes, extracting a pod’s data) Ecosystem Integration • Cloud-Native (CRD-based) APIs (e.g., use kubectl) • Prometheus/Grafana Integration • Logging Integration
  20. K10 + rancher feature demo Shows application-specific dynamic policy creation

    with compliance, scheduling, visibility, and auto-discovery backup Illustrates how we generate restore points and restore entire application stacks by repaving infrastructure restore Demonstrates cloning different application stacks across clouds and non- federated clusters migrate
  21. Application Blueprint K10 architecture a high-level overview page 025 Virtual

    or Physical Infrastructure Container Orchestration Platform Brownfield Greenfield Brownfield (enhanced) K10-Protected Applications Application Blueprint Greenfield (enhanced) 3 1 Uses Kubernetes API to discover applications and underlying components and perform lifecycle operations. Orchestrator APIs 1 Optional application-centric hooks can be invoked by easy-to-use blueprints defined in the Kanister framework. Application Framework 3 No proprietary storage layer. Minimal integration with infrastructure specific APIs for the following: • Block storage provider - Snapshot functionality, snapshot and block copy • Object/file provider - S3-compatible object store or other file storage like NFS for artifacts Infrastructure APIs 2 2 3
  22. cross-infrastructure application migration data portability workflow page 026 Volume Snapshot

    Portable Conversion App + Volume Rehydration Object Store Source Infrastructure (e.g., a public cloud) Destination Infrastructure (e.g., on-premises infra) • K10 supports migration between on-premises and public clouds • Data is deduplicated for efficient transfer • Data and metadata are always encrypted at rest and in-flight • Customers can provide encryption keys • Intermediate object store can be placed anywhere App Snapshot
  23. Kasten: Cloud-Native Data Management Backup & Recovery Disaster Recovery Application

    Mobility Multi & Hybrid Cloud Polyglot Persistence 01Application Centric Preferred approach given distributed and varied nature of data stores. 04Cloud-Native Designed to run within and integrate with Kubernetes on any public or private cloud 02Extensible Framework Easily customizable to handle any data service or logic. Developers retain choice. 03Separation of Concerns Clean split between policy and mechanism for alignment and definition of responsibilities.