Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OAM as Kubernetes Application Definition

OAM as Kubernetes Application Definition

This slides is about how to use OAM as Kubernetes application definition and build standard but extensible application platform with it. Real world use cases in Alibaba are included as well.

Lei (Harry) Zhang

June 25, 2020
Tweet

More Decks by Lei (Harry) Zhang

Other Decks in Technology

Transcript

  1. Application Component 1 Component 2 scaling auto route /index security

    group public scaling manual rollout strategy canary security group private The “Modern Application”
  2. Application Component 1 Operational Configurations Component 2 Operational Strategy 1

    Operational Strategy 3 Operational Strategy 2 Operational Strategy 4 How to group? How to group? What to run? What to run? The Pattern Behind “Modern Application”
  3. Application Component ApplicationConfiguration Workload Component Workload Trait Trait Trait Trait

    Scope Scope What to run? How to operate? trait scope Open Application Model
  4. Application Component ApplicationConfiguration Component scaling: auto scaling: manual route: /index

    rollout strategy: canary security group: public security group: private What to run? How to operate? trait scope Open Application Model as Kubernetes Application Definition They are all K8s API resources
  5. Component Component B Component C Component D Application is composed

    by Components which defines ”what to run” apiVersion: core.oam.dev/v1alpha2 kind: Component metadata: name: frontend annotations: description: Container workload spec: workload: # a k8s workload CR apiVersion: apps/v1 kind: Deployment spec: containers: - name: web image: php:latest env: - name: OAM_TEXTURE value: texture.jpg ports: - containerPort: 8001 name: http protocol: TCP Component A $ kubectl get components NAME WORKLOAD frontend deployment.apps.k8s.io
  6. Component Component B Component C Component D Workload is pluggable

    in Component apiVersion: core.oam.dev/v1alpha2 kind: Component metadata: name: redis annotations description: Azure RedisCache Instance spec: workload: # a redis instance provided by azure operator apiVersion: azure.microsoft.com/v1alpha1 kind: RedisCache metadata: labels: tag1: value1 tag2: value2 spec: location: westus resourceGroup: resgrp-azure-operators properties: sku: name: Basic family: C capacity: 1 enableNonSslPort: true Component A $ kubectl get components NAME WORKLOAD frontend deployment.apps.k8s.io redis azure.microsoft.com
  7. Trait and Application Configuration Trait: declarative operational combabilities apiVersion: core.oam.dev/v1alpha2

    kind: ApplicationConfiguration metadata: name: helloworld spec: components: # 1st component - componentName: frontend traits: - trait: apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler spec: minReplicas: 1 maxReplicas: 10 - trait: apiVersion: networking.alibaba-inc.com/v1 kind: APIGateway spec: hostname: app.alibaba.com path: / service_port: 8001 # 2nd component - componentName: redis Application Application Configuration Component B Component C Component D Traits Traits Traits Component A Traits ApplicationConfiguration: assigning traits to components
  8. Definition Object Register and discover workloads and traits API resources

    apiVersion: core.oam.dev/v1alpha2 kind: TraitDefinition metadata: name: manualscaler.core.oam.dev annotations: alias: scale spec: # can only be bound to api resources under apps.k8s.io group appliesTo: - apps.k8s.io # CRD of ManualScaler definition: manualscaler.core.oam.dev --- apiVersion: core.oam.dev/v1alpha2 kind: WorkloadDefinition metadata: name: service.serving.knative.dev annotations: alias: ksvc spec: definitionRef: name: service.serving.knative.dev $ kubectl get traits NAME DEFINITION APPLIES TO scale manualscaler.core.oam.dev apps.k8s.io route route.core.oam.dev apps.k8s.io tls tls.core.oam.dev apps.k8s.io $ kubectl get workloads NAME DEFINITION deployment apps.k8s.io ksvc service.serving.knative.dev
  9. apiVersion: core.oam.dev/v1alpha12 kind: ApplicationConfiguration metadata: name: helloworld spec: components: -

    componentName: frontend traits: - trait: apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: scale-hello spec: minReplicas: 1 maxReplicas: 10 - trait: apiVersion: networking.Alibaba-inc.com/v1 kind: APIGateway metadata: name: api spec: hostname: app.alibaba.com path: / service_port: 8001 applicationScopes: - my-vpc-network-public - componentName: redis applicationScopes: - my-vpc-network-private Application Network Scope Y Network Scope X Application Configuration Component B Component C Component D Trait Trait Trait Component A Trait A way to loosely couple components into groups with common characteristics. Application Scope
  10. And More ... • Out-of-box support • Dependency guarantee •

    e.g. PHP component waits for MySQL component to be ready (by status field, not just container started) • Declare data input/output • e.g. auto inject credential from MySQL component to PHP component’s ENV • Revision system • every change of Component object will generate a revision and can be tracked and referenced by ApplicationConfiguration/Trait
  11. Seamlessly Integrated with the Ecosystem OAM YAML files OAM Platform

    Kubernetes Cloud Resources GitOps Helm, CNAB, OCI packaging deploy OAM Spec What to run: Components Operational capability:Traits Container Function VM Cloud Resource scaling routing ingress manually create Prometheus Istio ... Helm, Kustomize, DSL (e.g. cdk8s) trigger
  12. A Typical OAM based App Platform Kubernetes (+ Crossplane) Components

    Traits/Scope ApplicationConfiguration Container Function Cloud Resources Traffic Management Canary Blue-Green A/B testing Scaling Dev Ops A Micro-App Engine built by OAM Traits System + Registry CLI/UI/Dashboard Capability System (Automatically expose capabilities from installed plugins)
  13. Summary of OAM • Right level of abstraction: developer level

    abstraction to consolidate the application management experience, • Team-centric: separate concerns for developers, operators, and infra engineers. • Platform agnostic: deploy with same YAML to serverless platform, k8s on prems, or any cloud. • Manageability and discoverability: detect conflicted operational strategies at deploy
  14. Call for collaboration • Open Application Model community (https://oam.dev) •

    The Model • https://github.com/oam-dev/spec#community • The OAM Kubernetes Plugin (join effort with Crossplane ) • https://github.com/crossplane/oam-kubernetes-runtime • What’s next? • A OAM based Micro-App Engine!
  15. Scale: - 10,000 nodes/cluster - 100,000 apps/cluster - 1,000,000 containers/cluster

    Velocity: - 100,000 deploys/day - 500~1000 replicas/app Gateway Route Traffic Monitor Alert Deployment App Instance HPA Function Challenges • Fragmentation in app layer, ~11 internal PaaS/Serverless • Siloes, lack reusability and interoperability • Infra centric, low level primitives leaked to end user Gateway Route Traffic Rollout Job Infra Ops Developers Operators Deployment Route Service Job PaaS A PaaS B Serverless C Application Management At Alibaba
  16. OAM in Alibaba at highest level App PaaS Operator platform

    Serverless Hybrid Cloud ISV IoT cloud resource definition service binding DevOps model workload definition trait definition OAM Standard Scenarios on Alibaba Cloud App Marketplace CI/CD Pipeline
  17. App/Serverless platforms in Alibaba • Alibaba Application Platform (EDAS) •

    distributed Java application platform • Public product • Alibaba Operator Platform • Stateful application platform • Internal platform • Alibaba Serverless Platform • Serverless (event driven) application platform • Both public product and internal platform
  18. OAM + Alibaba Application PaaS (EDAS) Feature OAM Category Description

    • ContainerizedWorkload • component • Stateless app • Virtual Machine • component • Legacy app • traffic rollout • trait • Traffic shifting • monitoring • trait • Prometheus configuration • logging • trait • SLS log collecting policy • auto-scaler • trait • HPA • manual-scaler • trait • Manual scaling • rollout • trait • Blue-green deployment • ingress • trait • Routing policy Balance extensibility of k8s with best end-user experience by OAM based abstractions Reusable components and traits to save engineering time Discoverability for capabilities in the system, detect strategy conflict at deploy • Before: traditional PaaS with close ecosystem • After: OAM as core application model, provide simplified end user experience but still keep the extensibility of k8s, make EDAS become into an open ecosystem Share with other platforms
  19. OAM + Alibaba Operator Platform feature OAM Category Description •

    OpenKruise • component • Alternative to StatefulSet • Operators • component • Workloads based on operator • In-place rollout • trait • In-place rollout • canary • trait • Canary deployment • khpa • Trait • HPA to scale Operators • pvc • trait • Claim persistent storage • backup • trait • Data backup • restore • trait • Data restore • ingress • trait • Routing policy • monitoring • trait • Integrated internal monitoring policy All speak OAM API, easy to integrate with other OAM systems OAM to decouple application from runtime, easy to upgrade infrastructure Reusable components and traits to save engineering time • Before: in-house app definition, lack of portability and interoperability across clusters • After: OAM as core application model to manage stateful applications, e.g. Kafka, Zookeeper and Database on any cluster Share with other platforms
  20. OAM + Alibaba Serverless Platform • Before: in-house app definition,

    reinvent every wheel for serverless • After: OAM as unified definition of various workload types, and reuse traits from may other systems like EDAS. feature OAM Category Description • ContainerizedWorkload • component • Serverless container • Function • component • FaaS workload • Knative service (WIP) • component • Knative serving workload • traffic-mgmt • trait • Istio based Traffic shifting • cron-scaler • trait • Cron HPA • manual-scaler • trait • Manual scale • auto-scaler • trait • HPA • sls-project, sls-logstore, sls- index • component • Alibaba Cloud Resources as BaaS Separate concerns make developers accept serverless model way easier Migrate from K8s/PaaS to Serverless only need to update its trait Reusable components and traits to save engineering time Connect and consume cloud resources thru OAM based BaaS Share with other platforms
  21. Traits/Scopes Scale: - 10,000 nodes/cluster - 100,000 apps/cluster - 1,000,000

    containers/cluster Applications: - 100,000 deploys/day - 500~1000 replicas/app Function Job Container Virtual Machine Gateway Route Traffic Alert Monitor Service Binding Rollout Ingress interpretability Infra Ops Application Configuration Application Configuration Application Configuration PaaS A PaaS B Serverless C Components Traits Components Traits Components Traits Components Application Management At Alibaba Today Developers Operators
  22. Kubernetes in Alibaba with OAM Modular, reusable, standardized, and manageable

    platform capabilities, best engineering efficiency Clear separate of concerns in the workflow of app management, best communication and collaboration efficiency For Platform • Simple, team-centric and unified experience • Enjoy “unlimited” capabilities provided by k8s ecosystem - the “open” PaaS For end users (Dev and Ops) Minimal migration effort from PaaS/K8s to OAM platform The simplest and unified approach to integrate and consume Alibaba Cloud Resources