Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending Kubernetes with Custom Resources and ...

Ian Lewis
September 29, 2018

Extending Kubernetes with Custom Resources and Operator Frameworks

Kubernetes is a container orchestration system that provides a lot of built in functionality to deploy and manage applications using containers. But it’s applications that extend Kubernetes’ that is where the true power of Kubernetes lies. Supporting stateful applications, renewing certificates, managing distributed job pipelines can all be made easier by extending the Kubernetes API. Custom Resource Definitions (CRD) allow you to extend the API with new data types and Operators let you write automation logic based on your new data types.

Developing operators has traditionally been hard and new frameworks for writing operators like kubebuilder, operator-sdk, and Metacontroller, have made that job much easier, but it's hard to know which one to choose. In this session, attendees will learn about the Kubernetes API, it's architecture, and how to build operators. We will explore the key differences between the different operator frameworks so that attendees are empowered to choose the right framework for their use case.

Ian Lewis

September 29, 2018
Tweet

More Decks by Ian Lewis

Other Decks in Technology

Transcript

  1. Problems w/ Deploying Memcached • Memcached needs client side load

    balancing • Needs some kind of service discovery • Don't want to update application code • Want to support replication + sharding topologies
  2. Deploy memcached • Deploy a proxy using a Deployment •

    Configure proxy using a ConfigMap • When backends change create new ConfigMap and trigger a rolling-update for the proxy memcached memcached memcached client proxy
  3. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v1 memcached memcached memcached Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  4. Deploy memcached • Backend Pod endpoints change memcached memcached 10.0.2.17

    10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  5. Deploy memcached kind: ConfigMap apiVersion: v1 metadata: name: conf-v2 data:

    ... "10.0.2.17", "10.0.2.18" ... memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  6. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  7. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy proxy
  8. Deploy memcached apiVersion: extensions/v1beta1 kind: Deployment metadata: name: proxy spec:

    template: … spec: volumes: - name: proxy-conf configMap: name: conf-v2 memcached memcached 10.0.2.17 10.0.2.18 ConfigMap 10.0.2.17 10.0.2.18 Deployment ConfigMap 10.0.2.17 10.0.2.18 10.0.2.19 proxy
  9. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References
  10. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  11. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  12. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: ...
  13. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References Deployment ReplicaSet ReplicaSet OwnerRef ReplicaSet OwnerRef OwnerRef
  14. API Objects • API Version • Kind • Metadata ◦

    Name ◦ Labels ◦ Owner References apiVersion: v1 kind: ReplicaSet metadata: ... ownerReferences: - apiVersion: apps/v1 controller: true blockOwnerDeletion: true kind: Deployment name: nginx-deployment uid: d9607e19-f88f-11e6-a518-42010a800195 ...
  15. Built-in Clients kube- apiserver kubelet kube- proxy kube- proxy kube-

    controller- manager kube- scheduler kubelet kubelet • kubelet • kube-proxy • kube-controller-manager • kube-scheduler
  16. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  17. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: <null> ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  18. Google Cloud Platform controller-manager Pod metadata.name: nginx-xxxx-xxxx spec.nodeName: node1 ReplicaSet

    metadata.name: nginx-xxxx Deployment metadata.name: nginx Deployment Controller ReplicaSet Controller Scheduler
  19. Extending Kubernetes • Need a place to store state -

    Data • Need to do something - Logic
  20. Custom Resource Definition (Data) • Type definition for a custom

    type • Allows the same CRUD + WATCH • Can describe higher level constructs apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: foo.example.com spec: group: example.com version: v1 names: kind: Foo plural: foos scope: Namespaced
  21. Controllers (Logic) • Typically runs in the cluster • Uses

    the Kubernetes API • One idiomatic client ◦ client-go • Many generated client libraries ◦ Go ◦ Python ◦ Java
  22. client-go • Most featureful • Used by Kubernetes built-in components

    • More like a controller framework than a client library
  23. Operator Frameworks • Provide a simplified controller API • Rely

    on code generation to provide API clients for CRDs
  24. Frequent Requests • Controllers can potentially run often • Easy

    to overload the API server • Some GETS could be X00MB of data Object
  25. Concurrent Updates • Overwriting object state • The API Server

    isn't a database • No transactions Object
  26. operator-sdk • Built by former CoreOS developers at Red Hat

    • Quick and dirty • Provides support for one controller per process • Caching of watched objects w/ client-go • Serial updates per CRD object
  27. Reuse Built-in Objects • Services, Deployments, ConfigMaps, Secrets • Built

    on the logic of other controllers • Architect based on what you would do manually API
  28. Use Multiple Controllers • Multiple controllers per process • Reuse

    caches, informers etc. • Keep controllers simple • Each controller manages/writes to one object type • All message passing is done through the API API
  29. kubebuilder • Built by Kubernetes developers at Google • More

    robust • Helps manage lifecycle of generated code • Supports controllers for built-in objects • Supports multiple controllers in single operator binary