Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Programming Pattern

Lei (Harry) Zhang
August 31, 2017
170

Kubernetes Programming Pattern

CRD, Controller, GRPC, code gen, this is the core of Kubernetes custmization! This is my speak at GopherCon China.

Lei (Harry) Zhang

August 31, 2017
Tweet

Transcript

  1. Contents • What I will talk? • Kubernetes basic in

    1 min • For Kubernetes developers: • The Golang programming patterns of Kubernetes (Controller, codegen etc) • Write your own Controller • gPRC based interface design in Kubernetes (CRI as example) • For Kubernetes users: • Effective pattern of programming based on Kubernetes • ⼴广告(Don’t worry, it’s not that kind of AD) • What I will not talk? • Kubernetes usage and under the hood • Internal systems or commercial software
  2. Kubernetes • The container orchestration and management project created by

    Google • Successor of Google Borg/Omega system • One of the most popular open source projects in this world • Written by, and heavily depends on Golang
  3. Again: Why Go? • All about community • A sad

    story: • https://github.com/google/lmctfy • Now think about a C/C++ based Kubernetes? • And, well designed programming patterns with powerful extensibility
  4. Understand Kubernetes in 1 min Container Pod Deployment I need

    co-scheduling I have many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose my Services DaemonSet I run as daemon StatefulSet I am stateful Job I only run for once CronJob I run periodically ConfigMap I read configure file Secret I need confidential data HPA I need auto-scaling
  5. Understand Kubernetes in 2 min • kubectl run nginx —image=nginx:1.7.9

    —replicas=3 apiVersion: apps/v1beta1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 • kubectl create -f deployment.yaml • kubectl create -f hpa.yaml apiVersion: autoscaling/v2alpha1 kind: HorizontalPodAutoscaler metadata: name: php-apache namespace: default spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: nginx-deployment minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 • API Object Oriented Programming
  6. Core of API “OO” 1.API objects stores in etcd 2.Control

    loops (Sync Loop) to reconcile API objects
  7. Example kubelet SyncLoop kubelet SyncLoop proxy proxy 3.1 New Pod

    detected 3.2 Bind Pod to a node etcd scheduler api-server
  8. Example kubelet SyncLoop kubelet SyncLoop proxy proxy 4.1 Detected bind

    operation 4.2 Start Pod on this machine etcd scheduler api-server
  9. Pattern 1: Controller • Control everything by Controller • Level

    driven, not edge driven edge level Image: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
  10. Controller • The heart of Kubernetes orchestrator • drives the

    cluster state based on the changes to the API objects for { desired := getDesiredState() current := getCurrentState() makeChanges(desired, current) } • Write your own controller!
  11. Why DIY? Container Pod Deployment I need co-scheduling I have

    many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose my Services DaemonSet I run as daemon StatefulSet I am stateful Job I only run for once CronJob I run periodically ConfigMap I read configure file Secret I need confidential data HPA I need auto-scaling My Awesome Object I have my own special case
  12. Demo • I want to have a Asta Xie object

    into k8s API • I want a controller to handle add/update/delete of all Asta Xie instances $ kubectl get astaxie NAME KIND astaxie1 AstaXie.v1.cr.client-go.k8s.io
  13. My AstaXie Object & Controller etcd kube-apiserver types register AstaXie

    Controller astaxie1 OnDelete OnUpdate OnAdd Kubernetes Custom Controller User operation
  14. A Real World Example • I want to have a

    Network object into k8s API • I want a controller to handle add/update/delete of all Network instances • onAdd: create Neutron network • onDelete: delete Neutron network • onUpdate: update Network object status • https://github.com/openstack/stackube/blob/master/pkg/network- controller/network_controller.go
  15. Pattern 2: Gode Generator • client-gen: generate typed Kubernetes AP

    client for type • client.Pod.Get().Resource(…).Do() • conversion-gen: seamless upgrades between API versions • apiVersion: k8s.io/v1alpha1 -> apiVersion: k8s.io/v1beta1 • deepcopy-gen: deepcopy • go get k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/deepcopy-gen • deepcopy-gen -i ./pkg/apis/v1 • defaulter-gen: set default values for fields • go-to-protobuf: generate protobuf messages for your types • informer-gen: generate informers that can be used to watch for updates to your types • openapi-gen: generate openapi compatible API documentation
  16. Pattern 3: gRPC based Interface • Decouple Kubernetes from external

    dependencies • kubelet -> gRPC -> dockershim -> dockerd • go2idl: gogoprotobuf based protobuf gen
  17. CRI Management kubelet Workloads Orchestration kubelet SyncLoop Scheduling api-server Etcd

    bind pod, node list pod GenericRuntime SyncPod CRI grpc dockershim remote (no-op) Sandbox Create Delete List Container Create Start Exec Image Pull List shim client api dockerd runtime pod CRI Spec
  18. Deployment • yum install -y kubelet kubeadm kubectl • sed

    -i '2 i\Environment="KUBELET_EXTRA_ARGS=--container- runtime=remote --container-runtime-endpoint=/var/run/xxx.sock -- feature-gates=AllAlpha=true"' /etc/systemd/system/kubelet.service.d/10- kubeadm.conf • kubeadm init • kubeadm join --token $token ${master_ip:port} • Done!
  19. But that’s only one part … • Kubernetes is also

    about design pattern in container world • decoupling containers • re-use images • well-designed architecture for your container workloads • “How can I build distributed micro-services with container?”
  20. Programming Pattern • Sidecar apiVersion: v1 kind: Pod metadata: name:

    test-app spec: containers: - name: app-container image: gcr.io/google_containers/testapp:v1 volumeMounts: - name: varlog mountPath: /var/log - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: {} - name: config-volume configMap: name: fluentd-config
  21. Programming Pattern • InitContainer apiVersion: v1 kind: Pod metadata: name:

    init-demo spec: containers: - name: nginx image: nginx ports: - containerPort: 80 volumeMounts: - name: workdir mountPath: /usr/share/nginx/html # These containers are run during pod initialization initContainers: - name: install image: busybox command: - wget - "-O" - "/work-dir/index.html" - http://kubernetes.io volumeMounts: - name: workdir mountPath: "/work-dir" dnsPolicy: Default volumes: - name: workdir emptyDir: {}
  22. Programming Pattern • Initializer apiVersion: apps/v1beta1 kind: Deployment metadata: annotations:

    "initializer.kubernetes.io/logging-agent": "true" name: helloworld-with-annotation spec: replicas: 1 template: metadata: name: helloworld-with-annotation spec: containers: - name: helloworld image: gcr.io/hightowerlabs/helloworld:0.0.1 apiVersion: v1 kind: ConfigMap metadata: name: logging-agent-initializer data: config: | - name: logging-agent image: gcr.io/google_containers/fluentd:1.30 env: - name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log - name: config-volume mountPath: /etc/fluentd-config volumes: - name: varlog emptyDir: {} - name: config-volume configMap: name: fluentd-config Automatically Inject
  23. Summary 1.How Kubernetes is using Golang? 1.What is Kubernetes and

    how it works (1 mins) 2.The heart of Kubernetes orchestration: Controller 3.Write your own Controller with CRD 4.code gen for deep copy, API conversion, API doc, encoding/decoding etc 5.gRPC based interface (e.g. CRI) 2.How we can do better to use Kubernetes? 1.Programming Patterns in Kubernetes 1.this is the main difference of Kubernetes with others 2.think about why everyone loves Borg