Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Make Kubernetes Controller Development H...

Yuki Ito
October 21, 2021

How to Make Kubernetes Controller Development Happier: mercari.go #17

Yuki Ito

October 21, 2021
Tweet

More Decks by Yuki Ito

Other Decks in Technology

Transcript

  1. What is Kubernetes Kubernetes is a portable, extensible, open-source platform

    for managing containerized workloads and services, that facilitates both declarative con fi guration and automation. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  2. Container Orchestration kubelet containerd runc Container Node runc Container Container

    Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
  3. Container Orchestration kubelet containerd runc Container Node runc Container Container

    Runtime Interface (CRI) runc Container Open Container Initiative (OCI) speci fi cation
  4. Create Container (Pod) -- - apiVersion: v 1 kind: Po

    d metadata : name: envo y spec : containers : - name: envo y image: envoy:lates t command : - '...'
  5. Controller Pattern A controller tracks at least one Kubernetes resource

    type. These objects have a spec fi eld that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
  6. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
  7. e.g. Built-in Deployment Controller -- - apiVersion: apps/v 1 kind:

    Deploymen t spec : replicas: 3 template : spec : containers : - image: envoy:latest
  8. Custom Resource De fi nition (CRD) ɾPod ɾReplicaSet ɾDeployment ɾService

    ɾIngress ɾHorizontalPodAutoscaler ɾPodDisruptionBudget ... e.g. Istio ɾVirtualService ... e.g. spanner-autoscaler ɾSpannerAutoscaler e.g. CrossPlane ɾCloudSQLInstance ... Native Resources Custom Resources + Extends Kubernetes using CRD
  9. e.g. spanner-autoscaler https://github.com/mercari/spanner-autoscaler apiVersion: spanner.mercari.com/v1alpha 1 kind: SpannerAutoscale r metadata

    : name: spannerautoscale r namespace: xx x spec : scaleTargetRef : projectId: #... instanceId: #... minNodes: 5 maxNodes: 1 0 maxScaleDownNodes: 1 targetCPUUtilization : highPriority: 60 SpannerAutoscaler
  10. kubernetes-sigs / kubebuilder > kubebuilder init --domain tutorial.kubebuilder.io > tree

    . . ├── confi g │ ├── defaul t │ │ ├── kustomization.yam l │ │ ├── manager_auth_proxy_patch.yam l │ │ └── manager_config_patch.yam l │ ├── manage r │ │ ├── controller_manager_config.yam l │ │ ├── kustomization.yam l │ │ └── manager.yam l │ ├── prometheu s │ │ ├── kustomization.yam l │ │ └── monitor.yam l │ └── rba c │ ├── auth_proxy_client_clusterrole.yam l │ ├── auth_proxy_role_binding.yam l │ ├── auth_proxy_role.yam l │ ├── auth_proxy_service.yam l │ ├── kustomization.yam l │ ├── leader_election_role_binding.yam l │ ├── leader_election_role.yam l │ ├── role_binding.yam l │ └── service_account.yam l ├── Dockerfil e ├── go.mo d ├── go.su m ├── hac k │ └── boilerplate.go.tx t ├── main.g o ├── Makefil e └── PROJECT
  11. Custom Resource De fi nition -- - kind: CustomResourceDefinitio n

    spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML)
  12. Code Generation -- - kind: CustomResourceDefinitio n spec : names

    : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
  13. DeepCopyObject client-go requires implementing DeepCopyObject func (in *Repository) DeepCopyObject() runtime.Object

    { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Object interface { GetObjectKind() schema.ObjectKin d DeepCopyObject() Objec t }
  14. kubernetes-sigs / controller-tools (controller-gen) -- - kind: CustomResourceDefinitio n spec

    : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) Go func (in *Repository) DeepCopyObject() runtime.Object { if c := in.DeepCopy(); c != nil { return c } return nil } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } type Repository struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec RepositorySpe c } type RepositorySpec struct { Repository string `json:"repository,omitempty"` }
  15. Code Generation ɾDe fi nition should be language agnostic ɾBut,

    do not want to write complex CRD YAML... controller-gen is useful, however...
  16. Istio de fi nes CRDs as Protocol Bu ff ers

    message VirtualService { repeated string hosts = 1 ; repeated string gateways = 2 ; repeated HTTPRoute http = 3 ; // ... } Protocol Bu ff ers https://github.com/istio/api/blob/1.11.4/networking/v1beta1/virtual_service.proto
  17. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  18. bufbuild / buf -- - version: v1beta 1 plugins :

    - name: g o path: ./bin/protoc-gen-g o out: . opt: paths=source_relativ e - name: deepcop y path: ./bin/protoc-gen-deepcop y out: . opt: paths=source_relative buf.gen.yaml
  19. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  20. De fi ne CRDs as Protocol Bu ff ers message

    Repository { string owner = 1 ; string repository = 2 ; } type Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers func (in *Repository) DeepCopyInto(out *Repository) { p := proto.Clone(in).(*Repository ) *out = * p } func (in *Repository) DeepCopy() *Repository { if in == nil { return nil } out := new(Repository ) in.DeepCopyInto(out ) return ou t } protoc-gen-go protoc-gen-deepcopy
  21. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
  22. De fi ne CRDs as Protocol Bu ff ers type

    Repository struct { state protoimpl.MessageStat e sizeCache protoimpl.SizeCach e unknownFields protoimpl.UnknownField s Owner string `protobuf:...` Repository string `protobuf:...` } Protocol Bu ff ers protoc-gen-go type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen
  23. De fi ne CRDs as Protocol Bu ff ers --

    - kind: CustomResourceDefinitio n spec : names : kind: Repositor y listKind: RepositoryLis t plural: repositorie s singular: repositor y scope: Namespace d versions : - name: v1alpha 1 schema : openAPIV3Schema : properties : spec : properties : repository : type: string CRD (YAML) type Repository struct { v1.TypeMet a v1.ObjectMet a // Proto Buffer Struc t Spec apisv1alpha1.Repositor y } kubetype-gen controller-gen
  24. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers protoc-gen-go / protoc-gen-deepcopy Protocol Bu ff er Go Struct (+ DeepCopy) Kubernetes Go Types Custom Resource De fi nitions (YAML) kubetype-gen controller-gen
  25. De fi ne CRDs as Protocol Bu ff ers Protocol

    Bu ff ers Advanced Custom Resource De fi nitions (YAML) istio /tools / cue-gen
  26. setup-envtest > setup-envtest use Version: 1.22. 1 OS/Arch: linux/amd6 4

    Path: /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 > ls /path/to/kubebuilder-envtest/k8s/1.22.1-linux-amd6 4 etcd kube-apiserver kubectl https://github.com/kubernetes-sigs/controller-runtime/tree/master/tools/setup-envtest
  27. Unit Testing with envtest et := envtest.Environment { BinaryAssetsDirectory: "/path/to/envtest-binaries"

    , CRDDirectoryPaths: []string{"/path/to/crd-yamls"} , ControlPlaneStartTimeout: 20 * time.Second , ErrorIfCRDPathMissing: true , AttachControlPlaneOutput: false , KubeAPIServerFlags: []string { "--advertise-address=127.0.0.1" , // ... } , } cfg, err := et.Start( ) if err != nil { return nil, nil, fmt.Errorf("failed to start envtest: %w", err ) } cli, err := client.New(cfg, client.Options { // ... })
  28. Unit Testing with envtest Caching envtest instance with TestMain func

    TestMain(m *testing.M) { os.Exit(func() int { cli, done, err := testutil.TestK8SClient() if err != nil { fmt.Fprintf(os.Stdout, "failed to create k8s client: %s\n", err ) return 1 } defer done( ) k8sClient = cl i return m.Run( ) }() ) }
  29. Unit Testing with envtest Isolating each test by creating dedicated

    Namespaces func TestXxx(t *testing.T) { // ... namespace := NewNamespace(t, ctx, k8sClient ) } func NewNamespace(t *testing.T, ctx context.Context, cli client.Client) string { t.Helper( ) name := uuid.New().String( ) ns := &corev1.Namespace { ObjectMeta: metav1.ObjectMeta { Name: name , } , } if err := cli.Create(ctx, ns); err != nil { t.Fatalf("failed to create namespace: %s", err ) } return nam e }
  30. Ska ff old + kind # ... spec : containers

    : - args : - /usr/bin/my-controller Pod Spec
  31. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)
  32. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  33. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dl v - exe c - --headles s - --listen=:5626 8 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # ... initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  34. ska ff old debug # ... spec : containers :

    - args : - /dbg/go/bin/dlv - exe c - --headles s - --listen=:56268 - .. . - /usr/bin/my-controlle r # ... volumeMounts : - mountPath: /db g # .. . initContainers : - image: gcr.io/gcp-dev-tools/duct-tape/g o # ... Pod Spec
  35. ska ff old debug 👨💻 kind cluster ska ff old

    debug Controller (Image) + Debugger (Delve)