Belgium Kubernetes Meetup: Intro to CRDs and Controllers

CustomResourceDefinitions (CRDs) are used to extend a Kubernetes cluster with 3rd-party API resources, without changing the source code of Kubernetes. This opens up Kubernetes to be used as a platform for 3rd-party products or inhouse solutions.

Transcript

1. Intro to CustomResources & Controllers Stefan Schimanski sttts@redhat.com / sttts @ GitHub @the_sttts Kubernetes Meetup Belgium – June 15 Hosted by
3. Restful http API / /version /api /api/v1/pods /api/v1/pods/<name> /api/v1/pods/<name>/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs /apis/batch/v1 /apis/batch/v1/jobs kube-apiserver kubelet proxy Node: $ kubectl create -f foo.yaml User: scheduler controller manager apiserver ingress controller Master: cloud native apps Pods:
4. Restful http API / /version /api /api/v1/pods /api/v1/pods/<name> /api/v1/pods/<name>/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs /apis/batch/v1 /apis/batch/v1/jobs /apis/<our-group>/v1/<your-resource> kube-apiserver kubelet proxy Node: $ kubectl create -f foo.yaml User: scheduler controller manager apiserver ingress controller Master: cloud native apps Pods:
5. $ kube-apiserver --secure-port 0 --etcd-servers http://127.0.0.1:2379 --service-cluster-ip-range 10.0.0.0/16 --storage-backend etcd2 --storage-media-type application/json $ etcd $ kubectl config set-cluster local --server=http://127.0.0.1:8080 $ kubectl config set-context local --cluster=local $ kubectl config use-context local $ kubectl get namespaces --v=7 $ kubectl get namespace default -o json $ kubectl annotate namespace default meetup=hello $ curl http://127.0.0.1:8080/api/v1/namespaces/default $ etcdctl ls --recursive $ etcdctl get /registry/namespaces/default $ etcdctl -o extended get /registry/namespaces/default
6. $ http GET http://127.0.0.1:8080/api/v1/namespaces/default { "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "meetup": “hallo" }, "creationTimestamp": "2017-03-10T07:51:39Z", "name": "default", "resourceVersion": “73", }, "spec": { "finalizers": ["kubernetes“] }, "status": { "phase": "Active“ } } $ http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations[\"meetup\"] = \"$(date)\"" | http PUT http://127.0.0.1:8080/api/v1/namespaces/default HTTP/1.1 200 OK Content-Length: 341 Content-Type: application/json Date: Fri, 10 Mar 2017 08:28:01 GMT
7. /apis/batch/v2alpha1/jobs Apigroup Version Resource HTTP paths: { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “backup“ }, “spec“: { ... } } http POST /namespaces/<name>/
8. { "apiVersion": "v1", "kind": "Status", "metadata": {}, "code": 409, "message": “...", "status": "Failure“ } /apis/batch/v2alpha1/jobs { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “backup“ }, “spec“: { ... } } Resource vs. Kind http path vs. logical object * I omitted the namespace in /apis/batch/v1/jobs/namespaces/default/backup
9. Custom Resources (CRs) & Custom Resource Definitions (CRDs) apiextensions/v1beta1
10. Custom Resource eyjafjallajökull-volcano.yaml apiVersion: belgium.meetup.com/v1 kind: Volcano metadata: name: eyjafjallajökull spec: height: 1666 location: Suðurland, Iceland type: Stratovolcano status: lastEruption: 2010-03-26T15:13:42.05Z active: true We want to store these
11. apiextensions-apiserver inside kube-apiserver https://github.com/kubernetes/apiextensions-apiserver In every Kubernetes 1.7+ cluster. kube-apiserver kube-aggregator kube resources apiextensions- apiserver 404 etcd "delegation" "aggregation"
12. Custom Resource eyjafjallajökull-volcano.yaml apiVersion: iceland.meetup.com/v1 kind: Volcano metadata: name: eyjafjallajökull spec: height: 1666 location: Suðurland, Iceland type: Stratovolcano status: lastEruption: 2010-03-26T15:13:42.05Z active: true We want to store these
13. Custom Resource Definition (CRD) apiextensions/v1beta1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volcanos.iceland.meetup.com spec: group: iceland.meetup.com version: v1 names: kind: Volcano plural: volcanos scope: Namespaced must match Defines how CRs are stored
14. Custom Resource Definition (CRD) apiextensions/v1beta1 apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volcanos.iceland.meetup.com spec: group: iceland.meetup.com version: v1 names: kind: Volcano plural: volcanos scope: Namespaced $ kubectl create –f volcanos-crd.yaml a moment ... status: acceptedNames: kind: Volcano listKind: VolcanoList plural: volcanos singular: volcano conditions: - type: NamesAccepted message: no conflicts found reason: NoConflicts status: "True“ - type: Established message: the initial names have been accepted reason: InitialNamesAccepted status: "True“ must match
15. $ kubectl get volcanos –v=7 • I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis • I0429 21:17:53.135811 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1 • I0429 21:17:53.138353 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1/namespaces/default/volcanos No resources found. volcanos → kind Volcano resource volcanos discovery LIST API group iceland.meetup.com/v1 We call this "REST mapping"
16. $ http localhost:8080/apis/ { "groups": [{ "name": "iceland.meetup.com", "preferredVersion": {"groupVersion": "iceland.meetup.com/v1", "version": "v1“}, "versions": [{"groupVersion": "iceland.meetup.com/v1", "version": "v1"}] }, ...] } $ http localhost:8080/apis/iceland.meetup.com/v1 { "apiVersion": "v1", "groupVersion": "iceland.meetup.com/v1", "kind": "APIResourceList", "resources": [{ "kind": "Volcano", "name": "volcanos", "namespaced": true, "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch“ ] }, ...] } resource name ⇒ /apis/iceland.meetup.com/v1/volcanos /namespaces/<name>/
17. $ kubectl get volcanos –v=7 • I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis • I0429 21:17:53.135811 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1 • I0429 21:17:53.138353 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1/namespaces/default/volcanos No resources found. volcanos → kind Volcano resource volcanos discovery LIST API group iceland.meetup.com/v1 We call this "REST mapping"
18. $ kubectl get jökla –v=7 • I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis • I0429 21:17:53.135811 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1 • I0429 21:17:53.138353 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1/namespaces/default/volcanos No resources found. jökla → kind Volcano resource volcanos discovery LIST API group iceland.meetup.com/v1 We call this "REST mapping" note: a "shortName"
19. $ kubectl get jokla –v=7 • I0429 21:17:53.042783 66743 round_trippers.go:383] GET https://localhost:6443/apis • I0429 21:17:53.135811 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1 • I0429 21:17:53.138353 66743 round_trippers.go:383] GET https://localhost:6443/apis/iceland.meetup.com/v1/namespaces/default/volcanos No resources found. jokla → kind Volcano resource volcanos discovery LIST API group iceland.meetup.com/v1 We call this "REST mapping" note: a "shortName"
20. $ kubectl create –f volcanos-crd.yaml $ kubectl create –f eyjafjallajökull-volcano.yaml apiVersion: iceland.meetup.com/v1 kind: Volcano metadata: name: eyjafjallajökull spec: height: 1666 location: Suðurland, Iceland type: Stratovolcano status: lastEruption: 2010-03-26T15:13:42.05Z apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volcanos.iceland.meetup.com spec: group: iceland.meetup.com version: v1 names: kind: Volcano plural: volcanos scope: Namespaced 1 2 Create the CRD Create CustomResources
21. $ kubectl create –f volcanos-crd.yaml $ kubectl create –f eyjafjallajökull-volcano.yaml apiVersion: iceland.meetup.com/v1 kind: Volcano metadata: name: eyjafjallajökull spec: height: 1666 location: Suðurland, Iceland type: Stratovolcano status: lastEruption: 2010-03-26T15:13:42.05Z apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volcanos.iceland.meetup.com spec: group: iceland.meetup.com version: v1 names: kind: Volcano plural: volcanos scope: Namespaced 1 2 Create the CRD Create CustomResources Follow spec+status pattern.
22. $ kubectl get volcanos –w --no-headers katla <none> {"apiVersion":"iceland.meetup.com/v1","kind":"Volcanos",... eyjafallajökull <none> {"apiVersion":"iceland.meetup.com/v1","kind":" Volcanos ",... $ curl -f 'http://127.0.0.1:8080/apis/iceland.meetup.com/v1/namespaces/default/volcanos?watch=true&resourceVersion=434‘ {"type":"DELETED","object":{"apiVersion":"iceland.meetup.com/v1","kind":"Volcanos","metadata":{"name":"katla","names pace":"default","selfLink":"/apis/iceland.meetup.com/v1/namespaces/default/volcanos/katla","uid":"8f5312c0-29c8-11e7- 88f9-4c3275978b79","resourceVersion":"435","creationTimestamp":"2017-04-25T15:05:03Z"},"spec":{...}}} {"type":"ADDED","object":{"apiVersion":"iceland.meetup.com/v1","kind":"Volcanos","metadata":{"name":"eyjafallajökull"," namespace":"default","selfLink":"/apis/iceland.meetup.com/v1/namespaces/default/volcanos/eyjafallajökull","uid":"b4318 cb5-29c8-11e7-88f9-4c3275978b79","resourceVersion":"436","creationTimestamp":"2017-04-25T15:06:05Z"},"spec":{...}}} Watch, i.e. event stream
23. Validation • The standard: OpenAPI v3 schema • based on JSON Schema apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volcanos.iceland.meetup.com spec: group: iceland.meetup.com version: v1 names: ... validation: openAPIV3Schema: <see next slide>
24. spec: type: Stratovolcano height: 1666 location: Suðurland, Iceland status: lastEruption: 2010-03-26T... active: true properties: spec: properties: type: anyOf: [{"pattern": "^Stratovolcano$"}, …] height: {"type": "integer", "minimum": 0} location: {"type": "string", "default": "Iceland"} required: ["type", "height", "location"] status: properties: active: {"type": "bool"}, lastEruption: {"type": "string", "pattern": "^[0-9]+-...$"}, required: [] OpenAPI v3 Schema a quantor (anyOf, oneOf, allOf exist) note: enum is forbidden (why?) regular expression probably in 1.12+ Custom Resource Helpful tools: kubernetes/kube-openapi#37 tamalsaha/kube-openapi-generator Some other tool from prometheus-operator? Rancher has another one, speak to @lemonjet
25. Zoom into apiextensions-apiserver kube-apiserver kube- aggregator kube resources apiextensions-apiserver 404 etcd "delegation" "aggregation" authn/z CR handlers CR handlers CR handlers ⟲Naming Controller ⟲CRD Finalizer request conversion & defaulting storage conversion & defaulting REST logic result conversion validation admission decoding encode GET CREATE LIST UPDATE DELETE WATCH mutating webhooks validating webhooks NoOps json.Unmarshal
26. "CRDs are limited" • no version conversion (only one version possible per CRD) • in 1.11 multiple versions, but no conversion • in 1.12+ conversions come, but slowly • no defaulting (PR exists) • no validation (beta in 1.9, Google Summer of Code project) • no subresources (scale, status) (beta in 1.11) • no admission (since 1.7: admission webhooks + initializers) • alpha CRDs are beta • no custom printing in kubectl (in 1.11: custom printer columns) • demand is high • many users: 300 people listening to CRD talks @ KubeCon EU 2018
27. Controllers where is the business logic?
28. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations[\"meetup\"] = \"$(date)\"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done ⟲ 1. read object (preferably event driven with watches) 2. change object, update the world 3. update object on apiserver 4. repeat
29. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | {jq ".metadata.annotations[\"meetup\"] = \"$RANDOM\""; sleep 1;} | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done HTTP/1.1 409 Conflict Content-Length: 310 Content-Type: application/json Date: Fri, 10 Mar 2017 08:27:58 GMT { "apiVersion": "v1", "code": 409, "details": { "kind": "namespaces", "name": "default“ }, "kind": "Status", "message": "Operation cannot be fulfilled on namespaces \"default\": the object has been modified; please apply your changes to the latest version and try again", "metadata": {}, "reason": "Conflict", "status": "Failure“ } „optimistic concurrency“
30. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations[\"meetup\"] = \"$(date)\"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done ⟲ 1. read object (preferably event driven with watches) 2. change object, update the world 3. update object on apiserver 4. repeat expect version conflicts, remember: optimistic concurrency
31. $ kubectl get namespaces --watch --no-headers | while read NS STATUS TIME ; do # do whatever you like here, e.g. change the namespace echo "$NS changed“ done ⟲ W atch = nohot looping $ curl -f 'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘ {"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
32. Outlook – Custom Resources • Kubernetes 1.11+ • ⍺: Multiple versions without conversion – design proposal • ⍺: Server Side Printing Columns – “kubectl get” customization – #60991 • β: Subresources – ⍺ since 1.10 – #62786 • OpenAPI additionalProperties allowed now (mutually exclusive with properties) • Kubernetes 1.12+ • Multiple versions with declarative field renames • ⍺: Graceful Deletion – being discussed – #63162 • ⍺: Pruning – in validation spec unspecified fields are removed – blocker for GA • ⍺: Defaulting – defaults from OpenAPI validation schema are applied • Strict create mode? Discuss: #5889 – my favorite CRD UX issue Related: CRD OpenAPI validation spec not served by kube-apiserver