Extending Kubernetes 101

Talk at ContainerConf 2018, see also https://www.containerconf.de/veranstaltung-7566-extending-kubernetes-101.html and https://github.com/mhausenblas/operator-101

Transcript

1. Extending Kubernetes 101
 Michael Hausenblas @mhausenblas
 Developer Advocate, Red Hat
 2018-11-15, ContainerConf, Mannheim
2. Hit me up on Twitter: @mhausenblas 2 • Developer Advocate @ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas
3. Hit me up on Twitter: @mhausenblas 3 admin SRE developer infosec architect PM PHB
4. Kubernetes 101
5. Hit me up on Twitter: @mhausenblas 5
6. Hit me up on Twitter: @mhausenblas 6 Kubernetes kubernetes.io • Container lifecycle management • Declarative API + control loops • Robust, flexible, scalable • Extensible
7. Hit me up on Twitter: @mhausenblas 7 • infrastructure admin • namespace admin • developer Roles and responsibilities
8. How can I customize Kubernetes?
9. Hit me up on Twitter: @mhausenblas 9 • in-tree (upstream) via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle
10. Hit me up on Twitter: @mhausenblas 10 • configuration files and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API
11. Hit me up on Twitter: @mhausenblas 11 Extension patterns Example: manage a custom resource Example: authn/authz Example: network, storage, kubectl
12. Hit me up on Twitter: @mhausenblas 12 Cloud providers github.com/kubernetes • in-tree libraries/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I
13. Hit me up on Twitter: @mhausenblas 13 kubelet: network/device/storage plugins • Network—standard: CNI
 github.com/containernetworking/cni 
 kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins • Devices—GPUs, FPGAs, etc.
 kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins • Storage—20+ in-tree, up-and-coming standard: CSI
 kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes 
 kubernetes.io/blog/2018/04/10/container-storage-interface-beta I
14. Hit me up on Twitter: @mhausenblas 14 kubelet: container runtimes • Container runtime—standard: CRI (since Kubernetes 1.5)
 kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes • Nowadays multiple options: • runc • containerd • Kata containers • gVisor • hyper.sh cri-o.io I
15. Hit me up on Twitter: @mhausenblas 15 kubectl plugins • Extend the set of commands
 kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I
16. as simple plugin in action: kubectl inspect
17. Extending the Kubernetes API
18. Hit me up on Twitter: @mhausenblas 18 Quick control plane refresher
19. Hit me up on Twitter: @mhausenblas 19 The life of an API request Flow diagram is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1. persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks
20. Hit me up on Twitter: @mhausenblas 20 What are (in-tree) core resources? A kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/
21. Hit me up on Twitter: @mhausenblas 21 Access extensions in the API server • Admission controllers (in-tree, via configuration of the API server)
 https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control
 https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha)
 A
22. Hit me up on Twitter: @mhausenblas 22 Custom resources • Support for “known” resources beyond core resources
 kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources
 blog.openshift.com/kubernetes-deep-dive-api-server-part-3a • Use the API server to manage custom resources in etcd for you • Custom resource definition (CRD) and instances • Use the CLI to interact with custom resources in the usual way,
 for example: kubectl get mycustomresource A
23. Hit me up on Twitter: @mhausenblas 23 Custom resource—example A
24. Hit me up on Twitter: @mhausenblas 24 Custom controller • Implement control loops beyond what thee (in-tree)
 controller manager supports • Custom controller • dealing with core resources
 github.com/kelseyhightower/secrets-controller • dealing with custom resources (aka operator)
 github.com/kubernetes/sample-controller A
25. Hit me up on Twitter: @mhausenblas 25 Custom resources and controllers A resource controller core custom in-tree custom Kubernetes control plane operator simple controller X X X X X X
26. Operators
27. Hit me up on Twitter: @mhausenblas 27 Operators operator = custom resource + custom controller • Motivation: application lifecycle management • Use one of 30+ available operators or write your own with: • Kubebuilder • Kubernetes Operator Kit • kutil • Metacontroller • Operator SDK A
28. Hit me up on Twitter: @mhausenblas 28 Operator use cases • zero-downtime upgrades of the app the operator supervises • workflow automations • policy enforcement • managing stateful workloads • resizing of followers in a distributed datastore • backup & restore of a database • re-balancing of a distributed message queue A
29. Hit me up on Twitter: @mhausenblas 29 Operator examples • etcd • Prometheus • Postgres • Vitess MySQL • MongoDB • Couchbase • Kafka A
30. a simple operator in action: NoDefaultsPolicy github.com/mhausenblas/operator-101
31. $ operator-sdk new nodefpol-operator
32. $ operator-sdk add api --api-version=nodefpol.k8space.io/v1alpha1 --kind=NoDefaultsPolicy
33. $ operator-sdk add controller --api-version=nodefpol.k8space.io/v1alpha1 --kind=NoDefaultsPolicy
34. $ kubectl -n ndp-demo apply -f deploy/crds/nodefpol_v1alpha1_nodefaultspolicy_crd.yaml $ OPERATOR_NAME=nodefpol-operator operator-sdk up local --namespace "ndp-demo"
35. grep ‘//TODO(user)’
36. Hit me up on Twitter: @mhausenblas 36 Extension API servers • Full control but a lot of effort and responsibility
 kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server • Typically more LOC than an controller or operator • You might end up to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog
 kubernetes.io/docs/concepts/extend-kubernetes/service-catalog
 openservicebrokerapi.org A
37. Hit me up on Twitter: @mhausenblas 37 Scheduler extensions A scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc.
 jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work • You can modify policies or run multiple schedulers (with pod opt-in)
 kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers
 embano1.github.io/post/sched-reconcile • You can use a Webhook 
 github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I
38. Hit me up on Twitter: @mhausenblas 38 Other stuff you can customize in Kubernetes • Monitoring & alerting (Prometheus/Grafana), logging (ELK/EFK stack) • Secret management (encryption at rest, Vault) • Ingress
 kubernetes.io/docs/concepts/services-networking/ingress • DNS
 kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers • kube-proxy
 kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive
39. Resources
40. Hit me up on Twitter: @mhausenblas 40
41. Hit me up on Twitter: @mhausenblas 41 • Tim Hockin—Kubernetes Extensibility
 speakerdeck.com/thockin/kubernetes-extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101
 kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van-slyck- microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools
 admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html • Toader Sebastian—A complete guide to Kubernetes Operator SDK
 banzaicloud.com/blog/operator-sdk/ • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos
 robszumski.com/building-an-operator/ Articles and slide decks
42. Hit me up on Twitter: @mhausenblas 42 • github.com/kubernetes/kubectl/tree/master/pkg/pluginutils • github.com/carolynvs/kubectl-flags-plugin • github.com/jordanwilson230/kubectl-plugins • github.com/kelseyhightower/denyenv-validating-admission-webhook • github.com/kubernetes-sigs/controller-tools • github.com/kubernetes-sigs/kubebuilder • metacontroller.app • github.com/yaronha/kube-crd • github.com/operator-framework/operator-sdk • github.com/operator-framework/awesome-operators • reactiveops.github.io/rbac-manager Repos, examples, tooling
43. Hit me up on Twitter: @mhausenblas 43 • kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/ • kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ • kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/ • kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • kubernetes.io/docs/reference/access-authn-authz/webhook/ • kubernetes.io/docs/setup/scratch/#cloud-provider • kubernetes.io/blog/2018/01/extensible-admission-is-beta/ Kubernetes docs and blog posts
44. Hit me up on Twitter: @mhausenblas 44 • Tim Hockin & Michael Rubin—Kubernetes Distributions and ‘Kernels'
 https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup
 https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive
 https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You
 https://www.youtube.com/watch?v=PYLFZVv68lM 
 Videos
45. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews learn.openshift.com