Share
Embed
Bending Kubernetes to Your Needs
Talk at Cloud Matters London, see https://skillsmatter.com/meetups/11072-cloud-matters-july/
Michael Hausenblas
July 30, 2018
Transcript
35 slides
-
Bending Kubernetes to Your Needs An overview of customization options
in Kubernetes Michael Hausenblas @mhausenblas Developer Advocate, Red Hat 2018-07-30, Cloud Matters, London -
Hit me up on Twitter: @mhausenblas 2 • Developer Advocate
@ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas -
Hit me up on Twitter: @mhausenblas 3 admin SRE developer
infosec architect PM PHB -
Hit me up on Twitter: @mhausenblas 10 • in-tree (upstream)
via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle -
Hit me up on Twitter: @mhausenblas 11 • configuration files
and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API -
Hit me up on Twitter: @mhausenblas 12 Extension patterns example:
manage a CRD example: authn/authz example: network, storage, kubectl -
Hit me up on Twitter: @mhausenblas 13 Cloud providers github.com/kubernetes
• libraries (in-tree)/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I -
Hit me up on Twitter: @mhausenblas 14 kubelet: network/device/storage plugins
• Network—standard: CNI https://github.com/containernetworking/cni https://mhausenblas.info/cn-ref/ • Devices—GPUs, FPGAs, etc. https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/ • Storage—20+ in-tree, up-and-coming standard: CSI https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/ I -
Hit me up on Twitter: @mhausenblas 15 kubelet: container runtimes
• Container runtime—standard: CRI (since 1.5) https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/ • Nowadays multiple options: • containerd • Kata containers • gVisor • hyper.sh http://cri-o.io I -
Hit me up on Twitter: @mhausenblas 16 kubectl plugins •
Extend the set of commands https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I ~/.kube/plugins -
Hit me up on Twitter: @mhausenblas 18 Interlude 1: the
control plane -
Hit me up on Twitter: @mhausenblas 19 Interlude 2: the
life of an API server request Above is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1 persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks -
Hit me up on Twitter: @mhausenblas 20 Interlude 3: core
resources (in-tree) A https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ -
Hit me up on Twitter: @mhausenblas 22 Access extensions in
the API server • Admission controllers (in-tree, via configuration of the API server) https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha) A -
Hit me up on Twitter: @mhausenblas 23 Custom resources •
Extend “known” resources beyond core resources (pods, services, etc.) https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ https://blog.openshift.com/kubernetes-deep-dive-api-server-part-3a/ • Use API server to store custom resources in etcd for you • Use CLI to interact with custom resources in the usual way: kubectl get | create | delete … A -
Hit me up on Twitter: @mhausenblas 25 Custom controller •
Beyond the controller manager (which is in-tree!) • Custom controller • look after core resources https://github.com/kelseyhightower/secrets-controller • look after custom resources https://github.com/kubernetes/sample-controller A -
Hit me up on Twitter: @mhausenblas 26 Operators • operator
=~ custom resource + controller https://coreos.com/blog/introducing-operator-framework • Motivation: application lifecycle management • Use one of over 30 available operators or write your own using the framework https://github.com/operator-framework/awesome-operators A github.com/operator-framework -
Hit me up on Twitter: @mhausenblas 27 Extension API servers
• Full control but a lot of effort and responsibility https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • Typically more LOC than an controller/operator • Have to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog/ https://www.openservicebrokerapi.org/ A -
Hit me up on Twitter: @mhausenblas 28 Scheduler extensions •
A scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc. https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/ • You modify policies or run multiple schedulers (with pod opt-in) https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ https://embano1.github.io/post/sched-reconcile/ • You can use a webhook https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I -
Hit me up on Twitter: @mhausenblas 29 Other stuff you
can customize • Monitoring, alerting, logging • Secret management (encryption at rest) • Ingress https://kubernetes.io/docs/concepts/services-networking/ingress/ • DNS https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/ • kube-proxy https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ -
Hit me up on Twitter: @mhausenblas 31 • https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/ •
https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/ • https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • https://kubernetes.io/docs/reference/access-authn-authz/webhook/ • https://kubernetes.io/docs/setup/scratch/#cloud-provider • https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/ Kubernetes docs and blog posts -
Hit me up on Twitter: @mhausenblas 32 • Tim Hockin
& Michael Rubin—Kubernetes Distributions and ‘Kernels' https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You https://www.youtube.com/watch?v=PYLFZVv68lM Videos -
Hit me up on Twitter: @mhausenblas 33 • Tim Hockin—Kubernetes
Extensibility https://speakerdeck.com/thockin/kubernetes-extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101 https://kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van- slyck-microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools https://admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html • Toader Sebastian—A complete guide to Kubernetes Operator SDK https://banzaicloud.com/blog/operator-sdk/ • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos https://robszumski.com/building-an-operator/ Articles and slide decks -
Hit me up on Twitter: @mhausenblas 34 • https://github.com/kubernetes/kubectl/tree/master/pkg/pluginutils •
https://github.com/carolynvs/kubectl-flags-plugin • https://github.com/jordanwilson230/kubectl-plugins • https://github.com/kelseyhightower/denyenv-validating-admission-webhook • https://github.com/kubernetes-sigs/controller-tools • https://github.com/kubernetes-sigs/kubebuilder • https://metacontroller.app/ • https://github.com/yaronha/kube-crd • https://github.com/operator-framework/awesome-operators • https://github.com/operator-framework Repos and tools