Dockercon_2018_-_Kubernetes_Extensibility.pdf

Transcript

1. Eric Tune & Tim Hockin Google Kubernetes Extensibility
2. Kubernetes is a container management system
3. Kubernetes is a container management system platform
4. What is Kubernetes? ...an abstraction layer over infrastructure ...a framework for declarative APIs and distributed control Infrastructure Extensibility API Extensibility
5. Extensibility Goals Infrastructure Extensibility API Extensibility Support portability Support customization Autonomy Autonomy Scalable growth of project Encourage new uses
6. A major focus of the last 2 years of development From infrastructure to APIs, we have over a dozen extension points We have WAY more material than time! https://goo.gl/2qz8jW Kubernetes & Extensibility
7. Infrastructure Extensibility
8. Networks are like snowflakes There is no “one size fits all” for almost anything networking related We needed a way for users to customize how Kubernetes consumes networking infra Network Plugins
9. Old: built-in “plugins” (aka “send Tim a PR”) New: CNI - Container Network Interface • Started by CoreOS, now CNCF with community • “exec” interface with stdin/stdout/env API Widely used, also by other projects (e.g. Mesos) Underpins the default impl in Kubernetes Network Plugins (present)
10. Proposal open for a gRPC based API which covers more than just interfaces and IPAM Tighter coupling with Service API seems valuable Proposals open for multi-IP and multi-network Network Plugins (future)
11. Many storage technologies - physical and virtual, block and file • Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Many vendors want their products to support Kubernetes Storage Plugins
12. Old: built-in “plugins” (aka “send Tim a PR”) Old: Volume “flex” plugins via “exec” New: CSI - Container Storage Interface • Collaboration: Google, Mesosphere, Docker, Cloud Foundry • gRPC spec, with Kubernetes-specific adaptors • In development now, alpha in Kubernetes 1.10 Plan to transition most in-tree plugins to CSI Storage Plugins (present)
13. GPUs and other “accelerator” hardware is becoming very common Part of the larger resource model in Kubernetes gRPC based plugins Beta in Kubernetes 1.10 Device Plugins
14. Docker was baked-in, but people wanted to try new and interesting ideas • rkt, Containerd, CRI-O • Kata containers, Hyper.sh, gVisor Making it a plugin made the code better: win-win! CRI - gRPC based plugins Container Runtimes
15. • Stateful, daemon plugins • Upgradeable in-cluster plugins • Evolution: exec → RPC • Evolution: loose spec → tight • Containerized plugins FTW Lessons Learned gRPC Plugins Runtimes (CRI) Storage (CSI) Devices Key Management Networking (proposed)
16. Controllers observe diff act
17. Controllers THE fundamental design pattern in Kubernetes Examples: scheduler, kubelet, deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, ... Allows automation & extension of almost any existing API
18. resource resource resource Higher level of abstraction Lower level of abstraction
19. Kubernetes is designed to leverage clouds Built-in cloud-provider API (i.e. send me a PR) is hooked into many core control loops Now 8 implementations (and huge LOC count), so moving out-of-tree Cloud Providers
20. The API is a VIP (more or less) and virtual LB We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program $NETWORK Services
21. But Wait, There’s More! • Secret management (KMS) • HTTP load-balancing (Ingress) • NetworkPolicy • DNS • Scheduler extenders & whole schedulers • ...and that’s JUST the infrastructure (i.e. boring) parts
22. API Extensibility
23. • Add new types of resources to your cluster • Add custom policy hooks ◦ to custom and built-in APIs • "APIs that add and modify APIs" API Extensibility
24. • In Mac Edge, Windows Edge, and EE 2.0 • Supports API Extensions. • Certified Kubernetes • Docker Stacks uses API Extensions Kubernetes for Docker
25. Exploring Stacks Follow along at https://goo.gl/JT7v8Z
26. Exploring Stacks https://goo.gl/JT7v8Z $ cat docker-compose.yml version: "3.3" services: redis: image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend:
27. Exploring Stacks https://goo.gl/JT7v8Z $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running
28. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl config current-context docker-for-desktop
29. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s
30. Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s
31. compose resource service resource deployment resource Higher level of abstraction Lower level of abstraction
32. Kubernetes API Server Service API compose resource Deployment API service resource deployment resource
33. Kubernetes APIs Service API Deployment API service resource deployment resource something custom compose resource
34. Kubernetes APIs Service API Deployment API service resource deployment resource dockerd hypothetical /stacks compose resource
35. Kubernetes APIs Service API Deployment API service resource deployment resource dockerd hypothetical /stacks compose resource docker cli
36. Exploring Stacks API https://goo.gl/JT7v8Z # last time... $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running
37. Exploring Stacks API https://goo.gl/JT7v8Z № last time... $ docker stack deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running $ kubectl get stacks NAME AGE stackdemo 39s
38. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001
39. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 $
40. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 $ kubectl get stacks -s localhost:8001
41. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/name spaces/default/stacks localhost $ kubectl get stacks -s localhost:8001 NAME AGE stackdemo 1m
42. Kubernetes APIs Service API Deployment API service resource deployment resource kubectl compose.docker.com API compose resource
43. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get apiservices.apiregistration.k8s.io NAME AGE v1. 29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d
44. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name: v1beta2.compose.docker.com ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed
45. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d
46. Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d
47. Kubernetes APIs Service API API Registration API Kubernetes Cluster Deployment API Compose-API
48. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API
49. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API
50. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API
51. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose
52. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose redis docker CLI
53. Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes Cluster Deployment API Compose-API compose.docker.com API Compose redis redis redis redis
54. • Users ◦ Already have a client installed ◦ Already know how to find, trust it (TLS) and auth to the API • Controllers ◦ Can efficiently watch your resources • Admins ◦ Can separate your resources by Namespace ◦ Can authorize and audit log access to your resources Why Use an API Extension?
55. API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) API resource Controller
56. API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) API resource Controller
57. Extension API Server (EAS) API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
58. Extension API Server (EAS) API Aggregation & Extension API Servers (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
59. Extension API Server (EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller
60. API resource Controller Custom Resource Definitions
61. EAS Forked LoC: 0 Storage: provided Components: 1 Popularity: 100s Multiversioning: not yet Customizability: good CRD Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Customizability: better * http://github.com/sample-apiserver
62. Extension Ecosystem Devices 5 public plugins Storage 10 public plugins Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs
63. Extension Ecosystem • 4 Serverless frameworks • 6 PaaSes • 10 CI/CD systems • 14 different database controllers • 4 popular ML toolkits
64. Adding Types to the API • Extension API Servers • Custom Resource Definitions Adding Policy to the API • ValidatingAdmissionWebhooks • MutatingAdmissionWebhooks API Extensions
65. Admission: After authn/z but before storing the change. Affects mutations, not reads. Webhooks: The API Server calls your URL, synchronously Run in cluster via service or outside, e.g. serverless. Admission Webhooks
66. Old thinking: Better to make narrow specific interfaces, like ImagePolicyWebhook, for specific use cases. Can make easier to use. Overly general extensions may limit future optimization. Admission Webhooks
67. New thinking: Many custom resoures. Cluster owners need to write policy for core resources and for custom resources written by 3rd parties. Need to compose policies written by different parties. Admission Webhooks
68. Composability. Make all the changes before doing all the checks. MutatingAdmissionWebhooks - then- ValidatingAdmissionWebhooks Admission Webhooks
69. Kelsey Hightower: - reject pods that set environment variables https://github.com/kelseyhightower/denyenv-validating-admission-webhook CRD Authors : - add complex validation Validating Admission Webhooks
70. Istio: inject sidecar into all the pods Service Catalog: inject credentials into Mutating Admission Webhooks
71. - Mutate the pod template of a deployment - Install a flaky webhook matching all resources. Bad Ideas
72. •Kubernetes for Docker: • Super easy way to try Kubernetes •API Extensions: • Use them. Author them. On Docker. For Kubernetes. •Try it: • https://goo.gl/JT7v8Z Conclusion
73. v Questions? Learn more: https://goo.gl/JT7v8Z Thanks!