Share
Embed
Transcript
73 slides
-
What is Kubernetes? ...an abstraction layer over infrastructure ...a framework
for declarative APIs and distributed control Infrastructure Extensibility API Extensibility -
Extensibility Goals Infrastructure Extensibility API Extensibility Support portability Support customization
Autonomy Autonomy Scalable growth of project Encourage new uses -
A major focus of the last 2 years of development
From infrastructure to APIs, we have over a dozen extension points We have WAY more material than time! https://goo.gl/2qz8jW Kubernetes & Extensibility -
Networks are like snowflakes There is no “one size fits
all” for almost anything networking related We needed a way for users to customize how Kubernetes consumes networking infra Network Plugins -
Old: built-in “plugins” (aka “send Tim a PR”) New: CNI
- Container Network Interface • Started by CoreOS, now CNCF with community • “exec” interface with stdin/stdout/env API Widely used, also by other projects (e.g. Mesos) Underpins the default impl in Kubernetes Network Plugins (present) -
Proposal open for a gRPC based API which covers more
than just interfaces and IPAM Tighter coupling with Service API seems valuable Proposals open for multi-IP and multi-network Network Plugins (future) -
Many storage technologies - physical and virtual, block and file
• Cloud block devices, FC, iSCSI, NFS, Ceph, Gluster, ... Many vendors want their products to support Kubernetes Storage Plugins -
Old: built-in “plugins” (aka “send Tim a PR”) Old: Volume
“flex” plugins via “exec” New: CSI - Container Storage Interface • Collaboration: Google, Mesosphere, Docker, Cloud Foundry • gRPC spec, with Kubernetes-specific adaptors • In development now, alpha in Kubernetes 1.10 Plan to transition most in-tree plugins to CSI Storage Plugins (present) -
GPUs and other “accelerator” hardware is becoming very common Part
of the larger resource model in Kubernetes gRPC based plugins Beta in Kubernetes 1.10 Device Plugins -
Docker was baked-in, but people wanted to try new and
interesting ideas • rkt, Containerd, CRI-O • Kata containers, Hyper.sh, gVisor Making it a plugin made the code better: win-win! CRI - gRPC based plugins Container Runtimes -
• Stateful, daemon plugins • Upgradeable in-cluster plugins • Evolution:
exec → RPC • Evolution: loose spec → tight • Containerized plugins FTW Lessons Learned gRPC Plugins Runtimes (CRI) Storage (CSI) Devices Key Management Networking (proposed) -
Controllers THE fundamental design pattern in Kubernetes Examples: scheduler, kubelet,
deployments, kube-proxy, cloud providers, load balancers, volume provisioners, auto-scalers, ... Allows automation & extension of almost any existing API -
Kubernetes is designed to leverage clouds Built-in cloud-provider API (i.e.
send me a PR) is hooked into many core control loops Now 8 implementations (and huge LOC count), so moving out-of-tree Cloud Providers -
The API is a VIP (more or less) and virtual
LB We ship a default implementation (kube-proxy), but that can be replaced Controller: watch the API server for Services and Endpoints, program $NETWORK Services -
But Wait, There’s More! • Secret management (KMS) • HTTP
load-balancing (Ingress) • NetworkPolicy • DNS • Scheduler extenders & whole schedulers • ...and that’s JUST the infrastructure (i.e. boring) parts -
• Add new types of resources to your cluster •
Add custom policy hooks ◦ to custom and built-in APIs • "APIs that add and modify APIs" API Extensibility -
• In Mac Edge, Windows Edge, and EE 2.0 •
Supports API Extensions. • Certified Kubernetes • Docker Stacks uses API Extensions Kubernetes for Docker -
Exploring Stacks https://goo.gl/JT7v8Z $ cat docker-compose.yml version: "3.3" services: redis:
image: redis:alpine ports: - 6379 networks: - frontend deploy: replicas: 1 networks: frontend: -
Exploring Stacks https://goo.gl/JT7v8Z $ docker stack deploy --compose-file docker-compose.yml stackdemo
Waiting for the stack to be stable and running... - Service redis has one container running Stack stackdemo is stable and running -
Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s -
Exploring Stacks https://goo.gl/JT7v8Z $ kubectl get services NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 29d redis ClusterIP None <none> 55555/TCP 1s redis-random-ports NodePort 10.101.242.155 <none> 6379:31248/TCP 1s $ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE redis 1 1 1 1 2s -
compose resource service resource deployment resource Higher level of abstraction
Lower level of abstraction -
Kubernetes API Server Service API compose resource Deployment API service
resource deployment resource -
Kubernetes APIs Service API Deployment API service resource deployment resource
something custom compose resource -
Kubernetes APIs Service API Deployment API service resource deployment resource
dockerd hypothetical /stacks compose resource -
Kubernetes APIs Service API Deployment API service resource deployment resource
dockerd hypothetical /stacks compose resource docker cli -
Exploring Stacks API https://goo.gl/JT7v8Z # last time... $ docker stack
deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running -
Exploring Stacks API https://goo.gl/JT7v8Z № last time... $ docker stack
deploy --compose-file docker-compose.yml stackdemo Waiting for the stack to be stable and running... - Service web has one container running - Service redis has one container running Stack stackdemo is stable and running $ kubectl get stacks NAME AGE stackdemo 39s -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting
to serve on 127.0.0.1:8001 -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting
to serve on 127.0.0.1:8001 $ -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting
to serve on 127.0.0.1:8001 $ kubectl get stacks -s localhost:8001 -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl proxy -v 5 Starting
to serve on 127.0.0.1:8001 I0613 10:13:27.322416 82905 proxy_server.go:138] Filter accepting GET /apis/compose.docker.com/v1beta2/name spaces/default/stacks localhost $ kubectl get stacks -s localhost:8001 NAME AGE stackdemo 1m -
Kubernetes APIs Service API Deployment API service resource deployment resource
kubectl compose.docker.com API compose resource -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get apiservices.apiregistration.k8s.io NAME AGE
v1. 29d v1.apps 29d ... v1beta2.compose.docker.com 29d v2beta1.autoscaling 29d -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl describe apiservices.apiregistration.k8s.io v1beta2.compose.docker.com Name:
v1beta2.compose.docker.com ... API Version: apiregistration.k8s.io/v1beta1 Kind: APIService Metadata: ... Spec: ... Service: Name: compose-api Namespace: docker Status: Conditions: Message: all checks passed -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d -
Exploring Stacks API https://goo.gl/JT7v8Z $ kubectl get services -n docker
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE compose-api ClusterIP 10.110.211.86 <none> 443/TCP 17d $ kubectl get deployments -n docker NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE compose 1 1 1 1 29d compose-api 1 1 1 1 29d -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API compose.docker.com API -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API compose.docker.com API -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API compose.docker.com API Compose -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API compose.docker.com API Compose redis docker CLI -
Kubernetes APIs Service API API Registration API Compose. docker.com Kubernetes
Cluster Deployment API Compose-API compose.docker.com API Compose redis redis redis redis -
• Users ◦ Already have a client installed ◦ Already
know how to find, trust it (TLS) and auth to the API • Controllers ◦ Can efficiently watch your resources • Admins ◦ Can separate your resources by Namespace ◦ Can authorize and audit log access to your resources Why Use an API Extension? -
API Aggregation & Extension API Servers (EAS) Extension API Server
(EAS) API resource Controller -
API Aggregation & Extension API Servers (EAS) Extension API Server
(EAS) API resource Controller -
Extension API Server (EAS) API Aggregation & Extension API Servers
(EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller -
Extension API Server (EAS) API Aggregation & Extension API Servers
(EAS) Extension API Server (EAS) Extension API Server (EAS) API resource Controller -
Extension API Server (EAS) Extension API Server (EAS) Extension API
Server (EAS) API resource Controller -
EAS Forked LoC: 0 Storage: provided Components: 1 Popularity: 100s
Multiversioning: not yet Customizability: good CRD Forked LoC: 5000* Storage: you manage Components: 3 Popularity: 10s Multiversioning: yes Customizability: better * http://github.com/sample-apiserver -
Extension Ecosystem Devices 5 public plugins Storage 10 public plugins
Networking >20 public plugins Custom APIs >400 Github Projects with custom APIs -
Extension Ecosystem • 4 Serverless frameworks • 6 PaaSes •
10 CI/CD systems • 14 different database controllers • 4 popular ML toolkits -
Adding Types to the API • Extension API Servers •
Custom Resource Definitions Adding Policy to the API • ValidatingAdmissionWebhooks • MutatingAdmissionWebhooks API Extensions -
Admission: After authn/z but before storing the change. Affects mutations,
not reads. Webhooks: The API Server calls your URL, synchronously Run in cluster via service or outside, e.g. serverless. Admission Webhooks -
Old thinking: Better to make narrow specific interfaces, like ImagePolicyWebhook,
for specific use cases. Can make easier to use. Overly general extensions may limit future optimization. Admission Webhooks -
New thinking: Many custom resoures. Cluster owners need to write
policy for core resources and for custom resources written by 3rd parties. Need to compose policies written by different parties. Admission Webhooks -
Composability. Make all the changes before doing all the checks.
MutatingAdmissionWebhooks - then- ValidatingAdmissionWebhooks Admission Webhooks -
Kelsey Hightower: - reject pods that set environment variables https://github.com/kelseyhightower/denyenv-validating-admission-webhook
CRD Authors : - add complex validation Validating Admission Webhooks -
Istio: inject sidecar into all the pods Service Catalog: inject
credentials into Mutating Admission Webhooks -
- Mutate the pod template of a deployment - Install
a flaky webhook matching all resources. Bad Ideas -
•Kubernetes for Docker: • Super easy way to try Kubernetes
•API Extensions: • Use them. Author them. On Docker. For Kubernetes. •Try it: • https://goo.gl/JT7v8Z Conclusion