scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin
scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin
scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin
scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin
scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin WebAssembly Via WebAssembly plugin
The batch jobs requirements. ◦ Start several Pods at the same time (coscheduling) ◦ Elastic resource quota (capacityscheduling) • You’ll find many other usecases in kubernetes-sigs/scheduler-plugins Extend your scheduler
scheduling constraints on Pod spec Control the scheduling per Pod. KubeSchedulerConfiguration Control the scheduling per cluster. Extend the scheduler Extender Via webhook Plugin Via your own scheduler plugin WebAssembly Via WebAssembly plugin
called at a specific point in scheduling: • Filter: Filter in Scheduling Framework • Prioritize: Score in Scheduling Framework • Preempt: PostFilter in Scheduling Framework • Bind: Bind in Scheduling Framework Extender
all scheduling logic from the scheduler’s core impl • One scheduling factor = One plugin • We can extend the scheduler by creating your own plugins. Plugin (Scheduling framework)
and keep consistent with your Kubernetes version. • The scheduler should be only one in the cluster. ◦ Need to let all Pods go through a new scheduler in some ways. ◦ May need to convince people managing the scheduler in your cluster. (your infra team, cloud vendor, etc) ◦ May need to maintain multiple scheduling plugins owned by different teams in the scheduler. Hurdles for Plugin extension
and keep consistent with your Kubernetes version. • The scheduler should be only one in the cluster. ◦ Need to let all Pods go through a new scheduler in some ways. ◦ May need to convince people managing the scheduler in your cluster. (your infra team, cloud vendor, etc) ◦ May need to maintain multiple scheduling plugins owned by different teams in the scheduler. Can we call it a true pluggable system? 🤔 Hurdles for Plugin extension
in other languages. • Wasm runtimes execute wasm guests (xxxx.wasm) • Wasm guests import functions from host. ◦ = They cannot do other things. You may hear it around the browser stuff, but… WebAssembly
wasm guest to satisfy ABIs. We’re providing TinyGo SDK so that people can develop wasm plugins via a similar experience with Golang native plugins. – just need to implement interfaces. TinyGo SDK
plugin. • 👍 All extension points are available. • 👍 No need to change the scheduler’s code, no need to rebuild! • 👍 Easy to distribute plugins. (via http(s)) • 👍 It can be written in many languages. • 👎 A bad impact on the latency. • 👎 Wasm peculiar limitations. So… will wasm extension replace all plugins?
plugin. • 👍 All extension points are available. • 👍 No need to change the scheduler’s code, no need to rebuild! • 👍 Easy to distribute plugins. (via http(s)) • 👍 It can be written in many languages. • 👎 A bad impact on the latency. • 👎 Wasm peculiar limitations. So… will wasm extension replace all plugins?
The scheduler’s latency is super critical in your cluster. ◦ The bigger cluster you get, the faster scheduling is needed. • You need to do heavy-calculation or handle tons of various objects. ◦ Due to inlined GC and the latency to pass objects from host to guest. (both will be discussed in later section) So… will wasm extension replace all plugins?
can only operate their memory. • The guest memory is exported to host so that host can read or write anything. • Only numeric types are supported. The wasm function can only operate their memory
Guest: allocates enough memory for URI and gives the linear memory offset and maximum length in bytes. • Host: put the URI there and tell guest the length of it. How to pass things from host
scheduling cycle. → Why don’t we have a cache in wasm guest! We reduce the number of communication as much as possible with them: • Fetch objects from host only when it’s necessary for the guest. • Fetch the same object from host only once during one scheduling cycle. Lazy loading with caching
We have two levels of benchmark test: • Plugin-level benchmark tests, using the Golang benchmark test feature. ◦ How much it takes time in which part. • The scheduler_perf, running the wasm plugin in the scheduler and observing the scheduler’s metrics. ◦ How much actually the wasm slows down the scheduling. Benchmark tests
out to all contributors so far, especially Adrian for tons of contributions, and all people, especially Chris for all helps to bring me here, Wellington . We’re running through many things in 30 min 🏃💨, Thanks all! That’s all!
request on Pod. • Affinity requirement on Pod. (PodAffinity, NodeAffinity) • How Pods are spread into each domain now. (PodTopologySpread) • Taints on Nodes / Tolerations on Pod. • …etc Scheduling factors
composed of many Plugins. ◦ One scheduling factor = one plugin (NodeAffinity plugin, etc) • Each plugin is created to work on one or more extension points. ◦ Filter: filtering Nodes that don’t fit the requirements ◦ Score: scoring the remaining Nodes ◦ … Scheduling Framework
For example.. • Nodes that don’t have enough resource to run the Pod • Nodes that don’t match with a required NodeAffinity on the Pod • … Scheduling Framework - Filter
For example, give higher scores to Nodes • already have the container image(s) of the Pod • match with a preferred NodeAffinity on Pod • … Scheduling Framework - Score
up. You can see our investigation and discussion here: • https://github.com/kubernetes/kubernetes/issues/106705 • https://github.com/kubernetes/kubernetes/issues/100723 Several attempts to make it easier
– load and run the wasm binary in the host. In Golang, there are already some: - Dapr Wasm middleware - Trivy Modules - knqyf263/go-plugin Powered by WebAssembly with Golang