Kubernetes Controllers - are they loops or events?

1. Kubernetes Controllers Are they loops or events? Tim Hockin @thockin v1
2. Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
3. Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
4. Usually when we talk about controllers we refer to them as a “loop”
5. Imagine a controller for Pods (aka kubelet). It has 2 jobs: 1) Actuate the pod API 2) Report status on pods
6. What you’d expect looks something like:
7. Node Kubernetes API a kubelet b c Get all pods
8. Node Kubernetes API a kubelet b c { name: a, ... } { name: b, ... } { name: c, ... }
9. Node Kubernetes API a kubelet b c for each pod p { if p is running { verify p config } else { start p } gather status }
10. Node Kubernetes API a kubelet b c Set status c a b
11. ...then repeat (aka “a poll loop”)
12. Here’s where it matters
13. Node Kubernetes API a kubelet b c c a b kubectl delete pod b
14. Node Kubernetes API a kubelet c c a b kubectl delete pod b
15. Node Kubernetes API a kubelet c Get all pods c a b
16. Node Kubernetes API a kubelet c { name: a, ... } { name: c, ... } c a b
17. Node Kubernetes API a kubelet c I have “b” but API doesn’t - delete it! c a b
18. Node Kubernetes API a kubelet c Set status c a
19. This is correct level-triggered reconciliation Read desired state, make it so
20. Some controllers are implemented this way, but it’s inefficient at scale
21. Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling continuously
22. We need to achieve the same behavior more efficiently
23. We could poll less often, but then it takes a long (and variable) time to react - not a great UX
24. Enter the “list-watch” model
25. Node Kubernetes API a kubelet b c Get all pods
26. Node Kubernetes API a kubelet b c { name: a, ... } { name: b, ... } { name: c, ... }
27. Node Kubernetes API a kubelet b c Cache: { name: a, ... } { name: b, ... } { name: c, ... }
28. Node Kubernetes API a kubelet b c Watch all pods Cache: { name: a, ... } { name: b, ... } { name: c, ... }
29. Node Kubernetes API a kubelet b c Cache: { name: a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
30. Node Kubernetes API a kubelet b c Set status c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
31. We trade memory (the cache) for other resources (API server CPU in particular)
32. There’s no point in polling my own cache, so what happens next?
33. Remember that watch we did earlier? That’s an open stream for events.
34. Node Kubernetes API a kubelet b c c a b kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
35. Node Kubernetes API a kubelet c c a b kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
36. Node Kubernetes API a kubelet c Delete: { name: b, ... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
37. Node Kubernetes API a kubelet c Delete: { name: b, ... } c a b Cache: { name: a, ... } { name: c, ... }
38. Node Kubernetes API a kubelet c Cache: { name: a, ... } { name: c, ... } c a b API said to delete pod “b”.
39. Node Kubernetes API a kubelet c Cache: { name: a, ... } { name: c, ... } c a API said to delete pod “b”.
40. “But you said edge-triggered is bad!”
41. It is! But this isn’t edge-triggered.
42. The cache is updated by events (edges) but we are still reconciling state
43. “???”
44. The controller can be restarted at any time and the cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
45. Even if you miss an event, you can still recover the state
46. Ultimately it’s all just software, and software has bugs. Controllers should re-list periodically to get full state...
47. ...but we’ve put a lot of energy into making sure that our list-watch is reliable.