Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Kubernetes Controllers - are they loops or events?
Search
Tim Hockin
February 20, 2021
Technology
11
3.4k
Kubernetes Controllers - are they loops or events?
Tim Hockin
February 20, 2021
Tweet
Share
More Decks by Tim Hockin
See All by Tim Hockin
Kubernetes in the 2nd Decade
thockin
0
140
Why Service is the worst API in Kubernetes, and what we can do about it
thockin
1
550
Kubernetes Pod Probes
thockin
6
3.2k
Go Workspaces for Kubernetes
thockin
2
890
Code Review in Kubernetes
thockin
2
1.5k
Multi-cluster: past, present, future
thockin
0
370
Kubernetes Network Models (why is this so dang hard?)
thockin
9
1.7k
KubeCon EU 2020: SIG-Network Intro and Deep-Dive
thockin
8
1.2k
A Non-Technical Kubernetes Talk (KubeCon EU 2020)
thockin
3
550
Other Decks in Technology
See All in Technology
Gitlab本から学んだこと - そーだいなるプレイバック / gitlab-book
soudai
7
1.3k
Gradle Build Scanを使ってビルドのことを知ろう potatotips #87
tomorrowkey
2
150
Rustで「プリズモイダル法」を利用して「土量計算」をガチでやる
nokonoko1203
1
280
Microsoft for Startups Founders Hub_20240429 update
daikikanemitsu
1
2.4k
コードファーストの考え方。 Amplify Gen2から学ぶAWS次世代のWeb開発体験
yoshiitaka
1
310
Microsoft Intune 勉強会 第 2 回目
tamaiyutaro
2
380
Handling focus in 2024
tahia910
0
220
AWSに詳しくない人でも始められるコスト最適化ガイド
yuhta28
2
320
2024春 注目のWeb系 OSS & SaaS 3選
makies
0
170
Kernel MemoryでAzure OpenAI Serviceとお手軽データソース連携
mitsuzono
1
280
エンジニア候補者向け資料2024.04.24.pdf
macloud
0
3.3k
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
150
Featured
See All Featured
Teambox: Starting and Learning
jrom
128
8.4k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
The Mythical Team-Month
searls
216
42k
Producing Creativity
orderedlist
PRO
338
39k
Documentation Writing (for coders)
carmenintech
61
4k
A Tale of Four Properties
chriscoyier
152
22k
Embracing the Ebb and Flow
colly
80
4.2k
What's in a price? How to price your products and services
michaelherold
238
11k
The Art of Programming - Codeland 2020
erikaheidi
43
12k
How GitHub (no longer) Works
holman
305
140k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
Transcript
Kubernetes Controllers Are they loops or events? Tim Hockin @thockin
v1
Background on “reconciliation”: https://speakerdeck.com/thockin/kubernetes-what-is-reconciliation
Background on “edge vs. level”: https://speakerdeck.com/thockin/edge-vs-level-triggered-logic
Usually when we talk about controllers we refer to them
as a “loop”
Imagine a controller for Pods (aka kubelet). It has 2
jobs: 1) Actuate the pod API 2) Report status on pods
What you’d expect looks something like:
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c for each pod
p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b
...then repeat (aka “a poll loop”)
Here’s where it matters
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b
Node Kubernetes API a kubelet c c a b kubectl
delete pod b
Node Kubernetes API a kubelet c Get all pods c
a b
Node Kubernetes API a kubelet c { name: a, ...
} { name: c, ... } c a b
Node Kubernetes API a kubelet c I have “b” but
API doesn’t - delete it! c a b
Node Kubernetes API a kubelet c Set status c a
This is correct level-triggered reconciliation Read desired state, make it
so
Some controllers are implemented this way, but it’s inefficient at
scale
Imagine thousands of controllers (kubelet, kube-proxy, dns, ingress, storage...) polling
continuously
We need to achieve the same behavior more efficiently
We could poll less often, but then it takes a
long (and variable) time to react - not a great UX
Enter the “list-watch” model
Node Kubernetes API a kubelet b c Get all pods
Node Kubernetes API a kubelet b c { name: a,
... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Watch all pods
Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet b c Cache: { name:
a, ... } { name: b, ... } { name: c, ... } for each pod p { if p is running { verify p config } else { start p } gather status }
Node Kubernetes API a kubelet b c Set status c
a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
We trade memory (the cache) for other resources (API server
CPU in particular)
There’s no point in polling my own cache, so what
happens next?
Remember that watch we did earlier? That’s an open stream
for events.
Node Kubernetes API a kubelet b c c a b
kubectl delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c c a b kubectl
delete pod b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: b, ... } { name: c, ... }
Node Kubernetes API a kubelet c Delete: { name: b,
... } c a b Cache: { name: a, ... } { name: c, ... }
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a b API said to delete pod “b”.
Node Kubernetes API a kubelet c Cache: { name: a,
... } { name: c, ... } c a API said to delete pod “b”.
“But you said edge-triggered is bad!”
It is! But this isn’t edge-triggered.
The cache is updated by events (edges) but we are
still reconciling state
“???”
The controller can be restarted at any time and the
cache will be reconstructed - we can’t “miss an edge*” * modulo bugs, read on
Even if you miss an event, you can still recover
the state
Ultimately it’s all just software, and software has bugs. Controllers
should re-list periodically to get full state...
...but we’ve put a lot of energy into making sure
that our list-watch is reliable.