Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes: What is "reconciliation"?

Kubernetes: What is "reconciliation"?

A very brief exploration of what we mean when we talk about reconciliation in the context of Kubernetes APIs and controllers.

This is mostly animation-style pictures, so don't be afraid of the length of it :)

Tim Hockin

July 28, 2019
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. Google Cloud Platform Assume there’s a cloud API to make

    shapes. Why shapes? It’s just concrete enough to reason about, while not getting stuck in the details.
  2. Google Cloud Platform This API is fine, but I want

    to wrap it into a declarative system (e.g. Kubernetes)
  3. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Red kind: Shape name: Foo type: Square color: Red
  4. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Red kubectl edit shape “Foo” - type: Square - color: Purple
  5. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Update shape “Foo” - color: Purple
  6. Google Cloud Platform This is what we call “reconciliation”. Specifically,

    this is uni-directional reconciliation. What happens if...
  7. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Delete shape Foo
  8. Google Cloud Platform The shape I wanted has inadvertently been

    removed by a human or other system. If we only reconcile in one direction, we will never fix it! We need to observe that the underlying state has changed and re-assert the state we want.
  9. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple
  10. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple Create shape Bar - type: Circle - color: Red
  11. Google Cloud Platform What should the controller do? Does it

    expect to have exclusive use of all shapes? If so, clean up! Does it expect to share the shapes API with other users? If so, leave it alone! Right?
  12. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple CRASH kubectl delete shape “Foo”
  13. Google Cloud Platform The controller missed the deletion of the

    shape, but as we saw earlier, it ignored things it doesn’t know. This is a LEAK!
  14. Google Cloud Platform The controller has to know which shapes

    it owns and which it doesn’t. HOW to do that depends on the API. Examples: • Special name prefixes • Metadata (labels, tags, description) • Controller-specific checkpoints
  15. Google Cloud Platform This is sometimes called the “list-watch” pattern.

    Now the controller will keep things it owns in sync and ignores other things. What if...
  16. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple Foo labels: owner: k8s
  17. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple Create shape Bar - type: Circle - color: Red - labels: - owner: k8s Foo labels: owner: k8s
  18. Google Cloud Platform K8s Foo labels: owner: k8s API kind:

    Shape name: Foo type: Square color: Purple Bar labels: owner: k8s
  19. Google Cloud Platform K8s API List shapes where owner=k8s Foo

    labels: owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
  20. Google Cloud Platform K8s API Foo labels: owner: k8s Bar

    labels: owner: k8s kind: Shape name: Foo type: Square color: Purple Foo Bar
  21. Google Cloud Platform K8s API Foo labels: owner: k8s Bar

    labels: owner: k8s kind: Shape name: Foo type: Square color: Purple I don’t have a “Bar”
  22. Google Cloud Platform K8s API Delete shape “Bar” Foo labels:

    owner: k8s Bar labels: owner: k8s kind: Shape name: Foo type: Square color: Purple
  23. Google Cloud Platform K8s API Foo labels: owner: k8s kind:

    Shape name: Foo type: Square color: Purple
  24. Google Cloud Platform Note that while doing a full reconciliation

    at startup is necessary, it is not sufficient. Good controllers will reconcile against underlying APIs continuously or at least periodically.
  25. Google Cloud Platform How does this apply to real life?

    This pattern is found in almost every case where Kubernetes layers on top of some other API. Examples: • Cloud load-balancers for Services & Ingress • Cloud disks for PersistentVolumes • iptables rules for Services • Running containers for Pods
  26. Google Cloud Platform Sadly, not every controller gets this right.

    While every controller should strive for complete reconciliation, sometimes the underlying API makes it very hard or expensive or even just impossible. :(
  27. Google Cloud Platform There are some techniques that can mitigate

    the lack of mechanisms to denote ownership (or augment them).
  28. Google Cloud Platform K8s API Patch shape “Foo” - finalizer:

    shapes kind: Shape name: Foo type: Square color: Purple
  29. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple finalizers: - shapes
  30. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  31. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple finalizers: - shapes
  32. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes
  33. Google Cloud Platform K8s Foo API CRASH kubectl delete shape

    “Foo” kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  34. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes deletionTimestamp is set, but finalizer prevents removal of the object
  35. Google Cloud Platform K8s Foo API RECOVER kind: Shape name:

    Foo type: Square color: Purple finalizers: - shapes
  36. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple finalizers: - shapes
  37. Google Cloud Platform K8s API Delete shape “Foo” Foo labels:

    owner: k8s kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  38. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple finalizers: - shapes
  39. Google Cloud Platform K8s API Patch shape “Foo” to remove

    finalizer kind: Shape name: Foo type: Square color: Purple finalizers: - shapes
  40. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple deletion can be completed
  41. Google Cloud Platform K8s API Create ShapeRef “Foo” - ownerReference:

    Foo kind: Shape name: Foo type: Square color: Purple
  42. Google Cloud Platform K8s API kind: Shape name: Foo type:

    Square color: Purple kind: ShapeRef name: Foo
  43. Google Cloud Platform K8s API Make shape “Foo” - type:

    Square - color: Purple kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
  44. Google Cloud Platform K8s Foo API kind: Shape name: Foo

    type: Square color: Purple kind: ShapeRef name: Foo
  45. Google Cloud Platform K8s Foo API CRASH kind: Shape name:

    Foo type: Square color: Purple kind: ShapeRef name: Foo
  46. Google Cloud Platform K8s Foo API CRASH kubectl delete shape

    “Foo” kind: Shape name: Foo type: Square color: Purple kind: ShapeRef name: Foo
  47. Google Cloud Platform The controller did not observe the deletion

    of the Shape, but it does observe the dangling ShapeRef.
  48. Google Cloud Platform In most of these mechanisms, there’s some

    amount of “you broke it, you bought it”. If a user deletes the ShapeRef or removes the finalizer or edits the underlying metadata, the linkage can be broken. You broke it, you get to keep the pieces.