of truth that only talk via queues Reconciliation Loops Stateless components that talk to one store of truth There are only two good* ways to build distributed systems**:
new Deployment • Deployment controller notices that the deployment has no pods ◦ To reconcile this, it makes one • Pod controller notices that the pod is not assigned ◦ To reconcile this, it assigns it to a node • Kubelet notices it has an assigned pod that is not running ◦ To reconcile this, it creates it locally and starts it
is scheduled to run, and the scheduler passes it over • The Executor makes a new Workload for the Task Instance • Allocator notices that the Workload has no Runner ◦ To reconcile this, it assigns it to a runner • Runner notices it has an assigned Workload that is not running ◦ To reconcile this, it creates it locally and starts it
You can maybe get away with caching elsewhere Zero service-to-service comms Communication paths are failure modes Controllers should be simple loops Query database, reconcile, repeat