Controllers

A traditional ops team monitors dashboards, notices a crashed service, SSHes into a server, and restarts it. If it is 3 AM, the alert wakes someone up. If the person is slow, the outage lasts minutes or hours. If the alert is misconfigured, nobody notices until users complain.

In Kubernetes, nobody restarts crashed services. Nobody notices crashed services. The system detects and fixes the problem before a human could possibly react. A pod crashes? A new one is running within seconds. A node dies? Its pods are rescheduled to healthy nodes automatically. A deployment is rolled out with a bug? Traffic is drained from bad pods and the rollout is paused.

This is not magic. It is controllers. A controller is a program that runs in an infinite loop: observe the current state of the cluster, compare it to the desired state, and take the minimum action to close the gap. If the desired state says 3 replicas and the current state shows 2, the controller creates one more. If the desired state says version 1.25 and a pod is running 1.24, the controller terminates the old pod and creates a new one. The reconciliation loops lesson runs this loop on a real cluster and watches it undo a change made by hand.

Every major Kubernetes feature is implemented as a controller. Deployments, StatefulSets, DaemonSets, Jobs, Horizontal Pod Autoscaler, Node lifecycle. Understanding controllers means understanding how Kubernetes thinks.

Google's Borg system (Kubernetes' predecessor) ran on the same principle. The 2015 Borg paper describes controllers as "programs that reconcile desired and observed state." Kubernetes inherited this architecture directly.

Every controller follows the same structure:

while true:
  desired = read(desired state from API server)
  actual = observe(current state of cluster)
  diff = compare(desired, actual)
  if diff:
    act(minimum changes to close the gap)
  else:
    # System is converged. Nothing to do.
  wait(next event or periodic resync)

This loop runs continuously. Not on a schedule. Not on demand. Continuously. The controller manager (kube-controller-manager) runs dozens of these loops in parallel, one for each resource type.

Built-in controllers and what they manage:

Controller composition: A Deployment controller does not manage pods directly. It creates and manages ReplicaSets. The ReplicaSet controller manages pods. During a rolling update, the Deployment creates a new ReplicaSet (with the new image) and scales it up while scaling the old ReplicaSet down. Two simple controllers compose to produce complex rolling update behavior.

This layered design keeps each controller simple. The Deployment controller only knows about update strategy. The ReplicaSet controller only knows about replica count. Neither needs to understand the other's logic.