Kubernetes from First Principles

The first time most people encounter Kubernetes, they're handed a YAML file, told to run kubectl apply, and left to wonder why the cluster needs thirteen components to do what a single shell script used to do. This is the wrong frame. The shell script could not do what Kubernetes does. The shell script ran once. Kubernetes runs forever.

To understand why Kubernetes is designed the way it is, you have to start with the problem it was built to solve, not the feature list, but the actual operational pain that made the feature list necessary.

The problem before Kubernetes

Deploying software in the pre-container world meant a sequence of manual or scripted operations: provision a server, install dependencies, copy binaries, write init scripts, configure a load balancer, and then pray nothing changed between the staging environment and production.

It worked, mostly, at small scale. The failure modes emerged as the scale grew. A server dies at 3am: do you have an on-call engineer who knows exactly what was running on it? You need to deploy a new version: do you take downtime, or do you have a rolling deployment strategy encoded somewhere that isn't someone's memory? You need to go from 10 instances to 200 because traffic spiked: how fast can you do that, and how sure are you that instance 147 is configured identically to instance 12?

The scripts became elaborate. The state became implicit, spread across config files on live servers, init scripts with undocumented assumptions, and the institutional memory of whoever set things up three years ago. Nobody could tell you with confidence what was actually running at any given moment. The documentation was probably wrong. The servers were almost certainly not identical to each other.

This is the problem Kubernetes was built to solve. Not "how do we run containers." How do we run a fleet of services at scale, with self-healing, with reproducibility, with the ability to deploy and roll back safely, without requiring a human to be on call for every failure mode we can predict.

The reconciliation loop model

The central idea in Kubernetes is deceptively simple. You declare desired state: "I want 3 replicas of this container, with these resource limits, accessible on this port." You write that declaration in a YAML file and submit it to the cluster. From that point on, Kubernetes makes it its job to ensure that reality matches your declaration, not just once, but continuously.

A controller (a control loop, a program that watches some state and acts when it diverges from desired) compares desired state to actual state on a tight loop. If you have 3 replicas and one crashes, the actual count drops to 2. The controller notices, starts a new one, and the count returns to 3. You didn't ask it to do this. You declared desired state once, and the system maintains it indefinitely.

This is different from "a scheduler that places containers on machines." A scheduler runs once per placement decision. A reconciliation loop runs continuously and handles machine death, network failures, and any other disruption to the system's state. The reconciliation model generalizes: every resource type in Kubernetes (Deployments, Services, ConfigMaps, Ingresses) is managed by a controller that watches it and reconciles. The controllers compose into a system that can manage arbitrarily complex desired states.

The components

Understanding how the pieces fit together removes a lot of the apparent magic. The Kubernetes control plane is a small set of components with well-defined responsibilities.

The API server is the single source of truth for cluster state. Every other component, whether you're running kubectl get pods or a controller deciding where to place a workload, talks through the API server. Nothing writes to etcd directly except the API server. This gives you a single auditable point of entry for every state change.

etcd is the durable store for all cluster state. It is a distributed key-value store with strong consistency guarantees (it uses Raft, a consensus protocol for distributed systems). When you submit a YAML manifest, it ends up as data in etcd. The API server reads and writes etcd; everything else reads through the API server.

The scheduler watches for pods (the smallest deployable unit: one or more containers that share a network namespace) that have been created but not yet assigned to a node. It evaluates candidate nodes based on resource availability, affinity constraints, and policy, picks one, and writes the assignment back to the API server. That is all it does. It does not start the pod.

The kubelet runs on every worker node. It watches the API server for pods assigned to its node and ensures those pods are actually running. It starts containers, restarts them if they crash, reports their status back to the API server, and handles liveness and readiness probes (health checks that let Kubernetes know whether a pod is alive and ready to receive traffic). The kubelet is where intent becomes actual running processes.

Controllers watch specific resource types and reconcile. The Deployment controller watches Deployment objects and manages ReplicaSets (which manage the actual pods). The Service controller ensures the right pods are receiving network traffic. There are dozens of built-in controllers, each narrowly focused.

The flow for a typical deployment: you submit a Deployment manifest to the API server → it's stored in etcd → the Deployment controller notices → it creates a ReplicaSet → the ReplicaSet controller creates Pods → the scheduler assigns each Pod to a node → the kubelet on that node starts the containers. A request path that spans five components, but each step is simple and decoupled.

Why stateless workloads are easy

A stateless pod (a web server, an API service, a background worker) holds no data. It is fungible. Kill it, make another one, and from the system's perspective nothing was lost (assuming requests were routed away from it before death, which the readiness probe handles).

This is the sweet spot for Kubernetes. Rolling deployments are just a desired-state change: "transition from version 1.2 to version 1.3, replacing pods in batches of 2." Scaling is another desired-state change: "update the replica count from 10 to 50." Self-healing is the reconciliation loop noticing that actual count doesn't match desired count and correcting. Every operation that would have required a deployment script, a load balancer reconfiguration, and a careful human is now a YAML field change.

Horizontal Pod Autoscaling extends this: you declare desired CPU utilization, and a controller adjusts the replica count to hit that target. The humans are out of the loop for normal scaling events. This is the core value proposition: operational work that used to require human judgment is now encoded in the control loop.

Why stateful workloads are hard

A database pod is not fungible. It has identity. Its data lives on a specific disk. The leader election state is significant. You cannot kill a PostgreSQL primary and spin up an identical replacement without a carefully orchestrated promotion of a replica to primary and a reconfiguration of clients.

Kubernetes has a resource type called StatefulSet that gives pods stable network identity (pod-0, pod-1, pod-2 rather than random hashes) and persistent volumes that travel with pod identity across rescheduling. These primitives are necessary but not sufficient.

The coordination logic (how you promote a replica when the primary fails, how you handle split-brain, how you perform a schema migration without downtime) is specific to each database and can't be generalized. This is why the Operator pattern exists. An Operator is a custom controller that encodes the operational knowledge for a specific stateful system. The PostgreSQL Operator knows how to run Postgres. The Kafka Operator knows how to run Kafka. You can run these in Kubernetes, but the Kubernetes primitives alone don't get you there.

Kubernetes gives you the substrate. The database-specific operational logic still has to exist somewhere. Operators move it from runbooks and experienced humans into code, which is better, but not free.

The scheduling problem

Placing pods on nodes is a bin-packing problem (fit items of varying size into bins of limited capacity) with a set of constraints layered on top. The scheduler has to satisfy resource requests (a pod that asks for 4 CPUs and 8GB RAM can only go on a node with that much available), affinity and anti-affinity rules (run these pods together; run these pods apart), taints and tolerations (this node has GPUs; only pods that explicitly tolerate GPU nodes should land here), and topology spread constraints (distribute pods evenly across availability zones).

The scheduler isn't magic and it isn't optimal in the mathematical sense. It's a heuristic optimizer that runs in bounded time. It filters the candidate nodes down to those that satisfy hard constraints, then scores the remaining candidates on soft preferences, and picks the highest scorer. It makes a good decision quickly rather than an optimal decision slowly, which is the right tradeoff when you're scheduling thousands of pods per minute.

Getting scheduling right for a specific workload sometimes requires tuning. A machine-learning training job that needs GPUs, 64GB RAM, and needs to run on nodes in a specific availability zone is going to require more annotation than a stateless API service. The scheduler can handle it; you have to tell it what you want.

The complexity is real, but so is the alternative

Kubernetes has a justified reputation for operational complexity. The control plane has many moving parts. The YAML can be verbose. Debugging a pod stuck in CrashLoopBackOff requires understanding several layers. The networking model is non-obvious. People write entire books about Kubernetes and don't cover everything.

But here's the thing: the complexity that Kubernetes encodes (managing workload placement, health checking, rolling deployments, scaling, network routing) existed before Kubernetes. It was just scattered across shell scripts, Chef cookbooks, Ansible playbooks, and the brains of specific engineers who had been there long enough to understand the system. That complexity was less visible, not less real.

What Kubernetes actually does is make the complexity legible and declarative. The desired state is in version control. The behavior of the control loops is documented. The operational knowledge is in code, not memory. The system recovers from failures without human intervention, not because failures don't happen, but because the reconciliation loop handles them automatically.

The reconciliation model is also extensible. You can define custom resource types (Custom Resource Definitions) and write controllers that manage them. The same machinery that Kubernetes uses to manage Deployments and Services can be used to manage databases, message queues, ML training jobs, or anything else that has a meaningful notion of desired versus actual state.

The container orchestration use case is just the first application of a more general pattern. A system built to manage containers can be extended to manage almost anything with a meaningful notion of desired versus actual state. The complexity cost is real. It is also a one-time investment, rather than a recurring debt that accrues every time someone SSH's into a server and makes a change they don't document.