Infrastructure as Code: Why Immutability Matters

You SSH into a production server at 2am to hotfix a config. It works. Six months later, the server gets replaced in a region failover. The new server doesn't have your hotfix. Nobody remembers what you changed. The failure mode you fixed at 2am is now back, and this time it's during a real incident.

This is configuration drift, the slow accumulation of undocumented changes between what your infrastructure actually is and what you believe it to be. Infrastructure as code is the engineering discipline that prevents it. Immutability is the property that makes the prevention robust.

What drift is

Configuration drift happens when actual infrastructure state diverges from documented or expected state. It accumulates in predictable ways: hotfixes applied under pressure and never backported to the provisioning scripts; partial deployments where 7 out of 10 servers got the config change and the other 3 didn't; library upgrades applied manually to one server to debug an issue and never rolled out consistently; init scripts that reference a file path that changed two years ago, kept alive because the server that runs them was provisioned when the path was correct.

At small scale, drift is annoying. You have to figure out which server is the weird one and why. At scale, drift is dangerous, because it means your environments aren't identical. "Works on staging, broken in prod" is often drift. More seriously: your disaster recovery plan describes spinning up new infrastructure from a documented baseline. If that baseline has drifted from production, you're not recovering to what was running. You're recovering to an older, different configuration, and you won't know which differences matter until something fails.

The insidious thing about drift is that it's invisible. A running server doesn't announce that it has been modified since provisioning. You have to actively audit for it, and auditing is laborious, so it doesn't happen. Drift accumulates steadily until a high-pressure moment (a server being replaced, a disaster recovery test, an incident that requires replicating an environment) makes it suddenly, expensively visible.

The IaC model

Infrastructure as code (IaC) describes your infrastructure in code, declaratively, not as a sequence of commands but as a description of the state you want. Terraform, OpenTofu, CloudFormation, and Pulumi are the main tools. The pattern is the same: you write a description of the resources you want (3 EC2 instances with these specifications, this security group, this load balancer, these DNS records), and the tool computes the difference between the declared state and the actual state of your cloud account, then applies the changes needed to make them match.

The key property is that the code is the source of truth. If a resource exists in your cloud account but not in your Terraform code, a terraform plan will propose destroying it. If someone manually changes a security group rule, the next plan will propose reverting it. Any undeclared change creates visible drift that will be corrected on the next apply. The system enforces the code's primacy rather than relying on discipline.

Practically, this means: infrastructure changes go through pull requests, get code review, are tested in a non-production environment before being applied to production, and are tracked in git history with the same rigor as application code. Who changed what and why is recorded. Rolling back an infrastructure change is a git revert and a terraform apply, not a careful sequence of manual commands on a running system.

Why immutability matters

IaC eliminates undocumented changes, but it still allows for mutable infrastructure in the sense that you can declare "upgrade this instance's OS to Ubuntu 24.04" and Terraform will modify the running instance. This is better than doing it manually, but it accumulates a different kind of problem: the instance now has a history. It was provisioned with one OS version, upgraded to another. The upgrade path worked on this specific instance with this specific set of installed packages, accumulated caches, and temporary files from previous deployments. It might not work the same way on a freshly provisioned instance.

The immutable approach eliminates this class of problem by never modifying running instances. When you need to update something (the OS, the application code, a config file), you build a new machine image (an AMI in AWS terms: a snapshot of a configured disk) that incorporates the change, and replace the running instances with new instances booted from the new image. The old instances are destroyed. Nothing accumulates from before.

The term "cattle, not pets" captures the operational philosophy. A pet is a specific server with a name and a history that you carefully maintain. When it's sick, you diagnose it, repair it, and nurse it back to health. A cattle server is a fungible unit. When it's broken, you replace it. The replacement is identical to what you intended (because it was built from the image you intended), not to the specific history of the server you replaced.

In practice: you maintain an automated image build pipeline (Packer is the standard tool for this). Every infrastructure change produces a new image. Deployments swap the image version in your IaC code, Terraform replaces the instances (either one-by-one in a rolling fashion or all-at-once with a blue-green deployment), and the old instances are gone. No accumulated state. No drift. The build pipeline is the definition of what the server is; the running server is just an instantiation.

The benefits, concretely

If your entire production infrastructure is described in code, you can spin up an identical environment from scratch. That pays off in disaster recovery (apply the same code to a new region or account and get production back), in staging environments that accurately reflect production (same code, different variable values), and in debugging (spin up a copy of the failing environment without touching production).

The git history for your infrastructure repository is the complete audit trail for every infrastructure change. When did the security group rule change? Who changed it? Why? The commit message, the pull request, the review comments, all present. For compliance-sensitive environments this is often a requirement, but it is useful regardless.

Running the same Terraform code against a development AWS account with smaller resource sizes (different variable file, same code) gives you an environment that is structurally identical to production. The networking topology, the IAM policies, the service interactions, all the same. This is where "works in staging, broken in prod" failures live: structural differences between environments that nobody maintains because it is tedious. IaC makes them explicit and automatable.

A DR plan you can actually run, and have run in tests repeatedly, is categorically different from a DR plan that is documented but untested. If your infrastructure is code, running a DR test is "apply the code to a fresh environment." You find out what doesn't work before you need it in an actual emergency, and you fix it. This is where the value of IaC is most viscerally apparent: the teams with fully codified infrastructure find their DR tests succeed; the teams with partially codified infrastructure find that the undocumented parts fail.

The sharp edges

Terraform state files deserve special attention because they're the mechanism and the point of failure. Terraform maintains a state file that maps your declared resources to the actual cloud resources that exist. When you apply, Terraform reads the state file, queries the cloud API to see what's there, and computes a plan. The state file is the source of truth about what Terraform thinks exists.

If the state file drifts from reality (someone manually deleted a resource, or imported a resource that Terraform didn't create), the next plan may propose incorrect changes. Terraform state files should live in remote, shared, versioned storage (S3 with versioning and locking via DynamoDB is the standard AWS pattern) so they're not lost if a workstation is lost and so concurrent applies don't corrupt each other.

Stateful resources need careful handling that pure immutability can't solve. You cannot destroy and recreate a PostgreSQL instance the way you can destroy and recreate an EC2 instance, because the data doesn't persist. Terraform's prevent_destroy lifecycle rule marks a resource as something that should never be destroyed (Terraform will error rather than proceed if a plan includes destroying it). Databases, S3 buckets with data, and other resources whose destruction is catastrophic should have this set.

There's also the challenge of state import: if you have existing infrastructure that wasn't created by Terraform, bringing it under Terraform management requires importing it into the state file, then verifying that the declared configuration matches reality closely enough that an apply produces no changes. This is often tedious, but doing it correctly is necessary. Half-imported infrastructure, where some resources are managed by Terraform and their dependencies are not, leads to the same drift problems you were trying to solve.

Code is the source of truth, or it isn't

Infrastructure as code is a discipline, not just a tool choice. Using Terraform while continuing to make manual changes "just for now" doesn't get you the benefits; it gets you the overhead of maintaining IaC code plus the continued accumulation of drift. The policy has to be: if it's not in code, it doesn't exist. If it needs to exist, it goes in code first, then gets applied. No exceptions.

Immutability is what makes the discipline self-enforcing rather than self-disciplining. If you replace instances rather than modifying them, there's no running instance to SSH into and change. The workflow doesn't permit the drift vector. You modify the image build, produce a new image, and deploy it. The code is the mechanism, not just the documentation.

This is the functional programming principle applied to infrastructure: instead of maintaining mutable state and carefully tracking every change to it, you compute the desired state from first principles and replace the old state entirely. The deployment is the test of whether the code is correct. If you can recreate your infrastructure from your code and have it work, not theoretically but demonstrably in a tested DR exercise, your code is the source of truth. If you can't, it's documentation that may have drifted.