Flaky tests get a lot of attention. They have dashboards, triage rotations, and opinions so strong that engineers will pause entire feature tracks to argue about the correct quarantine strategy. Flaky builds (builds that fail for reasons unrelated to the code change) get almost none of this treatment, despite causing the same developer frustration and the same blocked deploys.
The pattern is recognizable: a CI run fails, the engineer looks at the logs, sees something about a network timeout or a dependency that wasn't found or a test that failed to acquire a port, concludes it's probably not their change, reruns, and an hour later the build is green. The blocked work resumes. Nobody files a bug. Nobody investigates the root cause. The same failure happens to a different engineer two days later, and the cycle repeats.
The root cause is almost always the same thing: the build depends on something that isn't declared as an input. The technical term is non-hermetic execution, and it's worth understanding precisely what that means and why it matters.
What hermeticity means
A hermetic build produces the same output given the same inputs, regardless of the machine it runs on, the time it runs, or what other builds ran before it. If you check out the same commit on two different machines and run the build, you should get byte-for-byte identical artifacts.
This is a stricter property than it sounds. It requires that all inputs are fully specified and fixed: source code, build tools, compiler version, system libraries, dependency versions, environment variables, the current time if the build reads it. Any input that can vary between runs is a potential source of divergence.
Most builds aren't hermetic. Most teams don't notice because the divergence is rare enough to be dismissed as a fluke rather than diagnosed as a structural problem. The flukiness is the tell: a hermetic build either always succeeds or always fails for a given input. "Usually works, sometimes doesn't" is diagnostic of a non-hermetic input that varies between runs.
Non-hermeticity in practice
Non-hermetic inputs take a few common forms.
Unpinned Docker base images are one common source. A Dockerfile that begins with
FROM python:3.11 pulls the image tagged 3.11 at build time.
The content behind that tag is mutable: the maintainer can push a new image to that
tag at any point. The same Dockerfile built today and built next month may produce
different environments. The fix is to pin by digest:
FROM python:3.11@sha256:<hash>. The digest is a cryptographic hash
of the image content and cannot refer to different content at different times.
System-installed tools without version pinning are another. A build step that runs
apt-get install python3 installs whatever version of Python 3 is current
in the package repository at build time. That version changes when the package
maintainer updates it. The same setup script gives you Python 3.10 in January and
Python 3.12 in October. The fix is explicit version constraints, or better, a tool
version manager with a committed lockfile.
Shared state between test runs introduces ordering dependencies. Tests that write to a shared database without cleaning up after themselves create this problem: test A passes when it runs first and fails when test B has already modified the relevant rows. Rerun the suite and the order might change. The failure is non-deterministic from the observer's perspective, but it's actually deterministic. The relevant input (database state) isn't being controlled.
Network calls during the build tie build success to external availability. Running
pip install or npm install against the live registry without a
lockfile means the registry going down fails the build, a yanked package fails the build,
and a transitive dependency releasing a breaking version fails the build in a way that
isn't related to any code change. Lockfiles (package-lock.json,
Pipfile.lock, go.sum) address this by pinning the resolved
dependency graph so that the same packages are installed every time.
Why it matters at scale
A 1% build flakiness rate sounds low. Run the numbers on a team of a hundred engineers running five CI builds each per day and you get fifty flaky build incidents per day. Each one follows the same pattern: engineer notices the failure, checks the logs, decides it's not their change, reruns, waits twenty minutes, proceeds. Call it thirty minutes of overhead per incident. That's twenty-five engineer-hours per day, more than three full-time engineers' worth of time, spent on CI reruns rather than engineering.
This math compounds. At a thousand engineers with the same 1% rate, it's a dedicated team. And 1% is optimistic. Flakiness rates in real build systems, without active investment in hermeticity, tend to be higher and tend to grow over time as more non-hermetic inputs accumulate.
There's a secondary effect that's harder to quantify but probably more costly: eroded trust in CI signal. When builds fail for reasons unrelated to code changes, engineers learn to dismiss failures as probably-flaky rather than investigating them. The test suite stops functioning as a reliable signal. A genuine regression slips through because the engineer who saw the red CI assumed it was the usual noise and reran until it was green. This is a real failure mode.
What Bazel does differently
Bazel, Google's open-source build system originally designed for their monorepo, is designed around hermeticity as a first-class constraint rather than a nice-to-have.
In Bazel, every build rule declares its inputs explicitly. If a build step reads a file that isn't declared as an input, Bazel won't make it available. If a step depends on a tool, that tool must be declared as a dependency with a pinned version. There's no ambient environment: no system Python, no globally installed npm packages, no environment variables leaking in from the shell.
Build outputs are content-addressed: the output of a build step is named by a cryptographic hash of its inputs. If you build the same target twice with the same inputs, you get the same hash and therefore the same output. This is what makes caching work correctly. If someone else on your team (or a previous CI run) already built a target with the same inputs, Bazel can pull the cached output from a remote cache rather than rebuilding. Same inputs, same hash, same output: the cache hit is safe.
The hermeticity is the prerequisite for the caching, and the caching is why large Bazel builds can be fast despite running across enormous codebases. In a well-configured Bazel setup, most CI runs are fast because most build steps hit the cache. The build is effectively incremental even from a fresh checkout.
This also enables remote execution: because every build step has fully declared inputs and outputs, it can be executed on any machine in a cluster. Steps that don't depend on each other can run in parallel across hundreds of workers. The build graph is the parallelism plan.
The tradeoff is configuration overhead. Bazel requires you to write explicit BUILD files that declare every target, its inputs, and its dependencies. Migrating an existing codebase to Bazel is a significant effort. For many teams, this cost is not justified, which is where the middle path comes in.
The middle path
You don't need Bazel to meaningfully improve build hermeticity. Most of the benefit comes from disciplined application of a few practices.
Commit lockfiles. For every language ecosystem that generates one
(package-lock.json for Node, Pipfile.lock or
poetry.lock for Python, go.sum for Go, Cargo.lock
for Rust), commit it to the repository and treat it as load-bearing. Run installs against
the lockfile, not against a version range. The lockfile pins the full resolved dependency
graph including transitive dependencies.
Pin base image digests in Dockerfiles. Use @sha256:<digest> rather
than a tag. Automate digest updates on a schedule and review the resulting diff rather
than letting it vary silently during builds.
Manage tool versions explicitly. A .tool-versions file (used by
asdf) or a .nvmrc (for Node version) or a Nix flake pins the
version of runtime tools. CI and local development use the same versions. "It worked on
my machine" becomes "we can check exactly what version of the tool you were using."
Containerize build steps. A build that runs inside a container with a pinned base image has an implicit isolation layer. The host machine's installed software doesn't bleed into the build environment.
These practices don't give you Bazel's remote caching or remote execution, but they eliminate most of the sources of build non-determinism that cause flaky builds in practice. The cost is low; the benefit is real.
Reproducible builds as a security property
Hermetic builds have a security implication that's worth naming explicitly.
If the same source code always produces the same artifact, then you can verify that the binary you're running matches the source it claims to be compiled from. Build it yourself, hash the result, compare to the hash of the distributed artifact. If they match, the artifact is what it claims to be. If they don't, something happened between source and binary: either a non-hermetic build input changed, or the build was tampered with.
This is what the reproducible-builds.org project is working toward. The goal is that any party can independently verify the correspondence between source and binary. This is a meaningful security guarantee: a compromised build server that injects malicious code into artifacts would produce artifacts that don't match independent builds.
If your builds aren't reproducible, you have a weaker guarantee about what you're actually deploying. The binary came from the CI pipeline, but you can't independently verify that the CI pipeline faithfully compiled the source. For most software this is an accepted risk. For high-stakes software (infrastructure tooling, security software, anything that runs with elevated privilege), it's worth taking seriously.
Non-hermetic builds aren't just an inconvenience that causes CI reruns. They're a measurement problem. When your build environment is not fully specified, you don't know exactly what you built. You know approximately what you built, under approximately these conditions, on approximately this machine, with approximately these dependencies. "Approximately" is a bad property for software you're deploying to production.
"It worked on my machine" is not a mystery. It's a build system problem. Two environments diverged somewhere because a non-hermetic input varied between them. That input can be found, pinned, and eliminated as a source of variance. The work is tedious. It's not glamorous. It pays for itself many times over in CI time that stops disappearing into reruns and investigations that lead nowhere.
Back to writing