CI/CD as Production Infrastructure

Your production services have SLOs, runbooks, on-call rotations, and postmortems when they break. Your CI pipeline takes forty-five minutes, nobody knows exactly why, it fails about three percent of the time for reasons unrelated to the code, and when it breaks badly enough the fastest fix is to push with --no-verify and figure it out later. The deploy frequency is limited partly by how long CI takes and partly by how many engineers are willing to wait in the queue on a given day.

This asymmetry is not primarily a tooling problem. It's a reliability culture problem. Production services get engineering investment because someone is paged when they degrade. CI pipelines degrade silently, and the cost is diffuse enough that no single engineer feels it sharply enough to own it.

The ownership gap

Production services have owners. When the checkout service degrades, someone's pager goes off. That someone has the context to investigate, the authority to prioritize the fix, and the accountability to prevent recurrence. The ownership model is imperfect but functional: degradation triggers a response, and the response is handled by the person best positioned to handle it.

CI pipelines typically have whoever isn't too busy. When CI is slow or flaky, the common response is to file a ticket in a backlog, get it bumped when something more urgent appears, and wait until the situation is bad enough that someone treats it as an emergency. The pipeline accumulates technical debt through ordinary operation: a test gets added that calls an external API, a build step gets a hardcoded environment assumption, a new service gets wired in without updating the dependency graph. Each individual addition is reasonable in isolation. Together they add three minutes to every build and two flaky failures per hundred runs.

The fix for the ownership gap is the same as the fix for any reliability problem: assign explicit ownership, define what healthy looks like, and make degradation visible. A platform engineering team or build systems team with a stated SLO for CI reliability and a budget to invest in improvements will produce better outcomes than a shared responsibility that everyone assumes someone else is handling.

Flaky tests as SLO violations

A test that fails five percent of the time for non-deterministic reasons is not a minor nuisance. It's a reliability bug. You would not tolerate a service that returned errors five percent of the time and then retried successfully on the second attempt. You would instrument it, investigate it, and fix it.

Apply the same standard to tests. A flaky test has a flakiness rate, the same way a service has an error rate. It can be measured. It should be tracked. And it should be subject to a budget: tests that exceed a flakiness threshold get quarantined, disabled, with a bug filed and an owner assigned. They don't stay in the main suite blocking deploys and eroding trust in CI signal.

The mechanism by which flakiness erodes trust is worth spelling out. When builds fail non-deterministically, engineers learn to pattern-match on failure types. "Oh, that's the Elasticsearch connection test, just rerun." Over time, the rerun becomes the reflex for any failure. A genuine regression gets dismissed as probably-flaky, rerun until green, and shipped. The test suite hasn't caught anything because nobody trusts what it's saying anymore. The suite exists but the signal doesn't.

Common sources of test flakiness: tests that share mutable state (database rows, file system entries, in-memory caches) and depend on execution order. Tests that call external services and fail when those services are slow or unavailable. Tests with time-sensitive assertions (assert event_occurred_within(100ms)) that fail on loaded CI machines. Tests that don't properly clean up after themselves, causing interference with later tests. Each of these is fixable. None of them fix themselves.

Build time as a latency metric

A slow CI pipeline has a cost that can be calculated. Developer wait time multiplied by the number of engineers multiplied by the number of builds per engineer per day. A forty-five minute CI pipeline at a company of two hundred engineers running three builds per day is four hundred and fifty engineer-hours per day, assuming every engineer blocks on CI the full forty-five minutes. The real number is lower, because engineers context-switch rather than staring at the CI dashboard, but the context-switching has its own cost.

The less obvious cost is what slow CI does to deploy frequency. A team that can only deploy twice a day because CI takes too long to run more builds is a team with large batches. Large batches mean large blast radius per deploy. When something breaks, the diff is larger, the debugging is harder, and the rollback surface is wider. The operational risk of each deploy goes up. Teams respond rationally by deploying less frequently, which makes batches larger, which makes each deploy riskier, which further reduces deploy confidence. The CI slowness isn't just overhead. It's compounding.

"We can only deploy twice a week" is almost always a policy that developed for reasons that now include, somewhere in the causal chain, a CI pipeline that made more frequent deploys impractical. Fix the CI pipeline and the policy often becomes revisable.

Treat build time as a percentile metric with a budget. P95 build time for main branch should be under some threshold. Measure it. Graph it over time. When it degrades, investigate the root cause rather than accepting the new normal.

Non-hermetic CI environments

Many sources of build flakiness trace back to CI environments that aren't fully isolated. A test that passes locally and fails in CI (or vice versa) is pointing at an environmental difference. Something in the execution environment is different, and that something is an undeclared input.

Tests that depend on execution order do so because they share state that isn't reset between tests. Tests that call external APIs fail non-deterministically because external APIs have their own availability and latency characteristics that the test doesn't control. Tests that depend on system time without controlling it fail on boundary conditions: the test was written at 11pm and the assertion breaks at midnight because the developer didn't consider that CI might run it at a different time. Environment-specific behavior usually traces back to a tool version difference or a platform difference between local development and CI.

The fix in each case is to move the varying input from implicit to explicit and then control it. Shared state should be reset between tests. External APIs should be mocked or stubbed in unit and integration tests (and tested against the real API in a separate end-to-end suite that can tolerate occasional failures). System time should be injectable so tests can control it. Tool versions should be pinned and consistent between local development and CI.

Fully hermetic CI environments, where every input is declared and fixed, produce deterministic results. The same code either always passes or always fails. When it fails, the failure is real and warrants investigation. When it passes, you have genuine confidence in the result. This is what a test suite is for.

Incremental computation

Running every test on every commit is the right default at small scale. At large scale, it's why CI is slow. A change to the billing service doesn't affect the authentication service's tests. Running them anyway is waste: compute spent on tests that cannot possibly catch a regression in the changed code.

Test impact analysis determines which tests are affected by a given change by tracing the dependency graph from the changed files to the tests that depend on them. Only affected tests run. A one-line change to a leaf-node utility function triggers a handful of tests rather than the full suite. Changes to a core shared library trigger everything that depends on it, which might still be the full suite, but that's correct: a core library change has a wide impact surface.

Build-level incrementality works at a coarser granularity: only rebuild and retest the build targets affected by the diff. Bazel, Nx, and Turborepo all approach this by tracking the dependency graph at the build target level and content-addressing build outputs so that unchanged targets can be restored from cache rather than rebuilt. A large Bazel monorepo with a warm remote cache can run CI in minutes for most changes, because most targets were already built by a previous run with identical inputs.

Some CI systems address this at the scheduling level rather than the build system level. Test splitting distributes a test suite across parallel workers to reduce wall clock time. Flaky test quarantine prevents known-flaky tests from blocking builds. Neither of these substitutes for build-level incrementality, but they're lower-effort improvements that are available with most CI platforms today.

Security: the CI supply chain

CI systems have access to production secrets. Signing certificates. Deploy credentials. API keys for production infrastructure. The CI pipeline, by definition, runs code: the build scripts, the test suite, any pre-deploy checks. In an open-source project with external contributors, this means CI runs code from pull requests submitted by people who are not employees and who have not been vetted beyond their GitHub account.

The attack surface here is real and has been exploited. A malicious pull request that adds a line to a CI script can exfiltrate secrets from the environment. Injected code in a build step can tamper with build artifacts before they're published. A compromised dependency that runs during tests can pivot to production infrastructure using credentials that CI makes available.

The mitigations follow from least-privilege principles. Separate pipelines for pull requests from external contributors (no production secrets) and for code that has been merged to main (full access). Don't make production credentials available until code has been reviewed and merged by a trusted team member. Use short-lived credentials rather than long-lived API keys. OIDC-based authentication with cloud providers lets CI jobs request tokens scoped to specific operations rather than holding static credentials. Sign build artifacts so you can verify that a given binary was produced from a given source by a trusted build system, not modified after the fact.

CI is a privileged position in the software delivery chain. It sits between source code and production. Every decision about what CI can access, what code it runs, and under what conditions it runs is a security decision, whether you're thinking of it that way or not.

Your CI pipeline ships code to production. It's the last automated gate before your software runs on real machines serving real users. If it's slow, that slowness compounds into reduced deploy frequency and larger blast radii. If it's flaky, that flakiness erodes trust in the test signal until the suite is theater rather than verification. If it's insecure, that exposure is a high-value target. It deserves ownership, SLOs, flakiness tracking, and performance budgets: the same reliability engineering attention as the code it ships. If it's nobody's job to keep it healthy, it won't be.