You build an async system. It handles ten times the throughput of the synchronous version. Engineers are pleased. You ship it. Then one day upstream sends a spike: a marketing email goes out, a cron job fires at the wrong moment, a partner starts hammering your API. Your queue fills up, memory explodes, and everything falls over at once. The queue absorbed the spike, all right. It just did it by buffering every message until there was no memory left.
You had no backpressure. This is a design failure, not a load failure, and it's one of the most common ones in async systems.
What backpressure is
Backpressure is the ability of a downstream consumer to signal to an upstream producer: slow down. Stop sending. Wait until I've caught up.
In a synchronous system, this happens automatically. If service A calls service B and B is slow, A blocks waiting for B's response. The blocking is annoying but it's also a built-in rate limiter: A can't send work to B faster than B can process it, because A is sitting there waiting. The slowness propagates backward through the call stack. Backpressure is implicit.
In an async system, the producer doesn't wait. It puts a message on the queue and moves on. This is the source of the performance gain, decoupling producer speed from consumer speed, and also the source of the hazard. The decoupling that makes the system fast is the same decoupling that removes the natural speed limit. Backpressure has to be built back in deliberately, because it was removed deliberately.
Why queues without backpressure are dangerous
An unbounded queue is unbounded memory. There's no other way to read it. If your consumer is processing at 1,000 messages per second and your producer is sending at 2,000 per second, the queue grows at 1,000 messages per second. At some point you run out of memory and crash.
The delay is what makes it worse, not better. In a synchronous system under load, requests start failing immediately: error rate goes up, engineers get paged, the problem is visible. In an async system with an unbounded queue, messages keep getting accepted, the queue keeps growing, and everything looks fine from the producer's perspective. By the time you crash, you've accepted millions of messages you'll never process. The queue that was supposed to buffer the spike has become a mechanism for accepting work you can't do and delaying the failure until it's as large as possible.
There's a secondary effect: tail latency. A message enqueued when the queue has 10 million items in it will wait in queue for however long it takes to drain those 10 million items. If your consumer processes at 1,000 messages per second, that's 10,000 seconds of queue wait, nearly 3 hours. The message is processed eventually, but "eventually" has become useless. The user who submitted that request is long gone.
How backpressure propagates
In synchronous systems, backpressure propagates through blocking. In async systems, you have to build the propagation mechanism yourself, and the mechanism looks different depending on where in the stack you're working.
TCP flow control is the canonical example of built-in backpressure. The receiver advertises a receive window, the amount of data it's willing to buffer. The sender cannot send more than the window allows. When the receiver's buffer fills up, the window shrinks to zero; the sender stops. When the receiver drains its buffer, the window opens again. The sender is directly rate-limited by the receiver's processing speed, through a mechanism built into the protocol. You get this for free if you're using TCP, which is why network engineers rarely have to think about it.
Reactive Streams, the specification underlying libraries like RxJava, Project Reactor, and Akka Streams, formalizes backpressure for async data processing. The core model: consumers request N items from an upstream producer, the producer sends at most N items, and the consumer requests more when it's ready. The producer can never run ahead of what the consumer has asked for. If you use a Reactive Streams-compatible library correctly, backpressure is built into every operator in your pipeline.
For Kafka-based systems, consumer lag is the signal. If your consumer group is falling behind, the difference between the latest offset and the committed offset is growing, that's the equivalent of a full receive window. The appropriate response is to scale consumers, throttle producers, or both. Kafka itself doesn't throttle producers based on consumer lag by default; you have to build that feedback loop externally, typically by monitoring lag metrics and triggering autoscaling or alerting when lag exceeds a threshold.
The patterns
Bounded queues with blocking producers are the simplest fix. Instead of an unbounded queue, you set a capacity, say 10,000 messages. When the queue is full, the producer blocks until space is available. This re-introduces synchronous behavior at the queue boundary, which is the point: you've put a ceiling on how far ahead the producer can run. The queue absorbs short spikes; sustained overload is reflected back to the producer as latency. The producer slows down. Its upstream slows down. The backpressure propagates.
The capacity to set depends on your latency budget and your message size. A queue of 10,000 messages that takes 10 seconds to drain means any message enqueued when the queue is nearly full will wait up to 10 seconds before processing. Set it large enough to absorb real spikes; small enough that the added latency at capacity is still within your SLA.
Load shedding is the harder alternative. Instead of blocking, you drop messages when you're over capacity. This sounds like a failure mode; it's actually a design choice. If your queue is full, you have two options: accept the message and eventually crash, or reject it now and let the caller handle the failure. Rejection is better. The caller can retry later, apply its own backoff, or report an error to the user. A crash gives the caller nothing except a timeout.
Load shedding works best at the edge, at the first point of entry to your system, before work has been done. Shedding a message at the edge is cheap: you've done nothing yet, and the caller can be informed immediately. Shedding work in the middle of a pipeline, after you've already done partial processing, is wasteful and complicated. The further upstream you can put your capacity signal, the better.
Circuit breakers complement load shedding. A circuit breaker monitors failure rates or latency for a downstream dependency. When the rate exceeds a threshold, the circuit "opens": subsequent calls to that dependency fail immediately without attempting the call. This prevents a slow or failed downstream service from blocking all your workers. After a timeout, the circuit enters a half-open state and lets a small number of calls through to test recovery. If they succeed, the circuit closes; if not, it stays open.
Circuit breakers protect the system from a bad downstream dependency. Bounded queues and load shedding protect it from too much upstream demand. You typically need both.
The asymmetry
There's a cognitive asymmetry here that explains why backpressure gets skipped. The benefits of an async system are immediate and visible: lower latency, higher throughput, decoupled deployment. The costs of missing backpressure are deferred and only visible under load. If you never hit the capacity limit in testing or staging, you ship without backpressure and everything is fine until it isn't.
Treat backpressure as a required feature, not an optimization. Before shipping an async component, answer two questions: what is the maximum size of my queue, and what happens when it's full? If the answer to the second question is "undefined" or "the producer keeps going," you have work to do.
An async system that can't push back on its producer isn't more resilient than a synchronous one. It's just slower to fail, and bigger when it does.
Back to writing