In a single-process system, a transaction either commits or rolls back. You write three rows, the commit succeeds, all three are visible. Or it fails, none of them are visible. The database handles the uncertainty; your application sees a clean binary outcome.
In a distributed system, you can have a transaction that neither commits nor rolls back. It just sits there, in an unknown state, while your application watches a timeout tick down and tries to decide what to do next. This is not a theoretical failure mode. It happens every time a coordinator crashes at the wrong moment, which is often enough to build around.
Two-phase commit: the obvious solution
Two-phase commit (2PC) is the canonical algorithm for distributed atomic transactions. The setup: one node acts as the coordinator, the rest are participants. The coordinator wants to commit a transaction that touches all participants.
Phase one is the prepare phase. The coordinator sends a "prepare" message to every participant. Each participant writes the transaction to durable storage (so it can commit or abort later, even after a crash), then replies "yes" if it's ready to commit, "no" if it can't.
Phase two is the commit phase. If everyone voted yes, the coordinator writes "committed" to its own durable log and sends "commit" to all participants. If anyone voted no, it sends "abort."
This works fine until the coordinator crashes between writing "committed" to its own log and sending "commit" to the participants. Some participants have committed. Others haven't. The coordinator is down. The participants who voted "yes" are blocked: they prepared, which means they're holding locks, and they can't proceed without hearing from the coordinator. They're waiting for a node that isn't coming back.
You can make the coordinator highly available with replication, which helps. You can time out and resolve the uncertainty by querying other participants about their outcome. But there are scenarios, specifically when the coordinator fails after some participants committed and before any of them can figure out what the coordinator decided, where 2PC is genuinely stuck. This is why 2PC is called a "blocking" protocol.
Why 2PC is avoided at scale
The blocking behavior is the practical problem. Participants hold locks during the prepare phase. If the coordinator goes down for 100ms, a routine hiccup in any real system, every participant involved in any in-flight transaction is locked, unable to process new writes to those rows, waiting for the coordinator to come back.
In a system handling thousands of transactions per second, a 100ms coordinator pause doesn't mean 100ms of latency on affected transactions. It means those transactions queue up, then time out, then retry, multiplying the work. A small coordinator outage can cause a cascading backlog that takes minutes to drain. This is roughly why most modern distributed systems avoid 2PC for hot paths. Google Spanner does offer atomic cross-shard transactions, but it puts enormous engineering effort into making coordinators reliable and making the protocol non-blocking when possible.
The saga pattern
The dominant alternative for long-running transactions in microservice architectures is the saga pattern. The idea: instead of one atomic transaction spanning multiple services, you model the business process as a sequence of smaller transactions, each entirely local to one service, with a compensating action for each step that undoes its effect if something later fails.
The classic example: book a flight, reserve a hotel, charge the card. As a distributed transaction this is painful: three services, three databases, 2PC over all of them. As a saga:
- Call flight service → book flight. Compensating action: cancel flight.
- Call hotel service → reserve room. Compensating action: cancel reservation.
- Call payment service → charge card. Compensating action: issue refund.
If step 3 fails, the saga runs the compensating actions for steps 2 and 1 in reverse order: cancel the hotel reservation, cancel the flight. The overall effect is a rollback, achieved through forward-running compensating transactions rather than a database rollback.
This removes the coordinator-lock problem. Each step is a normal, local transaction. There's no global lock held across services. The saga can be orchestrated (one service calls each step in sequence) or choreographed (each service emits an event when done, and the next service listens for it). Both approaches work; orchestration is easier to reason about, choreography is easier to scale.
What sagas don't give you
What sagas give up is isolation. In a classic ACID transaction, concurrent transactions can't see each other's intermediate state. In a saga, every intermediate state is committed to the database. If two sagas run concurrently, they can read each other's in-progress data.
Consider two users both trying to book the last hotel room. Saga A reads the room as available and reserves it. Saga B, running concurrently, also reads the room as available. Saga A has reserved it but not yet confirmed payment, so Saga B also tries to reserve it. Depending on how your reservation logic works, you might double-book.
Real systems deal with this through semantic locks (mark the resource as "pending" rather than "available" as the first step), pessimistic ordering (use a queue to serialize reservations for contested resources), or by accepting the anomaly and resolving it after the fact. None of these are free; they each add complexity and either add latency or accept occasional incorrect intermediate states.
Sagas are also hard to test for failure modes. It's easy to test the happy path. The compensating actions are the interesting part, and they run only when something has already gone wrong, which means they're the least-exercised code in the system and the code most likely to have bugs.
The outbox pattern
A specific problem worth calling out separately: atomically writing to a database and publishing a message to a queue. This comes up constantly in event-driven architectures. You process an order, write to your orders table, and publish an "order created" event to Kafka. If the write succeeds but the publish fails, you have a processed order with no event. If the publish succeeds but the write fails and you retry, you might publish the event twice.
The outbox pattern solves this without distributed transactions. Instead of publishing to Kafka directly, you write the event to an outbox table in the same database transaction as the order. A separate process (sometimes called a relay or a CDC consumer) watches the outbox table, often using change-data-capture on the database's write-ahead log, and publishes events to Kafka as they appear.
The atomicity guarantee comes from the database: either both the order row and the outbox row are committed, or neither is. The relay may publish the same event more than once (if it crashes after publishing but before marking the event as processed), which means your consumers need to be idempotent. But you've eliminated the "event published but order not written" failure entirely, which is usually the worse of the two failure modes.
No free lunch
Two-phase commit gives you atomicity and isolation, at the cost of blocking availability when coordinators fail. Sagas give you availability and non-blocking progress, at the cost of isolation: concurrent sagas see each other's intermediate states. The outbox pattern gives you atomic write-and-publish, at the cost of operational complexity (you need the relay process, you need idempotent consumers) and some delivery latency.
These aren't bugs in the patterns; they're the fundamental tradeoffs. The question to ask about any distributed system isn't "does it handle failures gracefully?" Every system claims to. The question is: which guarantee did you trade away, and does your application handle the cases where that guarantee doesn't hold?
Back to writing