"Eventual consistency" sounds like a database property. Engineers talk about it the way they talk about replication factors and storage backends: something the infrastructure team configured, something that happens below the application layer. It's not. It's a contract your application code has to uphold, and most applications don't.
The database's job is narrow: if you stop writing, all replicas will eventually hold the same data. That's the whole promise. Everything outside that, what your application does while the replicas are still catching up, is your problem.
What "eventually" actually means
Eventually is doing a lot of work in that sentence. Under normal conditions, replication lag in a system like DynamoDB or Cassandra might be single-digit milliseconds. When things are stressed, a node recovering from failure, a network hiccup, a backup process monopolizing I/O, lag can stretch to seconds or minutes.
The database doesn't know which situation you're in. It doesn't expose a "current lag" that your application can check before deciding whether a read is safe. You get a value back and you have no way to know if it's the value that was written 5ms ago or the value that was written 3 minutes ago before a node went down.
This is fine in the sense that it's a documented behavior, not a bug. But it means the application has to be written with this uncertainty in mind. Most applications aren't.
The cases your app probably doesn't handle
The classic failure is reading your own write. You write a record, the write goes to replica A. You immediately issue a read, which happens to hit replica B, which hasn't received the write yet. You get the old value. Your application shows the user a form they just submitted as if they never submitted it.
Some databases offer "read-your-own-writes" consistency as a feature, where the client library routes reads to the same replica you just wrote to, or uses a monotonic token to ensure you don't read older data. If yours doesn't, and you care about this, you need to handle it explicitly: sticky sessions, read-after-write tokens, or simply accepting the inconsistency and designing the UI around it.
A subtler case: two users editing the same record concurrently. User A reads a document, edits it, writes it back. User B read the same document 200ms earlier, also edits it, also writes it back. Depending on timing and which replicas each write hits, one of the edits silently disappears. Last-write-wins at the storage layer is not the same as last-write-wins from the user's perspective, because "last" is determined by the database's clock, not by which edit actually happened last in real time.
Then there's the distributed transaction problem in miniature. A payment succeeds, the charge goes through, but the confirmation message fails to deliver because the service handling confirmations was momentarily unavailable. The database has consistent state: the payment is recorded. But the user sees nothing and re-submits. You've now charged them twice. The database was consistent throughout. Your application was not.
What makes operations safe
Certain operations survive eventual consistency without breaking because they have mathematical properties that make ordering irrelevant.
Commutativity means the order of operations doesn't affect the result. Adding to a counter is commutative: increment by 3 then increment by 5 gives the same result as increment by 5 then increment by 3. Adding an element to a set is commutative. These operations don't care which replica processes them first. Once they've all been applied everywhere, the result is the same.
Idempotency means applying an operation twice gives the same result as applying it once. An upsert (insert if not present, ignore if already there) is idempotent. Deleting a record is idempotent; deleting something that's already deleted is a no-op. These operations survive retries safely. If a write was actually applied but the acknowledgement was lost, applying it again doesn't double-count or corrupt anything.
If your operation is both commutative and idempotent, you can apply it as many times as you want, in any order, and end up with the right answer. Designing operations to have these properties is one of the most useful things you can do when building on an eventually consistent store.
When you can't make it commutative
Sometimes the operation is inherently order-sensitive. "Set this record to the value the user just typed" is not commutative. Two concurrent updates will conflict, and there's no mathematically correct answer. It depends on which update the user intended to be final.
The simplest resolution is last-write-wins: whichever write has the later timestamp survives. This works when writes are truly independent and losing one is acceptable. It fails when clocks are skewed (the "later" timestamp might be physically earlier), and it fails when the user would be surprised to find their edit silently discarded.
Vector clocks give each write a version vector, a list of counters one per replica, that lets you detect whether two writes are causally related or concurrent. If write B's vector clock is strictly greater than write A's, B came after A and A can be discarded. If neither dominates the other, they're concurrent and you have a conflict that requires resolution. DynamoDB uses a variation of this. The downside: you still have to resolve the conflict somehow, and that logic lives in your application.
CRDTs (Conflict-free Replicated Data Types) are data structures designed to merge concurrent updates without conflicts. A grow-only set can only have elements added, never removed; merging two replicas is just a union. More sophisticated CRDTs handle removes, ordered sequences, and maps. The tradeoff is that you have to fit your problem into the CRDT's constraints. Not everything fits neatly, and the data structures can become expensive to synchronize at large scale.
The contract
When you pick an eventually consistent database, you're accepting a deal: the database will handle replication and convergence; you will handle the interval before convergence arrives. The database is not being negligent. It's being honest about the performance tradeoffs it's making, and handing you the responsibility for correctness above its layer.
The applications that break on eventual consistency are the ones that were designed as if the database were strongly consistent, as if every read would reflect every preceding write, and just happened to use an eventually consistent store because it was fast or cheap or the default. The database kept its half of the bargain. Nobody held up the application's half.
The fix isn't to switch to a strongly consistent database (though sometimes that's right). It's to identify which operations need to be commutative and idempotent, to decide explicitly what happens during conflicts, and to test the application under conditions where reads are stale. The bugs are there whether or not you look for them. Looking is better.
Back to writing