Your user clicks "Pay." The request reaches your server. Your server calls the payment processor. The response times out. Did the payment go through? You have no idea. What you do next will determine whether the user gets charged once, twice, or not at all.
If you don't retry, you might leave the user without their order even though they were charged. If you do retry, you might charge them twice. Both outcomes are bad. "Be careful about retries" is not a policy. It's anxiety dressed up as caution. The correct answer is a design pattern that makes the question moot.
The double-charge problem
Networks time out. Servers restart. Load balancers reroute requests mid-flight. A payment system that doesn't retry is one that fails visibly and often. A payment system that retries without care occasionally charges customers twice, and "occasionally" is a lot when you're processing thousands of transactions per day.
The underlying problem is that a timed-out request is ambiguous. From the client's perspective, the operation might have succeeded (the server processed it but the response was lost), or it might have failed (the server never received it, or crashed midway through). You can't tell which. And if you can't tell, retrying is a coin flip that costs your customer money if it comes up heads.
Never retry and always retry are both wrong. What you want is to retry safely: make it so that retrying the exact same request has no additional effect. That property has a name.
What idempotency means
Idempotency is the property where applying the same operation multiple times has the same effect as applying it once. Formally: f(f(x)) = f(x). The concrete version: "charge $50 to this card" should produce one charge of $50, regardless of how many times you call it.
Some operations are naturally idempotent. Setting a value (as opposed to incrementing it) is idempotent: setting a field to "confirmed" ten times leaves it at "confirmed." A DELETE in SQL is idempotent: deleting a row that's already gone is fine. An upsert (INSERT ... ON CONFLICT UPDATE) is idempotent if the values you're setting are deterministic.
Charging a card is not naturally idempotent. The payment processor has no way to know, without additional information, whether two identical requests are the same logical operation or two separate charges for the same amount. It will happily process both. Making it idempotent requires that additional information and a mechanism to use it.
Idempotency keys
The standard implementation is called an idempotency key. Before making the payment request, the client generates a unique identifier (typically a UUID) for this specific logical operation. It sends this key along with every attempt to execute that operation. The server maintains a mapping from key to result.
On first receipt of a key, the server executes the operation, stores the result keyed on the idempotency key, and returns it to the client. On subsequent receipts of the same key, it skips the operation entirely and returns the stored result. From the client's perspective, every call returned the same thing. From the server's perspective, the operation ran exactly once.
Stripe popularized this pattern, and most payment APIs implement some version of it.
The Stripe API accepts an Idempotency-Key header on any POST request.
If you send the same key twice, the second call returns the cached response from the
first. The implementation is their problem. Using the feature correctly is yours.
The implementation subtleties
If you're building the server side of this, the details matter.
Store the key atomically with the operation. The idempotency key and the operation result must commit to the database in the same transaction. If you store the key after the operation, there's a window where the operation succeeded but the key wasn't recorded. A retry would re-execute the operation. If you store the key before the operation, there's a window where the key was recorded but the operation never ran. A retry would return a cached failure that never actually failed. One transaction, one commit.
Store failures too. If the payment processor declines the card, record that outcome against the idempotency key. A retry of a failed operation should return the same failure, not re-execute the request. The exception is transient infrastructure failures (your database was down, you couldn't reach the processor at all): those haven't produced a result to store, so retry is appropriate.
Handle race conditions with database constraints. If two retries of the same key arrive simultaneously, before either has been processed, both will find no stored result and attempt to execute. The fix is a unique constraint on the idempotency key column. The database will reject one of the two inserts, and the losing request can wait briefly and then return the result stored by the winner. Handling this in application code with locks or checks is slower and less reliable.
Set an expiry window. Typical retention ranges from 24 hours to 7 days, calibrated to the retry window of your clients. If your retry logic gives up after an hour, keeping keys for a week is conservative but reasonable. Keeping them for a year is a storage cost for no benefit.
Natural vs. designed idempotency
Upserts are naturally idempotent when the values are deterministic. If you're setting a user's email address, an upsert is fine: calling it ten times leaves the email at the right value. Deletes are naturally idempotent too. The row is gone whether you delete it once or five times. Reads are idempotent in the trivial sense (they don't change state).
Appending to a list is not idempotent. Incrementing a counter is not idempotent. Sending an email is not idempotent. Charging a card is not idempotent. For these, you either wrap them in the idempotency key mechanism, or you redesign them. Sometimes redesigning is cleaner: instead of "send a welcome email on registration," you can have a flag in the database indicating whether the welcome email was sent, and check that flag before sending. The operation becomes "set sent_welcome = true and send the email," which is idempotent via the flag.
Redesigning for idempotency tends to surface implicit state that was always there but never made explicit. That's usually a good thing.
What keys don't solve
Idempotency keys handle the case where the same logical operation is retried. They don't handle the case where a user clicks "Pay" twice, quickly, before the first response comes back. That's a different problem (duplicate submission) and it requires client-side debouncing (disable the button after the first click), a separate pending-payment record, or both.
Keys also don't help if you meant to make two separate charges for the same amount to the same card. If you generate the same UUID twice (UUID collisions are not literally impossible, just vanishingly unlikely), one charge will be silently dropped. Use a UUID library that generates properly random v4 UUIDs and this isn't a practical concern. But it's the assumption the whole thing rests on.
The mechanism also requires both sides to participate. If the payment processor you're integrating with doesn't support idempotency keys, you're working without a net. Some older processors don't. In those cases, you need your own deduplication at a higher level: typically checking for a recent charge of the same amount to the same card before submitting, with appropriate timing windows. Messier, but it's the fallback.
The pattern generalizes
Idempotency keys are most visible in payment systems, but the pattern appears anywhere you have distributed state and unreliable networks, which is everywhere. Email delivery APIs use it. Job queues implement it under the name "exactly-once delivery" (that framing is slightly misleading: the delivery might happen multiple times, but the effect happens once). Cloud infrastructure APIs use it: "create this virtual machine" should be safe to retry.
The pattern is also one half of a broader principle: at-least-once delivery plus idempotent consumers equals effectively-exactly-once behavior. You give up on the hard guarantee (only deliver once, ever, under all failure conditions, which is extremely difficult to implement correctly) and instead make duplicate delivery harmless.
An idempotency key is a promise: "I've seen this exact request before, and here's what happened." Without it, retries are gambles. Every one is a spin on the double-charge wheel. With it, retries are boring, which is exactly what you want financial infrastructure to be.
Back to writing