"Exactly-once processing" is one of the most requested features in streaming systems. It's also, in the way most people mean it, a lie, or at least a compression of the truth that causes real bugs when you take it literally.
The claim usually sounds like: "Kafka supports exactly-once semantics, so each message is processed exactly once." This is approximately true in a narrow technical sense. It's misleading in any practical sense, because the guarantee doesn't extend past the boundary of the Kafka ecosystem. The moment your consumer writes to an HTTP API or an external database that isn't part of Kafka's transaction, you're back to something weaker.
At-least-once and at-most-once
At-most-once means a message is delivered zero or one times. The implementation is simple: fire and forget. The producer sends the message and doesn't wait for an acknowledgement. If delivery fails, the message is lost. This is acceptable for metrics, log lines, or any data where a gap in the record is survivable.
At-least-once means a message is delivered one or more times. The producer retries until it gets an acknowledgement; the consumer commits its offset only after successfully processing the message. If the consumer crashes between processing and committing, it'll re-read and reprocess the message after recovering. You never lose messages, but you may process them twice. This is the default and the sensible baseline for most systems.
Which failure mode is worse depends on the domain. For financial transactions, processing twice is catastrophic, and losing one probably is too, so you'll want the strongest guarantees available. For an email notification system, sending twice is annoying; not sending at all might be worse. For updating a view cache, processing twice with an idempotent write is completely harmless. Know your domain before reaching for the strongest guarantee.
What "exactly-once" actually means
The key distinction is between delivery and effects. Exactly-once delivery, meaning a message arrives at the consumer exactly once full stop, is impossible to guarantee in an asynchronous distributed system. Networks drop packets; processes crash; acknowledgements get lost. Any system that retries must accept the possibility of duplicates.
Exactly-once semantics is achievable: the effect of processing a message happens exactly once, even if the message is delivered multiple times. The message might arrive twice, but if processing is idempotent, the second arrival has no effect. Or you commit the result transactionally together with the acknowledgement, so there's no window where the message is processed but not committed.
These are very different claims. One is about counting message arrivals (not achievable in general). The other is about the semantics of what your system does in response (achievable with careful design).
Kafka's approach
Kafka achieves exactly-once semantics through three mechanisms. Understanding each one separately helps you see where the guarantee applies and where it doesn't.
Idempotent producers: when idempotent mode is enabled, Kafka assigns each producer a unique ID and attaches a monotonically increasing sequence number to each message. If a producer retries a message after a failed acknowledgement, the broker checks the sequence number and deduplicates. If the sequence was already received, the broker discards the duplicate instead of writing it again. This gives you exactly-once delivery from producer to broker, within a single partition.
Transactional API: producers can open a transaction that spans multiple partitions and topics. Either all the writes in the transaction commit atomically, or none do. This lets a consumer-processor-producer pipeline (read message, transform it, write result to another topic) commit the input offset and the output message as a single atomic operation, with no window where the output is written but the input offset isn't committed.
Consumer isolation: consumers can be configured with isolation level
read_committed, which means they only see messages from committed
transactions. A message written inside a transaction that later aborts is
never visible to committed-isolation consumers. This prevents half-written
transaction data from leaking downstream.
Together, these give you exactly-once semantics for a pipeline that lives entirely inside Kafka: produce to Kafka, transform within Kafka, consume from Kafka. The guarantee holds.
The gotcha: end-to-end
The guarantee breaks the moment your pipeline crosses the Kafka boundary into a non-transactional sink.
Your consumer reads a message, processes it, and writes the result to a PostgreSQL table. Between the write to PostgreSQL and the commit of the Kafka offset, the consumer crashes. On recovery, it re-reads the message, reprocesses it, and writes to PostgreSQL again. PostgreSQL has no idea this is a duplicate. You're at-least-once.
The same applies to any sink that doesn't participate in Kafka's transaction protocol: HTTP APIs, Redis, external databases that don't support two-phase commit with Kafka. Kafka's guarantee doesn't extend to them, and there's no way to make it extend to them without the sink supporting a compatible transactional mechanism.
This is not a failure of Kafka. It's a fundamental constraint of distributed systems: you can only have atomic, exactly-once semantics across systems that share a transaction protocol. If your sink is outside that boundary, you need a different strategy.
Practical guidance
Design your consumer to be idempotent, and treat the delivery guarantee as at-least-once regardless of what Kafka promises.
An idempotent consumer on an at-least-once pipeline is functionally equivalent to exactly-once, with no transactional overhead beyond the idempotency mechanism itself. Include a deduplication key in every write. For database writes, use upserts keyed on the message ID. For external APIs, check if the operation already succeeded before retrying. Record processed message IDs in a durable store with a TTL that covers your worst-case redelivery window.
The idempotency logic is usually simple. The discipline of applying it consistently is not. A common failure mode: the happy path is idempotent, but the error handling path isn't. The retry logic fires on a transient HTTP error, resubmits the request, and the external API processes it twice because the first attempt had actually succeeded.
When you're evaluating a streaming platform's exactly-once claim, the question to ask is not "does your system support exactly-once?" The question is: what is the transactional boundary of that guarantee, and does it cover my sink? If the sink is a managed Kafka topic and your entire pipeline is in Kafka Streams or Flink with Kafka connectors on both ends, the answer might genuinely be yes. If the sink is your REST API, the answer is no, and you need idempotent consumers.
Exactly-once is achievable. It's just not free, and it's not universal. The architecture that gets you there without illusions is: exactly-once within the transactional boundary, idempotent consumers at every boundary crossing. Not because the platform is lying to you, but because the platform is honest about what a transaction means.
Back to writing