"We're using Kafka, so we won't lose messages." This gets said in architecture reviews, and it's not wrong in the way "the earth is flat" is wrong. It's wrong in the way "our servers are in a safe neighborhood" is wrong. The durability Kafka provides is real, but it has specific preconditions. Skip them and you will lose messages, silently, with no error returned to the producer.
The log
Kafka's fundamental data structure is an append-only, ordered log. Messages in a topic are divided across partitions, and each partition is an ordered sequence of records with monotonically increasing offsets. Record 0 comes before record 1; record 1 comes before record 2. Nothing is ever updated in place. Old records stay where they are until a retention policy removes them.
Consumers track their own position in the log, called their "offset." Kafka doesn't care when you read, or even whether you read. A consumer that falls behind by a million messages can catch up at its own pace. A new consumer can replay from offset 0. This is different from a queue in a structural way: consuming a message from a queue removes it. Kafka's log is read-only from Kafka's perspective; consumers are just readers at a position.
The consequence is that Kafka can serve multiple consumer groups from the same log without any interference between them. It's also why Kafka is replayable. You can reset a consumer's offset to reprocess data, something that's impossible in a traditional queue once messages are acknowledged.
Replication
A single Kafka broker storing a partition is a single point of failure. If that machine dies, the data on it is gone. Kafka's durability story depends entirely on replication.
Each partition is replicated across N brokers, where N is the replication factor. One broker is the leader for that partition; the others are followers. Producers always write to the leader. Followers continuously pull new messages from the leader and replicate them.
The leader maintains an In-Sync Replica set, the ISR. A replica is considered in-sync if it has replicated up to within a configurable lag of the leader (by default, within 10 seconds and within the latest messages). If a follower falls too far behind, it's removed from the ISR. If it catches up, it's added back. The ISR is the set of brokers that are definitely up to date.
What acks=all means
When a producer sends a message, it can configure how many acknowledgements it needs before it considers the send successful. There are three options:
acks=0: fire and forget. The producer sends and immediately moves
on without waiting for any response. Maximum throughput, zero durability. A
broker crash after the network call is in-flight means the message is gone.
acks=1: the leader acknowledges. The producer waits for the leader
to write the message to its local log. Better than acks=0, but not safe: the
message survives a leader crash only if followers have replicated it before the
crash. If the leader crashes immediately after acknowledging but before any
follower replicates, the message is gone. This is the silent data loss scenario.
acks=all (or equivalently acks=-1): all in-sync
replicas acknowledge. The producer waits until the leader has written the message
and every member of the ISR has replicated it. Only then does the producer get a
success response. This is the configuration that actually gives you durability:
the message is on multiple machines before you proceed.
acks=all doesn't come free. It adds latency: you're waiting for
the slowest member of the ISR on every write. For high-throughput pipelines where
some message loss is acceptable (analytics events, logs), acks=1 is
a reasonable tradeoff. For financial records, order events, or anything where
loss is unacceptable, it isn't.
Leader election and the ISR
When a partition leader fails, Kafka elects a new leader. The straightforward choice: elect the most up-to-date replica from the ISR. Any replica in the ISR has all acknowledged messages, so this is safe. No committed message is lost.
The problem: what if the ISR is empty? This happens when all followers have fallen behind, maybe due to a network partition between the leader and its followers, or a transient overload that caused replicas to lag beyond the threshold and get removed from the ISR. Now the only in-sync replica was the leader, and the leader just died.
Kafka gives you a choice, controlled by unclean.leader.election.enable:
If set to false (the safe default in recent Kafka versions): Kafka
waits for an ISR member to come back. The partition is unavailable until then.
No messages are lost; producers get errors until the partition recovers.
If set to true: Kafka elects an out-of-sync replica. The partition
becomes available immediately. But that out-of-sync replica may be missing
messages that were acknowledged by the previous leader. Messages you were told
were durably stored. They're gone.
Unclean leader election is a durability-versus-availability tradeoff with a
sharp edge: the messages you lose are the ones you were explicitly told were
safe. Most production systems that care about durability set this to
false.
The failure modes
min.insync.replicas sets the minimum ISR size required before a
write is accepted. If you have a replication factor of 3 and
min.insync.replicas=2, writes require at least 2 in-sync replicas.
Losing one broker is fine; you still have 2 in-sync. Losing 2 simultaneously
means the ISR drops to 1, below the minimum, and the topic becomes read-only:
producers get errors. This is the right behavior. It prevents silent data loss
at the cost of availability.
The silent data loss scenario in detail: producer configured with
acks=1, replication factor 3. Producer writes to the leader. Leader
acknowledges immediately, before any follower replicates. Leader crashes. One of
the followers, which haven't seen the message, elect a new leader. The message is
gone. The producer saw a success response. No error was logged anywhere.
The safe configuration for durability: replication factor of at least 3,
min.insync.replicas=2, acks=all on the producer, and
unclean.leader.election.enable=false. Any single broker failure is
survivable with no data loss. Two simultaneous broker failures make topics
unavailable. Not data loss; unavailability that recovers when a broker comes
back. Three simultaneous broker failures require unclean.leader.election
to recover, at which point you've accepted some loss as the price of
availability.
What Kafka doesn't give you
Kafka provides ordering within a partition, not globally across partitions. If your topic has 12 partitions and you write messages A, B, C in sequence, those messages may land in different partitions and be consumed in arbitrary order relative to each other. Ordering guarantees require either a single partition (which limits throughput) or a partitioning key that routes related messages to the same partition.
Cross-topic atomicity, writing to topic A and topic B in one atomic operation, requires Kafka's transactional producer API, which adds meaningful complexity and some throughput overhead. By default, two writes to two topics are independent and can partially succeed.
Kafka is not a database. There's no indexed lookup. Consumers read by offset, which is a position, not a key. If you want "give me the message with order_id 12345," you need to either consume everything and filter, or maintain a separate index elsewhere. This is by design; Kafka's strength is sequential throughput, not random access.
Durability is a configuration, not a feature
Kafka's durability guarantees are real. Deployed correctly, it is genuinely
hard to lose a committed message. But "deployed correctly" has a specific
meaning: replication factor 3 or more, acks=all, appropriate
min.insync.replicas, and unclean leader election disabled.
The defaults are not all set to their safest values. They're set to balance
durability with throughput for a general audience. acks=1 is fast.
Unclean leader election historically defaulted to enabled. If you've inherited
a Kafka deployment and haven't audited these settings, it's worth doing before
the next incident review is about whether you actually need to reprocess the
last four hours of data.