Exactly-Once Processing

There are three common delivery guarantees in messaging and stream processing. At-most-once means a message is delivered zero or one times, so you may lose data but never duplicate it. At-least-once means a message is delivered one or more times, so you never lose data but may process duplicates. Exactly-once sits between them as the hardest one: the effect of a message lands once and only once.

The word exactly-once is slightly misleading. Networks cannot guarantee a message is physically transmitted only one time, because an acknowledgment can be lost after the message was already received, forcing a retry. What systems really guarantee is exactly-once effect, sometimes called effectively-once. The message may be delivered twice, but the observable outcome, like a row written or a counter incremented, happens once.

The classic failure that makes this hard: a consumer reads a message, updates the database, then crashes before recording that it finished. On restart it reads the same message again and double-applies the update. Exactly-once means that second apply is a no-op.

There are two main techniques, often combined. The first is idempotent writes. You design the operation so that applying it twice has the same result as applying it once. A common pattern is to attach a unique id to each message and have the consumer keep a record of ids it has already applied, rejecting repeats. An UPSERT keyed on the message id, or a database UNIQUE constraint, does the same job at the storage layer.

The second is atomic transactions that bundle the read offset and the write together. Kafka's exactly-once semantics work this way: a producer gets a transactional id and a per-partition sequence number so the broker can drop duplicate sends, and a consumer commits the input offset inside the same transaction as the output write. Either both the output and the new offset commit, or neither does. Downstream readers using read_committed isolation never see records from aborted transactions.

Stream processors extend this with checkpointing. Flink takes a consistent snapshot of all operator state plus source offsets using a distributed snapshot algorithm based on barriers flowing through the dataflow. On failure it rolls every operator back to the last completed checkpoint and replays from the recorded offsets, so each input contributes to state exactly once. Sinks that write outside the system still need idempotent or transactional writes to make the end-to-end guarantee hold.

Use exactly-once when duplicates corrupt correctness: moving money, billing a card, decrementing inventory, or computing financial aggregates. Charging a customer twice because a retry slipped through is a real business problem, not a rounding error.

It is not free. Transactional producers and two-phase commits add latency and reduce throughput, often by a meaningful margin compared with plain at-least-once. The exactly-once guarantee usually holds only within the boundary of one system. The moment your pipeline writes to an external service that is not part of the transaction, like sending an email or calling a third-party API, you are back to at-least-once unless that endpoint is itself idempotent.

A practical rule: prefer at-least-once delivery plus idempotent consumers for most pipelines. It is simpler, faster, and gives the same observable outcome for any operation you can make idempotent. Reserve true transactional exactly-once for the cases where you genuinely cannot make the write idempotent, or where you need atomicity across the read offset and multiple outputs at once.

Imagine a payment service that consumes a charge-customer topic. With at-least-once, a broker that does not receive an ack will redeliver the charge, and a naive consumer charges the card twice.

The idempotent fix: each charge message carries a unique payment_id. Before calling the card processor, the consumer inserts that payment_id into a payments table with a UNIQUE constraint. If the insert fails because the id already exists, the consumer knows it already handled this message and skips the charge entirely. The card is charged once no matter how many times the message is redelivered.

The transactional fix in Kafka Streams: read the charge, write the result to an output topic, and commit the source offset, all inside one transaction with processing.guarantee set to exactly_once_v2. If the consumer crashes mid-flight the transaction aborts, the offset is not advanced, and the whole step replays cleanly with no partial effect.