Exactly-Once Semantics

A payment processor runs a stream job that counts daily revenue per merchant. The job crashes at 2 AM, recovers, and replays the last 30 seconds of Kafka. Without exactly once, those 30 seconds of transactions get counted twice. The dashboard shows $11,000 when the real number is $10,000. The merchant reconciles against their bank, files a support ticket, and your on call engineer spends the next 4 hours figuring out why the numbers are wrong.

This is not a hypothetical. Robinhood disclosed a production incident in March 2020 where a stream processing pipeline with a non transactional HTTP sink duplicated order events during a broker failover. The postmortem concluded that HTTP APIs without idempotency keys cannot deliver exactly once no matter what the upstream processor guarantees.

Exactly once means each input event affects the output exactly once. Not zero times (data loss), not more than once (duplicates). The mechanism is conceptually simple: checkpoint the processor state and the source offsets together as an atomic unit, and on recovery, rewind both to the same consistent point. But the details are where production breaks. Kafka transactions, Flink two phase commit sinks, transactional.id fencing, read_committed isolation levels, and the precise handshake between checkpoint barriers and Kafka producer commits. Get any one of these wrong and you have at least once with invisible duplicates.

By the end of this lesson you will know how Kafka transactions work at the protocol level, how Flink's TwoPhaseCommitSinkFunction coordinates with checkpoint barriers, what makes the end to end chain break, and how to audit every link in your pipeline.

Exactly-once means each input event affects the output exactly once, not zero times (data loss) and not more than once (duplicates). This is the holy grail because it means your stream processor produces correct results despite failures.

Three levels of delivery guarantees:

How Flink achieves exactly-once:

1. Periodic checkpoints capture operator state and input positions
2. On failure, restore from the last checkpoint
3. Replay input from the checkpointed position
4. Deterministic processing produces the same output
5. External sinks use two-phase commit to avoid duplicate writes

The catch: Exactly-once within the processor is achievable. Exactly-once end-to-end (including external systems) requires idempotent sinks or two-phase commit protocols. Writing to Kafka with exactly-once is well-supported. Writing to a REST API with exactly-once requires idempotency keys.