We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
In short
Exactly-once processing is a guarantee that every message or record affects the system's final state one time and only one time, even when machines crash, networks drop packets, or work gets retried. Real systems achieve it not by literally delivering each message once, but by combining at-least-once delivery with idempotent writes or atomic transactions so that duplicate deliveries produce no extra effect.
In short
Exactly-once processing is a guarantee that every message or record affects the system's final state one time and only one time, even when machines crash, networks drop packets, or work gets retried. Real systems achieve it not by literally delivering each message once, but by combining at-least-once delivery with idempotent writes or atomic transactions so that duplicate deliveries produce no extra effect.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Exactly-Once Processing across a run of lessons, not one page. These 4 alone are about 122 minutes of step-by-step reading, every one with a quiz.
Processing every event exactly once despite failures, the holy grail of stream processing.
The holy grail of messaging, process every message once and only once. Here's why it's nearly impossible and how to fake it.
System design of a distributed task scheduler. Covers job queues, cron jobs, exactly-once runs with leases, retries and dead letter queues.
Detect and discard duplicate messages before they cause double-processing.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Exactly-Once Semanticspay once,
yours for life
There are three common delivery guarantees in messaging and stream processing. At-most-once means a message is delivered zero or one times, so you may lose data but never duplicate it. At-least-once means a message is delivered one or more times, so you never lose data but may process duplicates. Exactly-once sits between them as the hardest one: the effect of a message lands once and only once.
The word exactly-once is slightly misleading. Networks cannot guarantee a message is physically transmitted only one time, because an acknowledgment can be lost after the message was already received, forcing a retry. What systems really guarantee is exactly-once effect, sometimes called effectively-once. The message may be delivered twice, but the observable outcome, like a row written or a counter incremented, happens once.
The classic failure that makes this hard: a consumer reads a message, updates the database, then crashes before recording that it finished. On restart it reads the same message again and double-applies the update. Exactly-once means that second apply is a no-op.
There are two main techniques, often combined. The first is idempotent writes. You design the operation so that applying it twice has the same result as applying it once. A common pattern is to attach a unique id to each message and have the consumer keep a record of ids it has already applied, rejecting repeats. An UPSERT keyed on the message id, or a database UNIQUE constraint, does the same job at the storage layer.
The second is atomic transactions that bundle the read offset and the write together. Kafka's exactly-once semantics work this way: a producer gets a transactional id and a per-partition sequence number so the broker can drop duplicate sends, and a consumer commits the input offset inside the same transaction as the output write. Either both the output and the new offset commit, or neither does. Downstream readers using read_committed isolation never see records from aborted transactions.
Stream processors extend this with checkpointing. Flink takes a consistent snapshot of all operator state plus source offsets using a distributed snapshot algorithm based on barriers flowing through the dataflow. On failure it rolls every operator back to the last completed checkpoint and replays from the recorded offsets, so each input contributes to state exactly once. Sinks that write outside the system still need idempotent or transactional writes to make the end-to-end guarantee hold.
Use exactly-once when duplicates corrupt correctness: moving money, billing a card, decrementing inventory, or computing financial aggregates. Charging a customer twice because a retry slipped through is a real business problem, not a rounding error.
It is not free. Transactional producers and two-phase commits add latency and reduce throughput, often by a meaningful margin compared with plain at-least-once. The exactly-once guarantee usually holds only within the boundary of one system. The moment your pipeline writes to an external service that is not part of the transaction, like sending an email or calling a third-party API, you are back to at-least-once unless that endpoint is itself idempotent.
A practical rule: prefer at-least-once delivery plus idempotent consumers for most pipelines. It is simpler, faster, and gives the same observable outcome for any operation you can make idempotent. Reserve true transactional exactly-once for the cases where you genuinely cannot make the write idempotent, or where you need atomicity across the read offset and multiple outputs at once.
Imagine a payment service that consumes a charge-customer topic. With at-least-once, a broker that does not receive an ack will redeliver the charge, and a naive consumer charges the card twice.
The idempotent fix: each charge message carries a unique payment_id. Before calling the card processor, the consumer inserts that payment_id into a payments table with a UNIQUE constraint. If the insert fails because the id already exists, the consumer knows it already handled this message and skips the charge entirely. The card is never charged twice. One gap remains: if the consumer crashes after the insert but before the charge, the redelivered message is skipped and the charge never happens, so real systems also pass payment_id to the card processor as an idempotency key.
The transactional fix in Kafka Streams: read the charge, write the result to an output topic, and commit the source offset, all inside one transaction with processing.guarantee set to exactly_once_v2. If the consumer crashes mid-flight the transaction aborts, the offset is not advanced, and the whole step replays cleanly with no partial effect.
Apache Kafka
Provides exactly-once semantics through idempotent producers, transactional ids, and read_committed consumers; Kafka Streams exposes it with processing.guarantee=exactly_once_v2.
Apache Flink
Uses distributed checkpoint barriers to snapshot operator state and source offsets, replaying from the last checkpoint on failure for end-to-end exactly-once with transactional sinks.
Google Cloud Dataflow
Built on the Apache Beam model, deduplicates records and commits state atomically to give exactly-once processing for streaming pipelines.
Stripe
Uses idempotency keys on API requests so a retried charge with the same key returns the original result instead of charging the card again.
Exactly-once delivery over an unreliable network is not possible, because a lost acknowledgment forces a resend. What is possible is exactly-once effect: the message may be delivered more than once, but idempotent writes or transactions ensure the final state changes only once.
At-least-once guarantees no message is lost but allows duplicates, so the consumer must tolerate processing the same message more than once. Exactly-once guarantees the effect lands a single time, which you build on top of at-least-once delivery using idempotency or transactions.
Producers use a transactional id and per-partition sequence numbers so the broker drops duplicate sends. Consumers commit the input offset inside the same transaction as the output write, so output and offset advance together or not at all, and readers use read_committed to skip aborted transactions.
No. For most pipelines, at-least-once delivery plus an idempotent consumer gives the same correct outcome with less latency and complexity. Reserve true transactional exactly-once for money movement, billing, inventory, and aggregates where a duplicate would be wrong and the write cannot easily be made idempotent.
It is a unique identifier the client attaches to a request so the server can recognize a retry. The server records keys it has already handled and returns the original result for repeats instead of redoing the work. Stripe uses this on its payment API to prevent double charges.
Exactly-Once Delivery
The holy grail of messaging, process every message once and only once. Here's why it's nearly impossible and how to fake it.
Design a Distributed Task Scheduler
System design of a distributed task scheduler. Covers job queues, cron jobs, exactly-once runs with leases, retries and dead letter queues.
Message Deduplication
Detect and discard duplicate messages before they cause double-processing
At-Least-Once Delivery
A messaging guarantee where every message is delivered one or more times. Simpler than exactly-once but requires consumers to handle duplicates via idempotency.
Idempotency
An operation that produces the same result whether you run it once or multiple times. Critical for safe retries in distributed systems.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
This page explains the idea. The "Exactly-Once Semantics" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Exactly-Once Semantics lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Exactly-Once Processing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
