We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Messaging Event Systems
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Messaging Event Systems
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
At-most-once: messages might be lost. At-least-once: messages might be duplicated. The obvious question: why not exactly-once? Every message processed precisely one time. No loss. No duplicates.
Engineers have wanted this since the dawn of distributed systems. And for decades, the was that exactly-once delivery is impossible in a distributed system. The Two Generals Problem (1975) and the FLP impossibility result (1985) showed that you cannot guarantee coordinated action between two parties over an unreliable network. If the acknowledgment can be lost, you can't distinguish "message processed but ACK lost" from "message never received."
And yet claims exactly-once semantics. Google Pub/Sub has exactly-once delivery. Azure Service Bus advertises it. Are they lying?
Not exactly. They've achieved something that looks like exactly-once from the application's perspective, even though the underlying network and system components still use at-least-once. The trick is combining producers, transactional writes, and consumer-side deduplication into a coordinated system that eliminates observable duplicates. True exactly-once delivery across arbitrary network boundaries remains impossible. But exactly-once processing, within the boundaries of a carefully designed system, is achievable.
Understanding the distinction between exactly-once delivery and exactly-once processing is the key to this lesson.
Consider the simplest possible scenario: Producer sends message to Consumer over a network.
Producer ──[message]──> Consumer
Producer <──[ACK]────── Consumer
Case 1: Consumer receives message, sends ACK, Producer receives ACK. Everything works. Once.
Case 2: Consumer receives message, sends ACK, ACK is lost on the network. Producer doesn't receive ACK. From Producer's perspective, it looks identical to Case 3.
Case 3: Message is lost on the network. Consumer never receives it. Producer doesn't receive ACK.
Producer can't distinguish Case 2 (message delivered, ACK lost) from Case 3 (message lost). Its only option is to retry, which creates a duplicate in Case 2.
Case 2 timeline:
Producer sends msg ──> Consumer processes msg
Producer waits... Consumer sends ACK ──✗── (ACK lost)
Producer times out
Producer resends msg ──> Consumer processes msg AGAIN ← duplicate!
This is fundamental. No amount of clever protocol design can solve it at the network level. The acknowledgment itself can always be lost, creating ambiguity about whether the original message was processed.
So how do and others claim exactly-once?
They don't solve it at the network level. They solve it at the application level by making duplicates invisible to the consumer.