We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
2 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Messaging Event Systems
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Messaging Event Systems
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
The postal service has a room full of undeliverable mail. Wrong addresses, no such person, refused delivery. These letters don't just vanish. They go to a dead letter office where workers investigate, try to find the intended recipient, and either redeliver or dispose of the mail.
In messaging systems, the equivalent is a dead letter queue (DLQ). When a message can't be processed, the consumer rejects it, it exceeds the retry limit, it expires, or its format is invalid, it gets moved to a special queue instead of being lost forever.
The DLQ serves two purposes. First, it prevents poisonous messages from blocking your main queue. Without a DLQ, a message that consistently fails would be retried indefinitely, blocking all messages behind it. Second, it preserves failed messages for investigation. Engineers can inspect the DLQ, understand why messages failed, fix the bug, and replay the messages.
Uber's trip processing pipeline generates roughly 1 million messages per minute. In 2022, a schema change in the rider service broke deserialization for trip-updated events. Without their DLQ strategy, those messages would have been lost or would have blocked the entire trip pipeline. Instead, 200K messages landed in the DLQ over 15 minutes, the alert fired at minute 2, the deployment was rolled back at minute 8, and the DLQ messages were replayed by minute 20. Zero data loss.
Every production messaging system needs a DLQ strategy. It's the difference between "messages disappear and nobody knows why" and "messages fail gracefully and we can recover."
Messages are routed to a DLQ when specific conditions are met:
Retry exhaustion. The message was delivered and rejected N times (e.g., 3 retries). After the last rejection, it's moved to the DLQ. This is the most common trigger.
expiration. The message sat in the queue longer than its time-to-live and was never consumed. This happens when consumers can't keep up with the message rate.
Consumer rejection. The consumer explicitly rejects the message with nack(requeue: false). "I can't handle this, don't give it back to me."
Queue overflow. The queue reaches its maximum length, and the overflow policy routes excess messages to a DLQ instead of dropping them.
DLQ configuration across systems:
| System | DLQ Mechanism | Max Retries Config | Metadata Preserved |
|---|---|---|---|
| RabbitMQ | Dead Letter Exchange (DLX) | x-death header tracks count | Original queue, timestamp, reason |
| Amazon SQS | Redrive policy pointing to DLQ | maxReceiveCount | ApproximateReceiveCount, all attributes |
| Azure Service Bus | Built-in DLQ per queue | MaxDeliveryCount (default: 10) | DeadLetterReason, DeadLetterErrorDescription |
| Google Pub/Sub | Dead letter topic | maxDeliveryAttempts | Delivery attempt count, subscription name |
| Manual (produce to DLT topic) | Application logic | Whatever you attach as headers |
The Kafka gap. Kafka has no native DLQ. You implement it yourself: when a consumer can't process a message, it publishes the message to a separate "dead letter topic" and commits the offset. Spring Kafka's @RetryableTopic annotation automates this pattern, creating retry topics (orders-retry-0, orders-retry-1) and a final dead letter topic (orders-dlt).