Poison Message Handling

Imagine a mail sorting facility where every package passes through an X-ray scanner. One package triggers an alarm. The worker puts it back on the conveyor belt. It comes through the scanner again, alarm. Back on the belt. Alarm. Back on the belt. Alarm. Meanwhile, thousands of legitimate packages are piling up behind it. The entire facility grinds to a halt because of one bad package.

That's a poison message. It's a message that, no matter how many times you try, will always fail processing. Maybe it contains malformed data that crashes the deserializer. Maybe it references a database record that was deleted. Maybe it triggers a bug in business logic that throws an unhandled exception.

The danger isn't the failure itself. It's the infinite retry loop. Most messaging systems redeliver failed messages. A poison message fails, gets redelivered, fails again, gets redelivered, fails again. It consumes all of your consumer's capacity on a message that will never succeed, while legitimate messages queue up behind it.

LinkedIn's messaging team documented an incident where a single message with a corrupted protobuf payload blocked an entire consumer group for 45 minutes. The consumer kept crashing on deserialization, restarting, picking up the same message, crashing again. They processed zero messages from that partition until an engineer manually skipped the offset. Their post-incident review led to the poison message detection pattern they still use today.

Poison message handling is the strategy for detecting these messages, isolating them, and allowing the rest of the queue to flow.

Strategy 1: Retry count threshold.

Track how many times each message has been delivered. After N failures (typically 3-5), route it to a dead letter queue.

Message arrives -> attempt 1 -> FAIL
Redelivered    -> attempt 2 -> FAIL
Redelivered    -> attempt 3 -> FAIL
-> Move to DLQ, stop retrying

Most brokers track this automatically:

• RabbitMQ: x-death header contains delivery count and original queue name
• SQS: ApproximateReceiveCount attribute (incremented on each receive)
• Azure Service Bus: DeliveryCount property (exact, not approximate)
• Kafka: no built-in tracking. You maintain the count in application code or message headers.

Strategy 2: Error classification.

Not all errors are equal. Some are transient (network timeout, retry might work). Some are permanent (invalid JSON, retrying won't fix it). Classify errors and handle accordingly:

try {
 await processMessage(msg);
 await msg.ack();
} catch (error) {
 if (isTransientError(error)) {
  await msg.nack({ requeue: true }); // Retry
 } else {
  await msg.nack({ requeue: false }); // Permanent -> DLQ
 }
}

Stripe classifies errors into three buckets: transient (503, timeout, connection reset), permanent (400, 422, deserialization failure), and unknown (everything else). Transient errors get 3 retries with backoff. Permanent errors go straight to the DLQ. Unknown errors get 1 retry (in case it was a fluke) then DLQ.

Strategy 3: Circuit breaker on the consumer.

If the consumer fails N messages in a row (say 10), it stops consuming entirely and alerts. This prevents a batch of poison messages from overwhelming the consumer while a human investigates. The circuit breaker reopens after a cooldown period or manual reset.

Strategy 4: Quarantine queue.

Instead of a generic DLQ, route suspected poison messages to a quarantine queue with a longer delay. After the delay, re-attempt processing. If the failure was caused by a temporary downstream issue (a missing database record that was being created by another service), the retry might succeed. Quarantine is for messages that might be recoverable. The DLQ is for messages that definitely aren't.