Message Deduplication

A customer buys a laptop for $1,200. The payment service processes the charge, but the acknowledgment gets lost on the network. The broker redelivers the message. The payment service processes it again. The customer sees two $1,200 charges on their credit card.

This is the duplicate problem, and it's inherent to at-least-once delivery. The broker guarantees your message won't be lost, but it might be delivered more than once. For idempotent operations (like "set status to active"), duplicates are harmless. For non-idempotent operations (like "charge $1,200"), duplicates are catastrophic.

Stripe processes over 250 million API requests per day. Their idempotency layer catches roughly 0.1% of those as duplicates. That's 250,000 duplicate requests per day that would have resulted in double-charges, double-refunds, or duplicate customer records without deduplication. The number sounds small as a percentage, but 250K erroneous financial transactions per day would end a payments company.

Message deduplication is the practice of detecting and discarding duplicate messages before they're processed. It's the first line of defense against the double-processing problem that plagues every messaging system operating with at-least-once guarantees.

The core idea: every message carries a unique identifier. Before processing, the consumer checks if it's already seen that ID. If yes, skip. If no, process and record the ID.

Producer-side deduplication (broker-level):

Some brokers deduplicate at the point of publishing:

Kafka's idempotent producer assigns a sequence number to each message. If the broker receives the same sequence number twice (because the producer retried after a network error), it silently discards the duplicate. This prevents duplicate production, not duplicate consumption. The distinction matters.

Consumer-side deduplication:

The consumer maintains a record of processed message IDs:

async function handleMessage(msg) {
 const messageId = msg.properties.messageId;

 // Check if already processed
 const exists = await db.query(
  'SELECT 1 FROM processed_messages WHERE id = $1',
  [messageId]
 );

 if (exists) {
  await msg.ack(); // Already handled, skip
  return;
 }

 // Process the message
 await processOrder(msg.body);

 // Record that we processed it (inside same transaction!)
 await db.query(
  'INSERT INTO processed_messages (id, processed_at) VALUES ($1, NOW())',
  [messageId]
 );

 await msg.ack();
}

The critical detail: the "check + process + record" must be atomic. If you process the message and crash before recording the ID, the redelivered message won't be recognized as a duplicate. Use a database transaction that wraps both the business logic and the ID recording. Shopify's payment processing pipeline runs exactly this pattern: one Postgres transaction wraps the balance update and the idempotency key insert.