Idempotent Consumer
A message consumer that produces the same result whether it processes a message once or multiple times. Achieved by tracking processed message IDs or using natural idempotency.
What is Idempotent Consumer?
In short
An idempotent consumer is a message consumer that produces the same end result no matter how many times it processes the same message. It does this by recognizing duplicates, usually by storing the IDs of messages it has already handled or by writing in a way that repeating the operation changes nothing.
What it is
Almost every real message broker delivers at-least-once. Kafka, RabbitMQ, AWS SQS, and Google Pub/Sub all guarantee that a message will arrive, but they cannot guarantee it arrives exactly once. A consumer might process a message, then crash before it can acknowledge it, so the broker redelivers the same message. Network timeouts, consumer rebalances, and retries after a transient failure all create duplicates.
An idempotent consumer is the answer to that reality. Idempotent means an operation can be applied many times and the system lands in the same state as if it had been applied once. So if a payment-confirmation message gets delivered three times, an idempotent consumer charges the customer once, not three times.
This is not the same as exactly-once delivery, which is extremely hard and often impossible across a network. Instead you accept at-least-once delivery and make the processing side safe against repeats. The slogan engineers use is at-least-once delivery plus idempotent processing equals effectively-once.
How it works under the hood
The most common approach is a dedup table, sometimes called an inbox table or a processed-messages table. Every message carries a unique ID, either assigned by the producer or derived from its content. Before doing any work, the consumer checks whether that ID already exists in the table. If it does, the message is skipped. If not, the consumer does the work and records the ID.
The catch is that the business write and the dedup record must commit together. If you process the order but crash before saving the ID, the next redelivery will process the order again. The fix is to write both inside one database transaction, so they succeed or fail as a unit. With Postgres you might do INSERT INTO processed_messages id VALUES the_id ON CONFLICT DO NOTHING in the same transaction as the order insert; if the insert hits a conflict, you know it is a duplicate and roll back the work.
The second approach is natural idempotency, where the operation is safe to repeat by its nature. Setting account status to active is idempotent because running it twice leaves the same value. Adding 50 dollars to a balance is not, because running it twice adds 100. When you can phrase the work as a set-to or an upsert keyed on a stable ID rather than an increment, you often avoid needing a dedup table at all.
Dedup tables grow forever if you let them, so production systems expire old IDs. A common pattern keeps message IDs for a window longer than the broker's maximum redelivery delay, for example 7 days, then deletes them with a TTL or a scheduled job.
When to use it and the trade-offs
Use an idempotent consumer whenever a duplicated message would cause real harm: double charges, duplicate emails, sending a shipment twice, double-counting inventory. In practice that means almost every consumer that writes to a database or calls an external API with a side effect.
The cost is an extra read and write per message, plus the storage and cleanup of the dedup table. Under high throughput the dedup table can become a hot spot, so teams shard it, use a fast key-value store like Redis or DynamoDB with a TTL instead of a relational table, or pick a partition key that spreads writes evenly.
The hardest trade-off is choosing the idempotency key. Using the broker's auto-generated message ID protects you against broker redeliveries but not against a producer that sends the same logical event twice with two different IDs. A business key, such as order_id plus event_type, protects against both but requires the producer to generate it deterministically. Pick the key that matches the duplicates you actually need to defend against.
A concrete example
Picture an order service that publishes an OrderPlaced event to Kafka, and a billing consumer that charges the card. Kafka is at-least-once, so a consumer rebalance can replay the last few messages.
The billing consumer reads OrderPlaced, opens a database transaction, and tries to insert the order's id into a charges table where id is the primary key. If the insert succeeds, it calls Stripe to charge the card using the same id as the Stripe idempotency key, then commits. If the insert fails with a duplicate key error, the consumer knows it already charged this order, rolls back, and acknowledges the message without calling Stripe again.
Notice the layering. The dedup table stops re-processing inside our system, and Stripe's own idempotency key stops a double charge even in the rare window where our process dies mid-call. That belt-and-suspenders design is normal in payment systems, because the cost of one duplicate is a real refund and an angry customer.
Where it is used in production
Apache Kafka
Delivers at-least-once by default, so consumers add a dedup table or use the transactional producer with idempotence enabled to handle replays safely.
Stripe
Exposes an Idempotency-Key header on its API so retrying a charge request returns the original result instead of charging twice.
AWS DynamoDB
Commonly used as the dedup store, holding processed message IDs with a TTL attribute so old keys expire automatically.
AWS SQS
Standard queues are at-least-once and rely on consumer-side idempotency; FIFO queues add a deduplication ID window of 5 minutes as a partial safeguard.
Frequently asked questions
- What is the difference between idempotent consumer and exactly-once delivery?
- Exactly-once delivery tries to guarantee a message is delivered one time at the broker level, which is very hard across a network. An idempotent consumer accepts at-least-once delivery and makes the processing side safe against duplicates, so the end result is the same as if the message arrived once. The combination is often called effectively-once.
- How do I generate the idempotency key?
- Use a stable, unique identifier for the logical event. The broker's message ID guards against broker redeliveries. A business key such as order_id plus event_type guards against a producer that sends the same event twice with different broker IDs. Choose based on which duplicates you need to defend against, and make the producer generate it deterministically.
- Why must the dedup record and the business write share a transaction?
- If you do the work but crash before recording the message ID, the next redelivery sees no record and processes the message again. Committing the business change and the dedup record in one transaction means they always succeed or fail together, so a crash never leaves you half-done.
- Won't the dedup table grow forever?
- It will if you never clean it. Keep IDs only for a window longer than the broker's maximum redelivery delay, for example 7 days, then expire them with a TTL in Redis or DynamoDB or a scheduled delete job in a relational database.
- Can I skip the dedup table entirely?
- Sometimes. If the operation is naturally idempotent, like setting a status to active or upserting a row keyed on a stable ID, repeating it changes nothing and you need no dedup table. You only need one when the operation has a side effect that repeats, such as incrementing a balance or sending an email.
Learn Idempotent Consumer hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Idempotent Consumer as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Idempotency
An operation that produces the same result whether you run it once or multiple times. Critical for safe retries in distributed systems.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
At-Least-Once Delivery
A messaging guarantee where every message is delivered one or more times. Simpler than exactly-once but requires consumers to handle duplicates via idempotency.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.