We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A message consumer that produces the same result whether it processes a message once or multiple times. Achieved by tracking processed message IDs or using natural idempotency.
In short
An idempotent consumer is a message consumer that produces the same end result no matter how many times it processes the same message. It does this by recognizing duplicates, usually by storing the IDs of messages it has already handled or by writing in a way that repeating the operation changes nothing.
In short
An idempotent consumer is a message consumer that produces the same end result no matter how many times it processes the same message. It does this by recognizing duplicates, usually by storing the IDs of messages it has already handled or by writing in a way that repeating the operation changes nothing.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Idempotent Consumer across a run of lessons, not one page. These 4 alone are about 84 minutes of step-by-step reading, every one with a quiz.
Design consumers that produce the same result whether a message is processed once or ten times.
Caching a result set freezes it, and the deep pagination it is usually bought to fix disappears when you change the question instead.
Scale message processing by having multiple consumers race to handle messages from the same queue.
Detect and discard duplicate messages before they cause double-processing.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Idempotent Consumerpay once,
yours for life
Almost every real message broker delivers at-least-once. Kafka, RabbitMQ, AWS SQS, and Google Pub/Sub all guarantee that a message will arrive, but in their normal modes they do not guarantee it arrives exactly once. A consumer might process a message, then crash before it can acknowledge it, so the broker redelivers the same message. Network timeouts, consumer rebalances, and retries after a transient failure all create duplicates.
An idempotent consumer is the answer to that reality. Idempotent means an operation can be applied many times and the system lands in the same state as if it had been applied once. So if a payment-confirmation message gets delivered three times, an idempotent consumer charges the customer once, not three times.
This is not the same as exactly-once delivery, which is extremely hard and often impossible across a network. Instead you accept at-least-once delivery and make the processing side safe against repeats. The slogan engineers use is at-least-once delivery plus idempotent processing equals effectively-once.
The most common approach is a dedup table, sometimes called an inbox table or a processed-messages table. Every message carries a unique ID, either assigned by the producer or derived from its content. Before doing any work, the consumer checks whether that ID already exists in the table. If it does, the message is skipped. If not, the consumer does the work and records the ID.
The catch is that the business write and the dedup record must commit together. If you process the order but crash before saving the ID, the next redelivery will process the order again. The fix is to write both inside one database transaction, so they succeed or fail as a unit. With Postgres you might do INSERT INTO processed_messages id VALUES the_id ON CONFLICT DO NOTHING in the same transaction as the order insert; if the insert hits a conflict, you know it is a duplicate and roll back the work.
The second approach is natural idempotency, where the operation is safe to repeat by its nature. Setting account status to active is idempotent because running it twice leaves the same value. Adding 50 dollars to a balance is not, because running it twice adds 100. When you can phrase the work as a set-to or an upsert keyed on a stable ID rather than an increment, you often avoid needing a dedup table at all.
Dedup tables grow forever if you let them, so production systems expire old IDs. A common pattern keeps message IDs for a window longer than the broker's maximum redelivery delay, for example 7 days, then deletes them with a TTL or a scheduled job.
Use an idempotent consumer whenever a duplicated message would cause real harm: double charges, duplicate emails, sending a shipment twice, double-counting inventory. In practice that means almost every consumer that writes to a database or calls an external API with a side effect.
The cost is an extra read and write per message, plus the storage and cleanup of the dedup table. Under high throughput the dedup table can become a hot spot, so teams shard it, use a fast key-value store like Redis or DynamoDB with a TTL instead of a relational table, or pick a partition key that spreads writes evenly.
The hardest trade-off is choosing the idempotency key. Using the broker's auto-generated message ID protects you against broker redeliveries but not against a producer that sends the same logical event twice with two different IDs. A business key, such as order_id plus event_type, protects against both but requires the producer to generate it deterministically. Pick the key that matches the duplicates you actually need to defend against.
Picture an order service that publishes an OrderPlaced event to Kafka, and a billing consumer that charges the card. Kafka is at-least-once, so a consumer rebalance can replay the last few messages.
The billing consumer reads OrderPlaced, opens a database transaction, and tries to insert the order's id into a charges table where id is the primary key. If the insert succeeds, it calls Stripe to charge the card using the same id as the Stripe idempotency key, then commits. If the insert fails with a duplicate key error, the consumer knows it already charged this order, rolls back, and acknowledges the message without calling Stripe again.
Notice the layering. The dedup table stops re-processing inside our system, and Stripe's own idempotency key stops a double charge even in the rare window where our process dies mid-call. That belt-and-suspenders design is normal in payment systems, because the cost of one duplicate is a real refund and an angry customer.
Apache Kafka
Delivers at-least-once by default, so consumers add a dedup table or use the transactional producer with idempotence enabled to handle replays safely.
Stripe
Exposes an Idempotency-Key header on its API so retrying a charge request returns the original result instead of charging twice.
AWS DynamoDB
Commonly used as the dedup store, holding processed message IDs with a TTL attribute so old keys expire automatically.
AWS SQS
Standard queues are at-least-once and rely on consumer-side idempotency; FIFO queues add a deduplication ID window of 5 minutes as a partial safeguard.
Exactly-once delivery tries to guarantee a message is delivered one time at the broker level, which is very hard across a network. An idempotent consumer accepts at-least-once delivery and makes the processing side safe against duplicates, so the end result is the same as if the message arrived once. The combination is often called effectively-once.
Use a stable, unique identifier for the logical event. The broker's message ID guards against broker redeliveries. A business key such as order_id plus event_type guards against a producer that sends the same event twice with different broker IDs. Choose based on which duplicates you need to defend against, and make the producer generate it deterministically.
If you do the work but crash before recording the message ID, the next redelivery sees no record and processes the message again. Committing the business change and the dedup record in one transaction means they always succeed or fail together, so a crash never leaves you half-done.
It will if you never clean it. Keep IDs only for a window longer than the broker's maximum redelivery delay, for example 7 days, then expire them with a TTL in Redis or DynamoDB or a scheduled delete job in a relational database.
Sometimes. If the operation is naturally idempotent, like setting a status to active or upserting a row keyed on a stable ID, repeating it changes nothing and you need no dedup table. You only need one when the operation has a side effect that repeats, such as incrementing a balance or sending an email.
Idempotency
An operation that produces the same result whether you run it once or multiple times. Critical for safe retries in distributed systems.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
At-Least-Once Delivery
A messaging guarantee where every message is delivered one or more times. Simpler than exactly-once but requires consumers to handle duplicates via idempotency.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
This page explains the idea. The "Idempotent Consumer" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Idempotent Consumer lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Idempotent Consumer in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
