We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Distributed Systems Core
A pattern where a service writes events to an outbox table in the same database transaction as its state change. A separate process reads the outbox and publishes events, ensuring atomicity.
In short
The outbox pattern is a way to publish events reliably from a service by writing the event into an "outbox" table inside the same database transaction that changes the service's state, then having a separate process read that table and push the events to a message broker. Because the state change and the event row commit together or not at all, you never end up with a database update that has no matching event, or an event for something that never actually happened.
In short
The outbox pattern is a way to publish events reliably from a service by writing the event into an "outbox" table inside the same database transaction that changes the service's state, then having a separate process read that table and push the events to a message broker. Because the state change and the event row commit together or not at all, you never end up with a database update that has no matching event, or an event for something that never actually happened.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Outbox Pattern across a run of lessons, not one page. These 4 alone are about 187 minutes of step-by-step reading, 2 of them with code you run in the browser, all with a quiz.
Guarantee that your database write and message publish either both happen or neither does.
Two writers update the same Delta Lake table at once: who wins, and what does the loser see? I ran real writer processes against 3,475,226 NYC taxi trips, 5 runs per case. Two appends both committed every time. Two updates of different rows clashed every time, because they shared the same 9 files: one committed, the other got CommitFailedError. Splitting the table by the filter column let both commit. A check-then-append job started twice stored 317,150 trips for a day of 158,575 in 5 of 5 runs. In delta-rs 1.6.6, a WriteSerializable setting took effect only when spelled writeSerializable.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
Coordinate multi-step workflows across services by tracking state and routing messages through a sequence.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Transactional Outbox Patternpay once,
yours for life
Imagine an order service that saves a new order to Postgres and then publishes an OrderPlaced event to Kafka so the payment and shipping services can react. These are two separate systems: a database and a message broker. There is no single transaction that spans both.
If the service writes to the database and then crashes before publishing, the order exists but nobody downstream ever hears about it. If it publishes first and then the database write fails, you have told the world about an order that does not exist. This is the dual-write problem, and it shows up any time one action has to update local state and notify other services.
Retrying does not fix it cleanly either. A naive retry can publish the same event twice, or publish an event for a transaction that later rolled back. You need the state change and the intent to publish to be decided together, atomically, in one place. The one place where you already have a real transaction is your database.
You add a second table, often called outbox or events, to the same database the service already writes to. When the service handles a command, it does the business write and an insert into the outbox table inside one local transaction. The outbox row holds everything needed to publish later: an event id, the topic or aggregate type, a payload as JSON, and a status or created timestamp.
Because both writes share one transaction, the database guarantees they commit together. If the business write rolls back, the outbox row disappears with it. There is now exactly one durable record of intent.
A separate component, the message relay, moves rows from the outbox to the broker. There are two common ways to drive it. Polling publisher: a background worker runs a query like select from outbox where published_at is null order by created_at, publishes each event, then marks it sent. Change data capture: a tool such as Debezium tails the database write-ahead log, sees new outbox inserts, and streams them to Kafka with no extra query load on the database.
Delivery is at-least-once, so the same event can arrive more than once after a crash or retry. The fix is to ship a stable event id with every message and make consumers idempotent, usually by recording processed ids and ignoring duplicates.
Reach for the outbox pattern when a service owns a relational database and must publish events or call other services as part of the same logical operation, and losing or duplicating those events would corrupt downstream state. Order processing, payments, inventory, and any saga step are typical fits.
The cost is extra moving parts. You now run a relay or a CDC pipeline, monitor outbox lag, and clean up old rows so the table does not grow without bound. A polling relay adds query load and a small delay, often hundreds of milliseconds to a few seconds, between the commit and the event landing on the broker. CDC removes the polling load but adds operational weight from running Debezium and Kafka Connect.
It is also not a magic bullet for ordering. The relay preserves the order rows were committed in if you publish by created order and use a single partition key per aggregate, but with multiple workers or partitions you can reorder events, so design consumers to tolerate that.
If your service does not have a transactional database, or you genuinely need exactly-once cross-system writes, the outbox does not apply directly. In that case people reach for two-phase commit, which is slower and brittle, or accept at-least-once with idempotency, which is what the outbox embraces.
A checkout service receives a place order request. In one Postgres transaction it inserts a row into orders and a row into outbox with event_id, aggregate_type order, type OrderPlaced, and a JSON payload of the order details. The transaction commits, the HTTP response returns 200 to the customer.
Moments later, Debezium reads the Postgres write-ahead log, sees the new outbox insert, and publishes an OrderPlaced message to the orders Kafka topic keyed by order id. The payment service consumes it, checks whether it has already processed that event_id, and if not, charges the card.
If the checkout service had crashed right after commit, nothing is lost: the outbox row is durable and the relay picks it up on recovery. If the relay published the event twice after a restart, the payment service sees the same event_id, recognizes the duplicate, and skips the second charge. State stays correct on both sides without any distributed transaction.
Debezium
Open source CDC platform widely used to implement the outbox relay by tailing the database log and streaming outbox rows to Kafka.
Shopify
Uses transactional outbox style patterns to reliably emit domain events from order and commerce services without losing or duplicating them.
PostgreSQL
Common home for the outbox table; its logical replication and write-ahead log make both polling and CDC based relays straightforward.
Microsoft Azure
The eShopOnContainers and .NET microservices reference architectures ship an integration event log table that is the outbox pattern by another name.
Publishing after commit is the dual write that the outbox fixes. If the service crashes between the commit and the publish, the event is lost forever. The outbox makes the event durable inside the same transaction as the state change, so a separate relay can always retry it after a crash.
No. It gives at-least-once delivery. The same event can be published more than once after retries or crashes. You make the end result correct by attaching a stable event id to each message and making consumers idempotent so duplicates are ignored.
Polling is simpler to build and fine at low to moderate volume, but it adds query load and a bit of latency. CDC with a tool like Debezium scales better and avoids polling the table, at the cost of running Debezium and Kafka Connect. Pick polling to start, move to CDC when volume or latency demands it.
It can, if you publish rows in commit order and key all events for one aggregate to the same partition. With multiple relay workers or partitions, events for different aggregates can interleave or reorder, so consumers should not assume strict global ordering.
Yes. Once an event is confirmed published you should mark it and periodically delete or archive old rows, otherwise the table grows forever and slows down the relay query. A scheduled job that purges rows older than a retention window is the usual approach.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
Change Data Capture
Capturing row-level changes in a database and streaming them to other systems in real time. Debezium reads the write-ahead log and publishes changes to Kafka.
Saga Pattern
A way to manage distributed transactions across microservices using a sequence of local transactions with compensating actions for rollback.
CAP Theorem
When a network failure splits a distributed data store, each node must either refuse requests to stay consistent or keep answering with possibly stale data. It cannot guarantee both during the partition.
Consensus
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.
This page explains the idea. The "Transactional Outbox Pattern" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Transactional Outbox Pattern lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Outbox Pattern in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
