Outbox Pattern

Imagine an order service that saves a new order to Postgres and then publishes an OrderPlaced event to Kafka so the payment and shipping services can react. These are two separate systems: a database and a message broker. There is no single transaction that spans both.

If the service writes to the database and then crashes before publishing, the order exists but nobody downstream ever hears about it. If it publishes first and then the database write fails, you have told the world about an order that does not exist. This is the dual-write problem, and it shows up any time one action has to update local state and notify other services.

Retrying does not fix it cleanly either. A naive retry can publish the same event twice, or publish an event for a transaction that later rolled back. You need the state change and the intent to publish to be decided together, atomically, in one place. The one place where you already have a real transaction is your database.

You add a second table, often called outbox or events, to the same database the service already writes to. When the service handles a command, it does the business write and an insert into the outbox table inside one local transaction. The outbox row holds everything needed to publish later: an event id, the topic or aggregate type, a payload as JSON, and a status or created timestamp.

Because both writes share one transaction, the database guarantees they commit together. If the business write rolls back, the outbox row disappears with it. There is now exactly one durable record of intent.

A separate component, the message relay, moves rows from the outbox to the broker. There are two common ways to drive it. Polling publisher: a background worker runs a query like select from outbox where published_at is null order by created_at, publishes each event, then marks it sent. Change data capture: a tool such as Debezium tails the database write-ahead log, sees new outbox inserts, and streams them to Kafka with no extra query load on the database.

Delivery is at-least-once, so the same event can arrive more than once after a crash or retry. The fix is to ship a stable event id with every message and make consumers idempotent, usually by recording processed ids and ignoring duplicates.

Reach for the outbox pattern when a service owns a relational database and must publish events or call other services as part of the same logical operation, and losing or duplicating those events would corrupt downstream state. Order processing, payments, inventory, and any saga step are typical fits.

The cost is extra moving parts. You now run a relay or a CDC pipeline, monitor outbox lag, and clean up old rows so the table does not grow without bound. A polling relay adds query load and a small delay, often hundreds of milliseconds to a few seconds, between the commit and the event landing on the broker. CDC removes the polling load but adds operational weight from running Debezium and Kafka Connect.

It is also not a magic bullet for ordering. The relay preserves the order rows were committed in if you publish by created order and use a single partition key per aggregate, but with multiple workers or partitions you can reorder events, so design consumers to tolerate that.

If your service does not have a transactional database, or you genuinely need exactly-once cross-system writes, the outbox does not apply directly. In that case people reach for two-phase commit, which is slower and brittle, or accept at-least-once with idempotency, which is what the outbox embraces.

A checkout service receives a place order request. In one Postgres transaction it inserts a row into orders and a row into outbox with event_id, aggregate_type order, type OrderPlaced, and a JSON payload of the order details. The transaction commits, the HTTP response returns 200 to the customer.

Moments later, Debezium reads the Postgres write-ahead log, sees the new outbox insert, and publishes an OrderPlaced message to the orders Kafka topic keyed by order id. The payment service consumes it, checks whether it has already processed that event_id, and if not, charges the card.

If the checkout service had crashed right after commit, nothing is lost: the outbox row is durable and the relay picks it up on recovery. If the relay published the event twice after a restart, the payment service sees the same event_id, recognizes the duplicate, and skips the second charge. State stays correct on both sides without any distributed transaction.