Transactional Outbox Pattern

Your order service does two things when an order is placed: (1) write the order to the database and (2) publish an "OrderPlaced" event to Kafka. Simple enough.

But what happens when the database write succeeds and Kafka is down? The order is saved, but no event is published. The email service never sends the confirmation. The inventory service never decrements stock. The system is inconsistent.

What if you reverse the order, publish to Kafka first, then write to the database? If the database write fails, you've published an event for an order that doesn't exist. Downstream services react to a phantom order.

This is the dual-write problem. You need to update two separate systems (database and message broker) atomically. But they don't share a transaction. You can't wrap a Postgres write and a Kafka publish in the same ACID transaction.

The transactional outbox pattern solves this by converting the two-system write into a single-system write. Instead of publishing to Kafka directly, you write the event to an "outbox" table in the same database, in the same transaction as the business data. A separate process reads the outbox table and publishes to Kafka. Since the business write and the outbox write are in the same database transaction, they're atomic, both happen or neither does.

Step 1: Write business data + outbox entry in one transaction.

BEGIN TRANSACTION;

INSERT INTO orders (id, user_id, amount, status)
VALUES ('order-123', 'user-456', 99.99, 'placed');

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, created_at)
VALUES (
 'evt-789',
 'Order',
 'order-123',
 'OrderPlaced',
 '{"orderId":"order-123","userId":"user-456","amount":99.99}',
 NOW()
);

COMMIT;

If the transaction commits, both the order and the event are saved. If it rolls back, neither is. No inconsistency possible.

Step 2: A relay process publishes outbox entries to the message broker.

Two approaches:

Polling publisher: A background job polls the outbox table every N milliseconds, reads unpublished rows, publishes them to Kafka, and marks them as published.

// Every 100ms
const events = await db.query(
 'SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100'
);
for (const event of events) {
 await kafka.publish(event.event_type, event.payload);
 await db.query('UPDATE outbox SET published = true WHERE id = $1', [event.id]);
}

Change Data Capture (CDC): A tool like Debezium watches the database transaction log (WAL in Postgres, binlog in MySQL) and automatically publishes new outbox rows to Kafka in real time. No polling needed, near-zero latency.