Distributed Transactions Explained: Why They Are Hard

A transaction is a group of changes that must happen together or not at all. On one database this is easy. You move $500 from checking to savings. That is two updates in one transaction. The database makes sure both happen, or neither does.

Now put the checking account on a database in Virginia and the savings account on a database in Oregon. The two databases know nothing about each other. You still need both updates or neither. That is a distributed transaction: one logical change whose parts live on different machines.

If Virginia commits and Oregon fails, $500 leaves checking and never reaches savings. The money is gone. Engineers have worked on this problem since the 1970s. Every answer is some kind of coordination protocol, and each one trades safety, liveness and speed differently.

The lesson uses an online order to show what goes wrong. One PlaceOrder request must write to three independent databases: orders, inventory and payments.

The order database commits and acknowledges. The inventory database reserves one unit and acknowledges. Then the commit to the payment database times out. Maybe the write landed and the network lost the reply. Maybe it never landed. The coordinator has no way to tell.

Both choices now hurt. If the coordinator assumes failure and retries, the customer may be charged twice. If it assumes success and moves on, the customer may never be charged. It cannot simply roll back either, because two databases already said yes.

The client is stuck too. From its side, an error response is ambiguous. The order may exist and the card may have been charged. Without a real protocol, the only safe path is an idempotent write plus a reconciliation job. An idempotent write gives the same result if it runs twice. The reconciliation job fixes mismatches later. The lesson suggests a unique idempotency key per attempt, so the payment service can ignore a duplicate.

On paper, 2PC is correct. In production, its weak point is the locking window. Every participant holds row locks from prepare until commit. The lesson's example is a 2PC across four services with 100 ms of cross-region latency. That holds locks for at least 400 ms. Under load, every write to the same rows lines up behind those locks, and throughput collapses.

PostgreSQL shows the same risk in its own documentation. A transaction left in the prepared state keeps every lock it held. The docs warn against leaving prepared transactions open for long.

A saga avoids locks, but moves the work into business logic. The lesson's order saga is: reserve inventory, charge payment, create shipment, confirm order. Each step needs an undo: release inventory, refund payment, cancel shipment. Some undos are easy. A card capture can be refunded. Others are hard or impossible. You cannot cancel a shipment that is already on a truck. Every edge case becomes a conversation with the business team.

One mistake to avoid: making each saga step its own 2PC transaction. You get the lock fan-out of 2PC and the visible half-states of a saga. Pick one model and keep to it.

Before you pick a protocol, check whether you need one. Design the data model so that changes that must be atomic live on the same node. Then the transaction is local and the WAL does the work.

This matters most when a monolith is split into microservices. The database gets split too. An order that once updated three tables in one local transaction now touches three services. Many teams find that every business operation now needs a saga. Some move the data back into one distributed SQL database instead, because running sagas across ten services can cost more than running one database.

A smaller version of the same problem is the dual write: saving a row and publishing an event as one step. The outbox pattern solves that without any distributed transaction, by writing the event to a table in the same local transaction. The transactional outbox lesson walks through it.

To see the partial commit, the protocol comparison and the saga vs 2PC trade-offs step by step, the Distributed Transactions lesson is the next step. It leads into the two-phase commit, three-phase commit and saga lessons.