Asynchronous Replication

When MySQL, Postgres, or MongoDB replicates data, the first design decision is: does the primary wait for replicas before confirming the write?

If it waits, the write is slower but safer. If it doesn't wait, the write is fast but there's a window where the data exists on only one machine. If that machine dies during that window, the data is gone.

Asynchronous replication is the "don't wait" approach, and it's the default mode for almost every database. MySQL defaults to async replication. Postgres streaming replication defaults to async. MongoDB replica sets default to async. The reason is simple: it's fast.

Amazon Aurora, which powers a huge chunk of AWS's RDS workloads, uses a form of async replication internally. Shopify's entire database fleet runs async replication across their 900+ MySQL shards. For the vast majority of workloads, the speed benefit is worth the tiny risk.

But that tiny risk isn't zero. In 2020, a financial services company lost 30 seconds of transaction data during a primary failure because their async replicas hadn't caught up yet. Those 30 seconds contained hundreds of stock trades that had to be manually reconciled from exchange logs. "Async is fine for most things" doesn't mean "async is fine for everything."

The mechanics are straightforward:

1. A client sends a write (INSERT, UPDATE, DELETE) to the primary
2. The primary writes it to its local storage and WAL (Write-Ahead Log)
3. The primary immediately confirms the write back to the client. Done. Acknowledged.
4. In the background, the primary ships the WAL entries to replicas
5. Replicas receive and apply the changes at their own pace

The key point is step 3. The client gets "success" before any replica has seen the data. The replication happens asynchronously, hence the name.

This means there's always a window, however small, where:

• The primary has the data
• The replicas don't
• If the primary dies during this window, that data is lost

Under normal conditions, this window is measured in milliseconds. Replicas are usually less than 100ms behind. But under heavy write load, network issues, or replica maintenance, the window can grow to seconds or minutes.

The fundamental guarantee: Your write is durable on the primary's disk. It is NOT guaranteed to be on any replica. If the primary's disk is destroyed (rare but possible), unreplicated writes are lost.