We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Data Replication Distribution
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Replication Distribution
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Before streaming existed in Postgres (pre-9.0, released in 2010), physical replication worked by shipping entire WAL files. A WAL segment in Postgres is 16MB by default. The primary would fill up a 16MB segment, then copy the whole file to the replica. The replica would apply it.
See the problem? If the primary had just started a new segment and then crashed, the current segment was not full yet, so it had not been shipped. The replica was potentially 16MB of changes behind, minutes or even hours of transactions, depending on write volume. For a lightly loaded database, a single WAL segment might take hours to fill.
Streaming replication fixed this by shipping WAL records as they are generated, not waiting for a file to fill up. The replica connects to the primary over a persistent connection and receives changes in real-time. Replication lag dropped from "minutes to hours" to "milliseconds."
This is now the standard approach for Postgres replication. When someone says "Postgres replication" without any qualifier, they almost certainly mean streaming replication. The file-based approach (archive_command shipping) is now only used for long-term WAL archiving or disaster recovery to a remote site where a persistent TCP connection is impractical.
The architecture has four moving parts:
A WAL sender process on the primary maintains a persistent connection to each replica. As the primary writes new WAL records, the WAL sender pushes them to connected replicas immediately, record by record, not file by file.
A WAL receiver process on each replica accepts these records and writes them to the replica's local WAL. Then the replica's startup process replays these WAL records against the data files, bringing the replica up to date. If you want to see a real primary and standby set up end to end, with slots, synchronous commit and promotion measured in a lab, my lesson on active-passive configuration walks through every setting.
The key difference from file-based shipping:
| Aspect | File-Based Shipping | Streaming |
|---|---|---|
| Transfer unit | 16MB WAL segment files | Individual WAL records (bytes) |
| Minutes to hours | Milliseconds | |
| Connection | Periodic file copy (rsync, scp) | Persistent connection |
| Overhead | Low (batch transfer) | Slightly higher (continuous stream) |
| Recovery point | Up to 1 WAL segment behind | Typically < 1 second behind |
| Primary crash risk | Lose unfilled segment | Lose only in-flight records |
Streaming replication can be configured as either asynchronous or synchronous. In async mode (default), the WAL sender pushes records and does not wait for the replica to confirm. In sync mode, it waits. All the trade-offs from the earlier lessons apply here.
Replication slots are a Postgres feature (added in 9.4) that tracks which WAL records each replica has received. Without slots, the primary might recycle old WAL segments that a slow replica has not consumed yet. The slot prevents this, guaranteeing the replica can always catch up. But it also means the primary's disk usage grows if a replica falls far behind. A disconnected replica with an active replication slot is a ticking disk bomb.