We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
On January 31, 2017, a GitLab engineer ran a database deletion command on the wrong server. The primary PostgreSQL database was wiped. The team turned to their replicas. Every pipeline had been silently broken for months. The most recent usable backup was six hours old. Six hours of Git pushes, merge requests, and CI pipeline data vanished permanently.
GitLab published one of the most transparent postmortems in the industry. The technical root cause was human error. The systemic root cause was that nobody had measured or alerted on actual replication lag. The team believed their RPO was minutes. Their actual RPO, measured during the incident, was six hours.
Recovery Point Objective is the maximum amount of data loss a business can tolerate after a failure. RTO asks how quickly you recover. RPO asks how much you lose. A 5-second RPO means that after any failure, you can restore to within 5 seconds of the last committed state. A 6-hour RPO means you accept losing everything written in the last 6 hours.
The distinction matters because RPO determines your replication strategy, your backup frequency, and your storage costs. A 1-second RPO requires synchronous replication. A 24-hour RPO can survive on nightly backups. The cost difference between those two is an order of magnitude, and the right answer depends entirely on what the data is worth.
A payments ledger and a clickstream pipeline live on the same database cluster. Losing a single committed payment transaction triggers regulatory investigation and customer lawsuits. Losing five minutes of clickstream data means a slightly less accurate recommendation model tomorrow. Treating both with the same RPO wastes money on one and endangers the other.
RPO is a property of the data, not of the infrastructure. The same cluster can run synchronous quorum for the ledger table and asynchronous replication for the analytics tables. Spanner formalized this with per-row replication factors. YugabyteDB and CockroachDB followed. The insight took the database industry decades to internalize.
Square, Stripe, and Wise all run synchronous quorum replication for their core ledger within region. The cost of sync, typically 1 to 5 milliseconds per commit, is budgeted into the user experience. For money, regulation and double-entry accounting both assume durability, and the legal exposure of losing a committed transaction dwarfs the latency penalty.
Snowflake and ClickHouse analytical clusters typically run with tolerant RPOs because the source data can be replayed from upstream . Losing five minutes of clickstream events is recoverable by re-ingesting from the source of truth. This deliberate design choice lets analytics scale to petabytes without paying sync replication tax on every write.
The mature pattern is tiered replication tuned per data class. Customer money is tier 1 sync. Audit logs are tier 2 async. Derived analytics are tier 3 with no replica at all because they can be rebuilt from source.