We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
A plan and set of procedures for restoring systems after a catastrophic failure. Defined by RPO (how much data you can lose) and RTO (how long you can be down).
In short
Disaster recovery is the set of plans, tools, and procedures for restoring your systems and data after a catastrophic failure like a region outage, ransomware attack, or accidental mass deletion. It is measured by two numbers: RPO, the maximum amount of data you can afford to lose (often expressed in minutes or hours), and RTO, the maximum time you can be down before recovery completes.
In short
Disaster recovery is the set of plans, tools, and procedures for restoring your systems and data after a catastrophic failure like a region outage, ransomware attack, or accidental mass deletion. It is measured by two numbers: RPO, the maximum amount of data you can afford to lose (often expressed in minutes or hours), and RTO, the maximum time you can be down before recovery completes.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Disaster Recovery across a run of lessons, not one page. These 4 alone are about 109 minutes of step-by-step reading, every one with a quiz.
Plan and execute recovery from catastrophic failures. DR strategies from backup-restore to active-active.
How a GSLB decides which region a user reaches, and why the decision takes minutes to arrive even when it is made in microseconds.
Transfer complete log files from primary to standby, the oldest and simplest replication method.
Cross-region (cross data center) replication explained: sync vs async, replication lag, RPO and RTO, failover loss, and what AWS, Spanner and CockroachDB do.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Disaster Recoverypay once,
yours for life
Disaster recovery, often shortened to DR, is what you do when an entire system goes down hard and normal failover is not enough. A single crashed server is handled by redundancy and health checks. A disaster is bigger: an AWS region becomes unreachable, a database gets corrupted and replicates the corruption everywhere, a fire takes out a datacenter, or someone runs DROP TABLE on production. DR is the deliberate plan to bring service back from a known-good copy.
Two numbers define every DR plan. RPO, the Recovery Point Objective, is how far back in time you are willing to lose data. An RPO of 15 minutes means your last backup or replica can be at most 15 minutes behind, so a disaster loses at most 15 minutes of writes. RTO, the Recovery Time Objective, is how long the business can tolerate being down. An RTO of 1 hour means service must be fully restored within an hour of the incident.
RPO and RTO are business decisions before they are technical ones. A bank's payment ledger might demand an RPO of zero and an RTO of minutes. An internal analytics dashboard might happily accept an RPO of 24 hours and an RTO of a day. The cost of the DR setup scales steeply with how close to zero you push both numbers.
The foundation of DR is a copy of your data that lives somewhere the disaster cannot reach, usually a different geographic region or cloud account. How fresh that copy is determines your RPO. Daily snapshots give an RPO of up to 24 hours. Continuous replication of the database transaction log gives an RPO of seconds. Tools like PostgreSQL streaming replication, AWS RDS cross-region read replicas, and database log shipping all exist to shrink RPO.
RTO is determined by how much you have pre-built. The industry groups DR strategies into four tiers. Backup and restore is the cheapest: you keep backups in another region and rebuild everything during the disaster, giving an RTO of hours to days. Pilot light keeps a minimal core running, like a replicated database with the application servers switched off, and you scale up on failover. Warm standby runs a smaller but always-on copy of the full stack that you scale to full size. Active-active runs full production in two or more regions at once, so failover is close to instant.
Failover itself is usually a DNS or load balancer change that redirects traffic to the healthy region, combined with promoting the standby database to primary. The hard parts are detecting the disaster correctly, avoiding split-brain where two regions both think they are primary, and making sure the failover path has actually been tested.
Match the strategy to the RPO and RTO the business will pay for. Backup and restore is right for non-critical systems where a few hours of downtime is acceptable and you want to minimize standing cost. Pilot light and warm standby suit important systems that need recovery in minutes but cannot justify running two full production stacks. Active-active is for systems where downtime directly costs revenue or trust, like payments, ad serving, or healthcare records.
The central trade-off is cost versus recovery speed. Active-active can roughly double your infrastructure bill and is genuinely hard to build because the application must tolerate writes in multiple regions and resolve conflicts. Backup and restore is cheap but leaves you exposed to long outages and the risk that a backup you never tested fails to restore.
The most common failure is a DR plan that exists only on paper. A backup that has never been restored is not a backup, it is a hope. Mature teams run regular DR drills, sometimes called game days, where they deliberately fail over to the standby region and measure whether they actually hit their RTO and RPO. Netflix popularized this discipline with its Chaos Monkey and later region-evacuation exercises.
Imagine a payments company running its primary database in AWS us-east-1 with an RPO of 5 minutes and an RTO of 15 minutes. They keep a cross-region read replica in us-west-2 that lags the primary by 1 to 2 seconds, well inside their 5 minute RPO. Application servers run in both regions, but us-west-2 normally serves zero traffic to save cost, a warm standby setup.
One morning us-east-1 has a network partition and the primary becomes unreachable. Automated monitoring confirms the region is down, not just one host. The on-call engineer triggers the runbook: promote the us-west-2 replica to primary, scale up the us-west-2 application fleet, and update Route 53 to point the domain at us-west-2. Traffic shifts and the company is back online in 11 minutes, having lost about 2 seconds of in-flight transactions.
When us-east-1 recovers, they do not simply switch back. They rebuild us-east-1 as the new standby, verify it is replicating from the now-primary us-west-2, and only then plan a controlled failback during a quiet window. Rushing the failback is how teams turn one disaster into two.
Amazon Web Services
Provides the building blocks for most DR plans: cross-region S3 replication, RDS cross-region replicas, EBS snapshots, and Route 53 health-check failover.
Netflix
Runs active-active across multiple AWS regions and regularly evacuates an entire region in production drills to prove it can fail over without customer impact.
PostgreSQL
Streaming replication and write-ahead-log shipping let teams keep a hot standby seconds behind the primary, pushing RPO close to zero.
Cloudflare
Its global anycast network and load balancing route traffic away from failed datacenters automatically, acting as the traffic-steering layer of many DR designs.
RPO is how much data you can lose, measured in time before the disaster. An RPO of 15 minutes means recovery may lose up to 15 minutes of writes. RTO is how long you can be down, measured forward from the disaster. An RTO of 1 hour means service must be restored within an hour. RPO is about your backup or replication frequency; RTO is about how fast your failover process runs.
No. High availability handles small, expected failures like one server crashing, usually within the same datacenter or region, and keeps running with no human action. Disaster recovery handles large, rare failures like losing an entire region or corrupting your whole database, and often involves a deliberate failover to a separate location. Most serious systems need both.
A backup is necessary but not sufficient. A backup you have never restored is unproven, and a backup stored in the same region as production can be lost in the same disaster. Real DR requires backups in a separate location, a tested restore procedure, and a known RTO. Teams should run restore drills on a schedule, not just trust that backups exist.
Four tiers, from cheapest and slowest to most expensive and fastest: backup and restore (rebuild from backups, RTO in hours), pilot light (keep a minimal core running, scale up on failover), warm standby (a smaller always-on copy of the full stack), and active-active (full production in multiple regions at once, near-instant failover).
At least quarterly for critical systems, and after any major architecture change. Testing means actually failing over to the standby and measuring whether you hit your RTO and RPO, not just reading the runbook. Companies like Netflix run regular game days that deliberately take down infrastructure to keep recovery muscle memory sharp.
Global Server Load Balancing (GSLB)
How a GSLB decides which region a user reaches, and why the decision takes minutes to arrive even when it is made in microseconds
Log Shipping
Transfer complete log files from primary to standby, the oldest and simplest replication method
Cross-Region Replication: Sync vs Async, Lag, RPO and the Speed of Light
Cross-region (cross data center) replication explained: sync vs async, replication lag, RPO and RTO, failover loss, and what AWS, Spanner and CockroachDB do.
DR Testing
Regularly test your disaster recovery plan, because untested plans fail when you need them most
Availability
The percentage of time a system is operational and accessible. Measured in 'nines', 99.99% availability means about 52 minutes of downtime per year.
Redundancy
Duplicating critical components or functions so that if one fails, a backup takes over. The reason planes have two engines and databases have replicas.
Replication
Keeping copies of the same data on multiple servers. Improves read performance and provides fault tolerance if one server goes down.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Disaster Recovery" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Disaster Recovery lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Disaster Recovery in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
