We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
On April 5, 2022, an Atlassian maintenance script ran a cleanup operation that deleted production data for approximately 400 customer sites. The sites went offline. Recovery took 14 days for the last customers to be restored. Not hours, not a weekend. Fourteen days.
The postmortem revealed that Atlassian had backups. They had replicas. They had runbooks. What they did not have was a tested restoration process that could handle hundreds of customer datasets in parallel under pressure. The team had to rebuild restoration order on the fly while the outage was active, deciding which customers to restore first without a pre-existing priority framework.
Disaster recovery is the discipline of planning, building, and testing the ability to restore full service after a catastrophic failure. The failure could be a deleted database, a flooded data center, a ransomware attack, or a cloud provider region going unreachable. The recovery must bring the system back within the RTO and RPO targets the business has agreed to pay for.
The key word in that definition is "tested." A DR plan that has never been executed end to end is not a plan. It is a document. Atlassian had the document. What they needed was a rehearsed, time-boxed, sequenced recovery procedure that had been run against realistic data volumes before the first real incident arrived.
DR strategy selection is a direct function of RTO budget and infrastructure spend. There are four standard tiers, and each one roughly doubles the cost of the one below it while cutting recovery time by an order of magnitude.
Backup-restore is the floor. S3 versioned snapshots with cross-region give you 24-hour RPO at minimal cost. The problem is RTO: restoring a 500 GB database from S3 takes hours because you are downloading, decompressing, and replaying WAL. This is acceptable for tier 3 and tier 4 services where nobody notices a 4-hour gap.
Pilot light keeps the minimum infrastructure running in the DR region: a database replica, a base AMI, and the networking configuration. When a disaster hits, you scale up from the pilot light. RTO drops to 30 to 60 minutes depending on how much provisioning is automated. AWS Well-Architected explicitly recommends this as the default starting point for most production workloads.
Warm standby runs a scaled-down copy of the production environment in the DR region. The database replica is live, the application servers are running at reduced capacity, and failover can route traffic within minutes. RTO drops to 5 to 15 minutes. Cost is roughly 1.5 to 2x because you are paying for idle capacity that exists solely for disaster scenarios.
Active-active runs full production in two or more regions simultaneously. Traffic flows to both. A region failure means losing capacity, not losing availability. RTO is measured in seconds. Cost is 2 to 3x because every region must handle full load independently. Stripe, Netflix, and Google all run variants of this for their tier 0 services.
The decision is not which strategy is best. The decision is which strategy each service tier deserves based on the dollar cost of its downtime.