We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In April 2022, Atlassian ran a routine cleanup script that accidentally deleted data for 400 Jira Cloud and Confluence customers. They had backups. They had an automation framework. What they did not have was a rehearsed restore process that had ever been tested at the scale of 400 simultaneous site recoveries.
The stated RTO was hours. The actual recovery took up to 14 days for some customers. The gap between the stated number and the real number was exactly the gap between having a backup strategy and having a tested restore process.
Automated verification proves the backup file is restorable in a sandbox. Restore testing proves the team can execute the full recovery under their actual operational conditions: with the real runbook, the real infrastructure, the real communication channels, and the real time pressure. These are different tests and both are required.
A restore test is a full dress rehearsal of the recovery process, timed and scored. It measures what automated verification cannot: human coordination, runbook accuracy, infrastructure readiness, and real end-to-end RTO.
| What It Tests | What Automated Verification Cannot |
|---|---|
| Runbook accuracy | Dead links, renamed Slack channels, deleted Confluence pages |
| Team coordination | Who leads, who fixes, who communicates with customers |
| Infrastructure readiness | IAM permissions, network routes, entries for the restored environment |
| Application compatibility | Migrations, collations, extension versions, connection strings |
| Real RTO | The actual clock time from "page received" to "users can use the system" |
The output of a restore test is not pass/fail. It is a measured RTO number, a list of issues found, and tracked action items. The measured RTO is almost always larger than the team expects, sometimes by 2x or 3x.
Restore testing comes in three forms: scheduled drills (quarterly, planned, announced), game days (unannounced, surprise, stressful), and tabletop exercises (discussion-only, no real infrastructure). The first two produce real RTO measurements. The third produces conversation but no data.