DR Testing

In 2019, a mid-size fintech ran its first live DR drill after two years of operating their disaster recovery plan. The runbook said failover would complete in 15 minutes. The drill took 47 minutes. Three scripts failed because IAM roles had been changed since the runbook was written. The replica promotion step hung because a connection pool parameter had been updated in production but not in the DR region. The smoke test suite had not been maintained and tested endpoints that no longer existed.

Every one of those problems was invisible on paper. The runbook looked correct. The architecture diagram was accurate. The only way to find these gaps was to actually execute the failover under conditions that resembled production. That is DR testing.

An untested DR plan has a specific failure mode: it works perfectly in theory and falls apart on first contact with reality. The theory does not account for configuration drift, personnel changes, undocumented dependencies, or the simple fact that scripts rot when they are not run. Google's SRE team internally estimates that an unexercised DR plan loses roughly 10% of its effectiveness per quarter as the production environment drifts away from the plan's assumptions.

DR testing is the practice of regularly executing your disaster recovery procedures to verify they still work, measure their actual timing against RTO and RPO targets, and surface the gaps that only appear under real conditions.

DR testing is not a single activity. It is a spectrum of exercises with increasing realism, increasing cost, and increasing value. Mature programs use all four levels, not just the one they are most comfortable with.

What This Means in Production

Document review is the cheapest test. An engineer reads the runbook and flags steps that reference services, credentials, or configurations that no longer exist. This catches the most obvious drift, but it cannot catch timing problems, capacity gaps, or human coordination failures. Run it monthly. Cost: one engineer-hour.

Tabletop exercises put 4 to 8 people in a room with a scripted scenario and a facilitator. Nobody touches a system. The facilitator reads a failure narrative and the team walks through their response step by step. The output is always a list of unanswered questions: who pages the CDN vendor at 3 AM, which engineer holds the cross-region KMS key, how is the status page updated when the dashboard tool itself is down. Stripe runs quarterly tabletops that average two hours and produce roughly a dozen documentation fixes each round.

Simulated failover uses a non-production environment that mirrors production topology. The team runs the full runbook against synthetic data. This catches script failures, permission gaps, and automation bugs without risking customer traffic. Most teams start here before graduating to live drills.

Live production drills actually shift real traffic. Netflix calls them Chaos Kong exercises. The team kills a region or availability zone and measures actual RTO and RPO against targets. This is the only test that proves the plan works under real load with real dependencies. It is also the only test that can cause a real outage if something goes wrong, which is why blast radius limits are mandatory.