RTO

A team commits to a 15-minute Recovery Time Objective. They buy the infrastructure, write the runbook, and feel confident. The first real failover drill takes 22 minutes.

Seven minutes over target, which means the commitment is broken. But nobody can explain where the time went because they never broke the 15 minutes into phases. Was it detection? Decision-making? The actual failover? Verification? Nobody measured the phases independently, so the 15-minute number was a wish, not a budget.

AWS S3 in February 2017 took four hours to recover from an incident partly because nobody had measured index subsystem restart time at that scale. The total RTO claim was based on assumptions about individual phase durations that had never been tested.

Recovery Time Objective is the maximum acceptable duration between a failure event and full service restoration. The number itself is meaningless without a phase breakdown that assigns minutes to detection, decision, execution, and verification. The breakdown is where the engineering happens. The top-level number is where the business negotiation happens. Confusing the two is how teams end up with commitments they cannot meet.

A team commits to a 15 minute RTO. The first real failover takes 22 minutes. Nobody can explain where the time went because they never measured the phases. The number was a wish, not a budget.

An RTO target is a sum of independent phases: detection, decision, execution, and verification. A 15 minute target looks identical on paper whether 14 of those minutes are detection or 14 are data restore, but the remediation strategies are completely different. AWS S3 in February 2017 took four hours to recover in part because the index subsystems had never been restarted at that scale and nobody had budgeted restart time explicitly. The total RTO claim was wrong because no phase was ever measured in isolation.

A realistic 15 minute budget looks like 3 minutes detection, 2 minutes decision, 6 minutes execution, 4 minutes verification. Each phase has its own bottleneck and its own fix. Detection dominates if monitoring is weak. Decision dominates if the runbook requires waking a VP. Execution dominates if the automation is manual. Verification dominates if there is no automated smoke test suite. Watch the third step where a real drill blows the budget because of a confused chain of command, not because of anything technical.

What This Means in Production

Netflix calls out organizational RTO blowups as the most common finding from their Chaos Kong exercises. The technology works. The humans get confused about who is in charge. Designating an automatic incident commander, paging them on a separate channel, and giving them explicit authority to declare and execute solves more RTO problems than any automation work.

Detection time hides in alerting topology. A 60 second metric scrape plus a 60 second alert evaluation plus a 60 second pager delivery is 3 minutes minimum before any human knows. Cutting this requires synthetic checks running every 10 seconds with their own dedicated alerting path, which Datadog and Honeycomb both market as a separate product.

Execution time hides in manual steps. Any runbook step that says someone has to log in and run a command adds 30 to 90 seconds. A fully scripted failover with idempotent retries cuts execution time by an order of magnitude. Spinnaker pipelines and GitHub Actions runbook orchestration both formalize this.

Verification time hides in trust. If the team does not trust the automation, they keep checking. Smoke tests that exercise real customer paths and report green within 60 seconds eliminate the trust gap. Without them the IC stares at dashboards waiting for confidence that may never come.

The lesson is that RTO is socio-technical. Any phase you do not measure expands silently until it dominates.