We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Mean Time To Recovery: the average time from when a failure is detected to when the service is restored. A key reliability metric that drives investment in automation and runbooks.
In short
MTTR (Mean Time To Recovery) is the average time it takes to restore a service after a failure, measured from the moment the failure is detected to the moment the system is working again. A low MTTR means outages are short and customers barely notice; a high MTTR means failures turn into long, painful incidents.
In short
MTTR (Mean Time To Recovery) is the average time it takes to restore a service after a failure, measured from the moment the failure is detected to the moment the system is working again. A low MTTR means outages are short and customers barely notice; a high MTTR means failures turn into long, painful incidents.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers MTTR across a run of lessons, not one page. These 4 alone are about 57 minutes of step-by-step reading, every one with a quiz.
How long it takes to fix the problem, from acknowledgment to all-clear.
How long it takes to realize something is broken, the gap between failure and awareness.
How often things break, the frequency metric that shapes your reliability investment.
Restore a database to any moment in the past using WAL and base backups.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with MTTR (Mean Time to Resolve)pay once,
yours for life
MTTR is a reliability metric. You take every incident over a period, add up how long each one took to recover from, and divide by the number of incidents. If you had 4 outages last quarter that took 10, 20, 30, and 40 minutes to fix, your MTTR is 25 minutes.
The clock usually starts when the failure is detected, not when the customer first felt it, and stops when the service is fully restored. That detection gap matters: if your monitoring takes 15 minutes to notice a problem, that delay is often tracked separately as MTTD (Mean Time To Detect). Some teams measure MTTR from the moment of failure instead of detection, so always check which definition a team is using before comparing numbers.
The letters get reused for several related ideas. Recovery and Repair are the two common readings of the second R, and people also talk about Mean Time To Respond and Mean Time To Resolve. They are not the same. Repair counts only the hands-on fixing time; Recovery includes everything from alert to service restored. Pick one definition and stick to it.
MTTR is mostly an operations and automation problem, not a code problem. The biggest wins come from cutting the time spent figuring out what broke, not the time spent typing the fix. Good observability, clear dashboards, and alerts that point at the real cause shrink the investigation phase, which is usually where the minutes pile up.
Automation is the next lever. Health checks that auto-restart a crashed container, load balancers that pull an unhealthy node out of rotation, and orchestrators like Kubernetes that reschedule failed pods all recover without a human in the loop. The fastest recovery is the one nobody had to wake up for. When humans are needed, tested runbooks turn a 40 minute scramble into a 5 minute checklist.
The other half is making rollback cheap and instant. If a bad deploy can be reverted with one command or an automatic canary rollback, recovery is near-instant. Blue-green deploys, feature flags you can flip off, and immutable infrastructure all exist largely to make MTTR small.
MTTR pairs with MTBF (Mean Time Between Failures) to describe reliability. Availability is roughly MTBF divided by MTBF plus MTTR, so you can hit a high uptime number two ways: fail rarely, or recover fast. For most internet services, investing in fast recovery is cheaper and more realistic than trying to never fail, because failures are inevitable at scale.
The metric has sharp edges. An average hides the worst cases, so one 6 hour outage buried among many quick fixes can look fine on paper while having burned your customers. Tracking the median and the worst-case alongside the mean gives a truer picture. A small sample size also makes MTTR noisy: with only three incidents, a single bad one swings the whole number.
Chasing MTTR too hard can backfire. Teams under pressure to mark incidents resolved fast may slap on a quick workaround instead of a real fix, which leads to the same outage next week. MTTR is most honest when paired with a real postmortem process and a count of repeat incidents.
Say a payments API starts returning 500 errors at 2:14 because a new build has a bad config. Monitoring fires a PagerDuty alert at 2:16 (MTTD is 2 minutes). The on-call engineer opens the dashboard, sees error rate spiking right after a deploy, and triggers an automated rollback at 2:21. Traffic recovers by 2:24. MTTR for this incident is 8 minutes measured from detection.
Now imagine the same outage without good tooling: no deploy correlation on the dashboard, no one-click rollback, and the engineer has to SSH into boxes to read logs. The same root cause could take 45 minutes to find and fix. Same failure, same code, but MTTR is five times worse purely because of the recovery process.
This is why companies report MTTR in their SRE programs and tie it to error budgets. Google's SRE practice popularised the idea that you should engineer for fast, automated recovery rather than assume failures won't happen.
Kubernetes
Self-healing controllers restart crashed pods and reschedule them onto healthy nodes automatically, cutting recovery time to seconds with no human involved.
PagerDuty
Routes alerts to the right on-call engineer in seconds and tracks incident timelines, so teams measure and shrink the detection-to-resolution window.
AWS
Auto Scaling health checks replace failing EC2 instances and load balancers drain unhealthy targets, recovering capacity without manual intervention.
Google SRE
Pioneered error budgets and automated rollback culture, treating fast recovery as the primary path to high availability rather than zero failures.
Most commonly Mean Time To Recovery, the average time to restore a service after a failure. The second R is also read as Repair, and related terms include Mean Time To Respond and Mean Time To Resolve, so confirm which definition a team uses before comparing numbers.
Add up the total recovery time across all incidents in a period, then divide by the number of incidents. If three outages took 10, 20, and 30 minutes, MTTR is 60 divided by 3, which is 20 minutes.
MTTR measures how fast you recover from a failure; MTBF (Mean Time Between Failures) measures how often failures happen. Availability depends on both: rare failures or fast recovery can each raise uptime.
There is no universal number; it depends on the service. High-performing teams in DevOps research often recover from incidents in under an hour, and for critical user-facing services many aim for minutes. Compare against your own trend, not an absolute target.
Improve observability so you find the cause faster, automate recovery with health checks and orchestrators, make rollbacks one command, and keep tested runbooks. Most of the time saved comes from faster diagnosis, not faster typing.
MTTD
Mean Time To Detect: the average time between a failure occurring and being noticed. Shorter MTTD means better monitoring and alerting. You can't fix what you don't know is broken.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "MTTR (Mean Time to Resolve)" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the MTTR (Mean Time to Resolve) lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about MTTR in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
