Runbook
A documented set of step-by-step procedures for handling specific operational tasks or incidents. Good runbooks reduce MTTR by giving on-call engineers a clear action plan.
What is Runbook?
In short
A runbook is a documented, step-by-step procedure that tells an engineer exactly what to do to handle a specific operational task or incident, such as restarting a stuck service, failing over a database, or rotating an expired certificate. It turns tribal knowledge into a checklist anyone on call can follow at 3 AM without guessing.
What a runbook actually is
A runbook is a written set of instructions for one operational situation. It names the situation up front (for example, payment API returning 500s, or replication lag over 30 seconds), then lists the exact steps to diagnose and fix it. Good ones include the commands to run, the dashboards to open, the expected output at each step, and what to do if a step fails.
The point is to remove judgment under pressure. When a pager fires at 3 AM, the on-call engineer should not have to remember which host to SSH into or which feature flag to toggle. The runbook tells them. This is why runbooks directly reduce MTTR, the mean time to recovery, often by the largest single factor in an incident.
Runbooks are different from architecture docs or postmortems. An architecture doc explains how the system is built. A postmortem explains what went wrong after the fact. A runbook is the in-the-moment action plan. The best teams treat a postmortem action item as incomplete until it produces or updates a runbook.
How runbooks work in practice
A typical runbook has a fixed shape: a title that matches the alert that triggers it, a short symptom description, a severity hint, a list of numbered diagnostic steps, the remediation steps, and an escalation path with names and Slack channels. Many teams link the runbook URL directly inside the alert payload so the engineer is one click away the moment they get paged.
Steps should be copy-pasteable and idempotent where possible. Instead of writing restart the worker, write the literal kubectl rollout restart deployment/payments-worker -n prod command and the kubectl rollout status command to confirm it worked. Each step should state the expected result so the engineer knows whether to continue or escalate.
Over time, well-understood runbooks get automated. A manual runbook for clearing a full disk becomes a script, then a scheduled job, then a self-healing automation that runs without a human. This progression from documented to automated is the foundation of what Google's SRE practice calls reducing toil.
When to use one and the trade-offs
Write a runbook for any failure mode that has happened more than once, anything that pages a human, and any procedure that is risky or hard to remember (database failover, secret rotation, region evacuation). If a task is rare and trivial, a runbook may be overkill. If it is rare but catastrophic, you absolutely want one rehearsed in advance.
The main trade-off is staleness. A runbook that points to a renamed service or a deleted dashboard is worse than none, because it sends a stressed engineer down a dead end. Teams fight this by versioning runbooks in Git next to the code, reviewing them in incident postmortems, and running game days where engineers execute the runbook on purpose to catch rot.
Another failure is overconfidence in the steps. A runbook is a starting point, not a substitute for understanding. It should say why each step matters so an engineer can adapt when reality does not match the script, which during a real incident it often will not.
A concrete example
Imagine an alert fires: API error rate above 5 percent for 5 minutes. The runbook linked in the PagerDuty alert opens with the symptom, then step one: open the error rate dashboard and confirm which endpoint is failing. Step two: check the deploy log to see if a release went out in the last 30 minutes. If yes, step three: run the rollback command, which is spelled out literally, and watch the error rate return to baseline within 2 minutes.
If no recent deploy, the runbook branches: check the database connection pool dashboard, check downstream dependency health, and if both are clean, escalate to the database on-call with this specific Slack command. Every branch ends in either resolved or escalate to a named owner.
That structure is what lets a junior engineer who has never seen this failure resolve it in minutes instead of paging three senior people and losing an hour. The knowledge that used to live in one person's head now lives in a document the whole team can run.
Where it is used in production
Google SRE
Pioneered playbooks linked from every alert; their SRE books treat runbooks as the bridge from manual toil to full automation.
PagerDuty
Lets teams attach a runbook URL to each alert and ships PagerDuty Runbook Automation to execute documented steps with one click.
AWS
Systems Manager Automation runbooks codify operational procedures as executable documents that can patch, restart, or remediate resources.
GitLab
Publishes its entire incident and operational runbooks openly in a public repository, versioned alongside infrastructure code.
Frequently asked questions
- What is the difference between a runbook and a playbook?
- People often use them interchangeably. When teams do distinguish, a runbook is a procedure for one specific task (restart this service), while a playbook is broader and covers a whole class of incidents or a decision tree across several runbooks. Google uses playbook for the alert-linked document; AWS uses runbook for the executable one. The exact word matters less than whether the steps are clear and current.
- How does a runbook reduce MTTR?
- MTTR is the time from when an incident starts to when it is resolved. A large part of that time is the engineer figuring out what to do. A runbook removes that diagnosis-and-recall delay by handing them the exact steps, so they spend minutes executing instead of an hour investigating or paging others. It is usually the cheapest way to cut recovery time.
- Should runbooks be automated?
- Eventually, yes, for the well-understood ones. The healthy path is document the manual steps, then script them, then trigger the script automatically. Tools like AWS Systems Manager and PagerDuty Runbook Automation do this. Keep a human in the loop for anything risky or ambiguous, and keep the written explanation even after automating so people still understand what the automation does.
- Where should runbooks live?
- Store them in version control next to the code and infrastructure they describe, so changes get reviewed and the history is auditable. Then surface them where they are needed: link the runbook URL inside the alert itself so the on-call engineer reaches it in one click instead of searching a wiki during an outage.
- How do you keep runbooks from going stale?
- Tie updates to real events. Review and fix the runbook during every postmortem that used it, run periodic game days where engineers execute it against a real or staging system to surface broken steps, and treat a failed step during an incident as a bug to file. A runbook nobody has run in a year should be assumed wrong until verified.
Learn Runbook hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Runbook as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
MTTR
Mean Time To Recovery: the average time from when a failure is detected to when the service is restored. A key reliability metric that drives investment in automation and runbooks.
Toil
Manual, repetitive, automatable operational work that scales linearly with service size. SRE teams aim to keep toil below 50% of their time and automate the rest.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.