We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
A documented set of step-by-step procedures for handling specific operational tasks or incidents. Good runbooks reduce MTTR by giving on-call engineers a clear action plan.
In short
A runbook is a documented, step-by-step procedure that tells an engineer exactly what to do to handle a specific operational task or incident, such as restarting a stuck service, failing over a database, or rotating an expired certificate. It turns tribal knowledge into a checklist anyone on call can follow at 3 AM without guessing.
In short
A runbook is a documented, step-by-step procedure that tells an engineer exactly what to do to handle a specific operational task or incident, such as restarting a stuck service, failing over a database, or rotating an expired certificate. It turns tribal knowledge into a checklist anyone on call can follow at 3 AM without guessing.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Runbook across a run of lessons, not one page. These 4 alone are about 80 minutes of step-by-step reading, every one with a quiz.
Automate common operational procedures, from manual checklists to executable workflows.
Three thousand updates took 117 ms sent one at a time and 17 inside one CALL. A function that tries to COMMIT fails outright, and an EXCEPTION block inside a loop took 9 ms to 134 while catching nothing.
Binding a client to a specific server certificate to prevent man-in-the-middle attacks even when a trusted CA is compromised.
The specific measurements that tell you whether your service is healthy, the numbers behind your promises.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Runbook Automationpay once,
yours for life
A runbook is a written set of instructions for one operational situation. It names the situation up front (for example, payment API returning 500s, or replication lag over 30 seconds), then lists the exact steps to diagnose and fix it. Good ones include the commands to run, the dashboards to open, the expected output at each step, and what to do if a step fails.
The point is to remove judgment under pressure. When a pager fires at 3 AM, the on-call engineer should not have to remember which host to SSH into or which feature flag to toggle. The runbook tells them. This is why runbooks directly reduce MTTR, the mean time to recovery, often by the largest single factor in an incident.
Runbooks are different from architecture docs or postmortems. An architecture doc explains how the system is built. A postmortem explains what went wrong after the fact. A runbook is the in-the-moment action plan. The best teams treat a postmortem action item as incomplete until it produces or updates a runbook.
A typical runbook has a fixed shape: a title that matches the alert that triggers it, a short symptom description, a severity hint, a list of numbered diagnostic steps, the remediation steps, and an escalation path with names and Slack channels. Many teams link the runbook URL directly inside the alert payload so the engineer is one click away the moment they get paged.
Steps should be copy-pasteable and idempotent where possible. Instead of writing restart the worker, write the literal kubectl rollout restart deployment/payments-worker -n prod command and the kubectl rollout status command to confirm it worked. Each step should state the expected result so the engineer knows whether to continue or escalate.
Over time, well-understood runbooks get automated. A manual runbook for clearing a full disk becomes a script, then a scheduled job, then a self-healing automation that runs without a human. This progression from documented to automated is the foundation of what Google's SRE practice calls reducing toil.
Write a runbook for any failure mode that has happened more than once, anything that pages a human, and any procedure that is risky or hard to remember (database failover, secret rotation, region evacuation). If a task is rare and trivial, a runbook may be overkill. If it is rare but catastrophic, you absolutely want one rehearsed in advance.
The main trade-off is staleness. A runbook that points to a renamed service or a deleted dashboard is worse than none, because it sends a stressed engineer down a dead end. Teams fight this by versioning runbooks in Git next to the code, reviewing them in incident postmortems, and running game days where engineers execute the runbook on purpose to catch rot.
Another failure is overconfidence in the steps. A runbook is a starting point, not a substitute for understanding. It should say why each step matters so an engineer can adapt when reality does not match the script, which during a real incident it often will not.
Imagine an alert fires: API error rate above 5 percent for 5 minutes. The runbook linked in the PagerDuty alert opens with the symptom, then step one: open the error rate dashboard and confirm which endpoint is failing. Step two: check the deploy log to see if a release went out in the last 30 minutes. If yes, step three: run the rollback command, which is spelled out literally, and watch the error rate return to baseline within 2 minutes.
If no recent deploy, the runbook branches: check the database connection pool dashboard, check downstream dependency health, and if both are clean, escalate to the database on-call with this specific Slack command. Every branch ends in either resolved or escalate to a named owner.
That structure is what lets a junior engineer who has never seen this failure resolve it in minutes instead of paging three senior people and losing an hour. The knowledge that used to live in one person's head now lives in a document the whole team can run.
Google SRE
Pioneered playbooks linked from every alert; their SRE books treat runbooks as the bridge from manual toil to full automation.
PagerDuty
Lets teams attach a runbook URL to each alert and ships PagerDuty Runbook Automation to execute documented steps with one click.
AWS
Systems Manager Automation runbooks codify operational procedures as executable documents that can patch, restart, or remediate resources.
GitLab
Publishes its entire incident and operational runbooks openly in a public repository, versioned alongside infrastructure code.
People often use them interchangeably. When teams do distinguish, a runbook is a procedure for one specific task (restart this service), while a playbook is broader and covers a whole class of incidents or a decision tree across several runbooks. Google uses playbook for the alert-linked document; AWS uses runbook for the executable one. The exact word matters less than whether the steps are clear and current.
MTTR is the time from when an incident starts to when it is resolved. A large part of that time is the engineer figuring out what to do. A runbook removes that diagnosis-and-recall delay by handing them the exact steps, so they spend minutes executing instead of an hour investigating or paging others. It is usually the cheapest way to cut recovery time.
Eventually, yes, for the well-understood ones. The healthy path is document the manual steps, then script them, then trigger the script automatically. Tools like AWS Systems Manager and PagerDuty Runbook Automation do this. Keep a human in the loop for anything risky or ambiguous, and keep the written explanation even after automating so people still understand what the automation does.
Store them in version control next to the code and infrastructure they describe, so changes get reviewed and the history is auditable. Then surface them where they are needed: link the runbook URL inside the alert itself so the on-call engineer reaches it in one click instead of searching a wiki during an outage.
Tie updates to real events. Review and fix the runbook during every postmortem that used it, run periodic game days where engineers execute it against a real or staging system to surface broken steps, and treat a failed step during an incident as a bug to file. A runbook nobody has run in a year should be assumed wrong until verified.
Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
MTTR
Mean Time To Recovery: the average time from when a failure is detected to when the service is restored. A key reliability metric that drives investment in automation and runbooks.
Toil
Manual, repetitive, automatable operational work that scales linearly with service size. SRE teams aim to keep toil below 50% of their time and automate the rest.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Runbook Automation" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Runbook Automation lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Runbook in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
