Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
What is Incident Response?
In short
Incident response is the structured process a team follows to detect a security or reliability incident, contain the damage, remove the root cause, recover normal operations, and learn from what happened. It turns a chaotic outage or breach into a repeatable set of steps with defined roles, runbooks, and communication so the same problem does not keep hurting you.
What incident response actually is
An incident is any event that harms, or threatens to harm, the confidentiality, integrity, or availability of a system. A leaked API key, ransomware on a laptop, a database that is down for paying customers, a flood of credential-stuffing logins. Incident response is the discipline of handling those events on purpose instead of improvising every time.
Most teams follow a lifecycle made popular by the SANS and NIST frameworks. NIST 800-61 names four phases: preparation, detection and analysis, containment plus eradication plus recovery, and post-incident activity. SANS splits it into six steps: preparation, identification, containment, eradication, recovery, and lessons learned. The names differ but the shape is the same loop.
The point is not paperwork. The point is that during a real incident people are tired, scared of making it worse, and unsure who is allowed to pull the plug. A written plan with clear roles answers those questions in advance so the team spends its energy on the problem, not on coordination.
How it works step by step
Preparation happens before anything breaks. You write runbooks for likely scenarios, set up logging and alerting, define who the incident commander is, and run game days where you simulate a breach. If you have no logs you cannot investigate, so preparation is where most of the real work lives.
Detection and triage come next. An alert fires from something like a SIEM (Splunk, Elastic, or a cloud tool like AWS GuardDuty), a customer reports an outage, or a monitor catches an error spike. Someone declares an incident and assigns a severity. A SEV1 page that wakes people up is different from a SEV3 that waits until morning.
Containment stops the bleeding. You isolate a compromised host from the network, revoke a stolen token, fail over to a healthy region, or rate-limit an abusive client. Containment is deliberately separate from eradication because you often want to keep evidence intact for forensics before you wipe the box.
Eradication removes the root cause, recovery restores service and confirms it is healthy, and the final phase is the postmortem. You write a blameless timeline of what happened, why detection was slow or fast, and concrete action items so the same gap does not reopen.
When to invest and the trade-offs
Every team that runs production systems needs some incident response, but the depth scales with risk. A solo side project might just need backups and a phone alert. A bank or a healthcare platform needs a 24/7 on-call rotation, legal and PR playbooks, breach-notification timers, and tabletop exercises every quarter.
The main trade-off is investment versus speed under fire. Heavy process, formal severity matrices, and approval chains add overhead and can feel like bureaucracy on a quiet day. But the median time to contain a breach is measured in days, and IBM's annual breach report consistently shows that organizations with a tested incident response plan save roughly a million dollars per breach compared to those without one.
A second trade-off is containment versus evidence. Pulling a compromised server offline instantly stops the attacker but can destroy memory and logs a forensic team needs. Mature teams snapshot the disk and capture memory before isolating, so they keep both safety and the ability to answer how the attacker got in.
Watch out for two failure modes: a beautiful plan nobody has ever rehearsed, and blameful postmortems that make engineers hide mistakes. Both quietly destroy the value of the whole exercise.
A concrete example
Say your e-commerce checkout starts throwing 500 errors at 2am. An alert pages the on-call engineer, who declares a SEV2 and becomes incident commander. They open a dedicated Slack channel and a video bridge so all communication has one home.
Detection: dashboards show the payment service is timing out on database calls. Triage points at a connection pool that exhausted after a deploy three hours earlier. Containment: the team rolls back the bad deploy and the error rate drops within minutes, restoring checkout. They do not yet know the exact bug, but customers can buy again.
Eradication and recovery follow over the next day once the offending code is fixed and redeployed safely. Then comes the blameless postmortem: the deploy lacked a pool-size check, the alert fired late because the threshold was wrong, and nobody owned the on-call runbook for that service. Three action items get filed with owners and due dates. That is incident response working as a loop, not a one-off scramble.
Where it is used in production
PagerDuty
Built its whole business around incident response: routing alerts to the right on-call engineer, escalation policies, and live incident timelines.
Google SRE
Codified blameless postmortems and the incident commander role in its widely copied Site Reliability Engineering practices.
Atlassian
Runs and openly documents an incident management process with severity levels and a dedicated incident commander for Jira and Confluence outages.
AWS
Provides GuardDuty for threat detection and Detective plus Security Hub to drive cloud incident investigation and response.
Frequently asked questions
- What is the difference between incident response and disaster recovery?
- Incident response is the full process of handling any harmful event, including security breaches, and covers detection, containment, and learning. Disaster recovery is narrower: it is specifically about restoring systems and data after a major loss like a region outage or data center failure. Disaster recovery is essentially one tool that incident response may reach for during the recovery phase.
- What are the phases of incident response?
- The NIST model uses four: preparation, detection and analysis, containment plus eradication plus recovery, and post-incident activity. The SANS model spells out six: preparation, identification, containment, eradication, recovery, and lessons learned. They describe the same loop with different granularity.
- Who runs an incident?
- An incident commander runs it. This is one person who owns coordination, decisions, and communication during the incident, but is usually not the one typing fixes. Separating command from hands-on work keeps the response from descending into a leaderless scramble where everyone debugs and nobody coordinates.
- Why are postmortems blameless?
- Because people hide mistakes when they expect punishment, which destroys the data you need to actually fix systemic gaps. A blameless postmortem assumes everyone acted reasonably with the information they had and focuses on the process and tooling that let the failure happen, not on naming a culprit. Honest timelines lead to real fixes.
- What is an incident runbook?
- A runbook is a written, step-by-step guide for handling a specific scenario, like a leaked credential or a service being down. It lists who to page, what to check, what commands to run, and how to contain the issue. Good runbooks let a tired on-call engineer at 3am act correctly without having to invent the response from scratch.
Learn Incident Response hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Incident Response as part of a larger topic.
Incident Response
Structured response when things go wrong, roles, communication, and resolution steps
advanced · reliability resilience
Incident Response
Structured processes for detecting, containing, eradicating, and recovering from security incidents
intermediate · security architecture
MTTD (Mean Time to Detect)
How long it takes to realize something is broken, the gap between failure and awareness
intermediate · observability monitoring
MTTA (Mean Time to Acknowledge)
How long until a human says 'I'm on it', the gap between alert and action
intermediate · observability monitoring
MTTR (Mean Time to Resolve)
How long it takes to fix the problem, from acknowledgment to all-clear
intermediate · observability monitoring
See also
Related glossary terms you might want to look up next.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Postmortem
A blameless analysis conducted after an incident to document what happened, why, and how to prevent it from recurring. The most important output is the list of action items.
Penetration Testing
Authorized simulated attacks against a system to find security vulnerabilities before real attackers do. White-hat hackers probe for weaknesses in a controlled environment.
Threat Modeling
A structured process for identifying security threats, attack surfaces, and mitigations during system design. STRIDE and DREAD are common frameworks.
Security Audit
A systematic evaluation of a system's security posture against standards and best practices. Covers access controls, encryption, logging, vulnerability management, and compliance.