We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Security Testing & Operations
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
In short
Incident response is the structured process a team follows to detect a security or reliability incident, contain the damage, remove the root cause, recover normal operations, and learn from what happened. It turns a chaotic outage or breach into a repeatable set of steps with defined roles, runbooks, and communication so the same problem does not keep hurting you.
In short
Incident response is the structured process a team follows to detect a security or reliability incident, contain the damage, remove the root cause, recover normal operations, and learn from what happened. It turns a chaotic outage or breach into a repeatable set of steps with defined roles, runbooks, and communication so the same problem does not keep hurting you.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Incident Response across a run of lessons, not one page. These 4 alone are about 66 minutes of step-by-step reading, every one with a quiz.
Structured response when things go wrong, roles, communication, and resolution steps.
How long it takes to realize something is broken, the gap between failure and awareness.
How long until a human says 'I'm on it', the gap between alert and action.
How long it takes to fix the problem, from acknowledgment to all-clear.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Incident Responsepay once,
yours for life
An incident is any event that harms, or threatens to harm, the confidentiality, integrity, or availability of a system. A leaked API key, ransomware on a laptop, a database that is down for paying customers, a flood of credential-stuffing logins. Incident response is the discipline of handling those events on purpose instead of improvising every time.
Most teams follow a lifecycle made popular by the SANS and NIST frameworks. NIST 800-61 names four phases: preparation, detection and analysis, containment plus eradication plus recovery, and post-incident activity. SANS splits it into six steps: preparation, identification, containment, eradication, recovery, and lessons learned. The names differ but the shape is the same loop.
The point is not paperwork. The point is that during a real incident people are tired, scared of making it worse, and unsure who is allowed to pull the plug. A written plan with clear roles answers those questions in advance so the team spends its energy on the problem, not on coordination.
Preparation happens before anything breaks. You write runbooks for likely scenarios, set up logging and alerting, define who the incident commander is, and run game days where you simulate a breach. If you have no logs you cannot investigate, so preparation is where most of the real work lives.
Detection and triage come next. An alert fires from something like a SIEM (Splunk, Elastic, or a cloud tool like AWS GuardDuty), a customer reports an outage, or a monitor catches an error spike. Someone declares an incident and assigns a severity. A SEV1 page that wakes people up is different from a SEV3 that waits until morning.
Containment stops the bleeding. You isolate a compromised host from the network, revoke a stolen token, fail over to a healthy region, or rate-limit an abusive client. Containment is deliberately separate from eradication because you often want to keep evidence intact for forensics before you wipe the box.
Eradication removes the root cause, recovery restores service and confirms it is healthy, and the final phase is the postmortem. You write a blameless timeline of what happened, why detection was slow or fast, and concrete action items so the same gap does not reopen.
Every team that runs production systems needs some incident response, but the depth scales with risk. A solo side project might just need backups and a phone alert. A bank or a healthcare platform needs a 24/7 on-call rotation, legal and PR playbooks, breach-notification timers, and tabletop exercises every quarter.
The main trade-off is investment versus speed under fire. Heavy process, formal severity matrices, and approval chains add overhead and can feel like bureaucracy on a quiet day. But the median time to contain a breach is measured in days, and IBM's annual breach report consistently shows that organizations with a tested incident response plan save roughly a million dollars per breach compared to those without one.
A second trade-off is containment versus evidence. Pulling a compromised server offline instantly stops the attacker but can destroy memory and logs a forensic team needs. Mature teams snapshot the disk and capture memory before isolating, so they keep both safety and the ability to answer how the attacker got in.
Watch out for two failure modes: a beautiful plan nobody has ever rehearsed, and blameful postmortems that make engineers hide mistakes. Both quietly destroy the value of the whole exercise.
Say your e-commerce checkout starts throwing 500 errors at 2am. An alert pages the on-call engineer, who declares a SEV2 and becomes incident commander. They open a dedicated Slack channel and a video bridge so all communication has one home.
Detection: dashboards show the payment service is timing out on database calls. Triage points at a connection pool that exhausted after a deploy three hours earlier. Containment: the team rolls back the bad deploy and the error rate drops within minutes, restoring checkout. They do not yet know the exact bug, but customers can buy again.
Eradication and recovery follow over the next day once the offending code is fixed and redeployed safely. Then comes the blameless postmortem: the deploy lacked a pool-size check, the alert fired late because the threshold was wrong, and nobody owned the on-call runbook for that service. Three action items get filed with owners and due dates. That is incident response working as a loop, not a one-off scramble.
PagerDuty
Built its whole business around incident response: routing alerts to the right on-call engineer, escalation policies, and live incident timelines.
Google SRE
Codified blameless postmortems and the incident commander role in its widely copied Site Reliability Engineering practices.
Atlassian
Runs and openly documents an incident management process with severity levels and a dedicated incident commander for Jira and Confluence outages.
AWS
Provides GuardDuty for threat detection and Detective plus Security Hub to drive cloud incident investigation and response.
Incident response is the full process of handling any harmful event, including security breaches, and covers detection, containment, and learning. Disaster recovery is narrower: it is specifically about restoring systems and data after a major loss like a region outage or data center failure. Disaster recovery is essentially one tool that incident response may reach for during the recovery phase.
The NIST model uses four: preparation, detection and analysis, containment plus eradication plus recovery, and post-incident activity. The SANS model spells out six: preparation, identification, containment, eradication, recovery, and lessons learned. They describe the same loop with different granularity.
An incident commander runs it. This is one person who owns coordination, decisions, and communication during the incident, but is usually not the one typing fixes. Separating command from hands-on work keeps the response from descending into a leaderless scramble where everyone debugs and nobody coordinates.
Because people hide mistakes when they expect punishment, which destroys the data you need to actually fix systemic gaps. A blameless postmortem assumes everyone acted reasonably with the information they had and focuses on the process and tooling that let the failure happen, not on naming a culprit. Honest timelines lead to real fixes.
A runbook is a written, step-by-step guide for handling a specific scenario, like a leaked credential or a service being down. It lists who to page, what to check, what commands to run, and how to contain the issue. Good runbooks let a tired on-call engineer at 3am act correctly without having to invent the response from scratch.
Incident Response
Structured processes for detecting, containing, eradicating, and recovering from security incidents
MTTD (Mean Time to Detect)
How long it takes to realize something is broken, the gap between failure and awareness
MTTA (Mean Time to Acknowledge)
How long until a human says 'I'm on it', the gap between alert and action
MTTR (Mean Time to Resolve)
How long it takes to fix the problem, from acknowledgment to all-clear
Webhook Notifications
HTTP callbacks for automated incident response, trigger workflows on alert
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Postmortem
A blameless analysis conducted after an incident to document what happened, why, and how to prevent it from recurring. The most important output is the list of action items.
Penetration Testing
Authorized simulated attacks against a system to find security vulnerabilities before real attackers do. White-hat hackers probe for weaknesses in a controlled environment.
Threat Modeling
A structured process for identifying security threats, attack surfaces, and mitigations during system design. STRIDE and DREAD are common frameworks.
Security Audit
A systematic evaluation of a system's security posture against standards and best practices. Covers access controls, encryption, logging, vulnerability management, and compliance.
This page explains the idea. The "Incident Response" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Incident Response lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Incident Response in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
