Name: System Design Masterclass
Availability: InStock

A company detected unusual database queries at 2 AM. The on-call engineer panicked. They shut down the database server immediately to "stop the attacker." This killed active transactions, corrupted in-flight writes, and took the entire platform offline for 8 hours. The attacker had already exfiltrated the data hours earlier. The shutdown did not prevent any data loss but caused a massive customer-facing outage on top of the breach.

Contrast this with Cloudflare's response to the Okta compromise in January 2024. An attacker gained access to Cloudflare's Atlassian environment through stolen Okta tokens. Cloudflare's incident response team detected the activity within 24 hours, contained it to their Atlassian instance, rotated all 5,000+ production credentials within 48 hours, and published a detailed public postmortem within a week. No customer data was compromised. No service downtime.

The difference was not skill or resources. It was preparation. Cloudflare had runbooks, practiced procedures, pre-authorized containment actions, and a culture of blameless incident response. The panicking engineer had none of those.

NIST SP 800-61 defines four phases. Every major IR framework (SANS, PICERL) maps to this same structure with minor naming differences.

Phase	Activities	Goal	Duration
1. Preparation	Runbooks, tools, team training, communication templates, tabletop exercises	Be ready before an incident happens	Ongoing (months before any incident)
2. Detection and Analysis	Monitor SIEM alerts, triage, determine scope and severity, assign incident commander	Understand what happened and how bad it is	Minutes to hours
3. Containment, Eradication, Recovery	Isolate affected systems, remove threat actor, patch vulnerability, restore service	Stop the damage and get back to normal	Hours to days
4. Post-Incident Activity	Root cause analysis, blameless postmortem, process improvements, evidence retention	Prevent recurrence and improve response	1-2 weeks after resolution

Severity classification:

Level	Definition	Response Time	Escalation	Example
SEV 1	Active data exfiltration, full system compromise, ransomware	Immediate, all hands on deck	CEO, legal, PR, law enforcement	Attacker has database access and is copying data
SEV 2	Confirmed breach with limited scope	Within 1 hour	VP Engineering, security lead	Compromised service account with access to staging only
SEV 3	Suspicious activity, potential threat	Within 4 hours	On-call security engineer	Unusual login pattern from new country
SEV 4	Minor policy violation, no data at risk	Next business day	Security team ticket	Failed brute-force attempt blocked by

Containment strategies:

Strategy	When to Use	What NOT to Do
Network isolation	Compromised host. Remove from network but keep running for forensics	Do NOT power it off. Memory contains evidence
Credential rotation	Leaked secrets. Rotate immediately, revoke old	Do NOT just rotate the leaked one. Rotate everything the compromised system could access
Account disable	Compromised user. Disable account, preserve data	Do NOT delete the account. You need the audit trail
Service shutdown	Last resort. Only when the service itself is the attack vector	Do NOT shut down the database because a web server was compromised

Incident Response

The rest of this lesson

Continue this lesson

The Breach Response That Made Things Worse

The Incident Response Lifecycle