Incident Classification

PagerDuty processes more than 4 million incidents per year for its 22,000 customers. Their 2023 State of Digital Operations report found that roughly 40 percent of those alerts are infrastructure noise with no user impact. Sixty percent of SREs surveyed cited paging fatigue as their top burnout factor.

Incident classification is the system that separates signal from noise before a human is bothered. It takes raw alerts from Datadog, Prometheus, CloudWatch, or any other monitoring tool and routes them through a decision tree that asks: is this real? How bad is it? Who should handle it? What response contract applies?

Without classification, every alert gets the same treatment: a phone call at 3 AM. With classification, only the alerts that represent real customer impact generate a phone page. Everything else lands in a Slack channel where someone can look at it during business hours.

The goal is not fewer alerts. It is fewer wrong alerts reaching the wrong people at the wrong time.

Classification sits between alert detection and human response. It takes the raw event and enriches it with context before deciding how to route it.

The combination of these dimensions determines the routing: who gets paged, through which channel, with what SLA. A SEV1 security incident in the payments system routes to a different set of humans than a SEV3 infrastructure alert in the staging environment, even though both started as a monitor firing.

The decision must be fast (under 60 seconds), deterministic (same inputs always produce the same output), and upgradable (a SEV3 can become a SEV1 the moment new information arrives).