Alert Aggregation

A deploy breaks the checkout service at 02:14 AM. Forty nodes start returning 5xx within the same 10-second window. Without aggregation, that is 40 separate phone calls landing on one engineer's nightstand before they can even read the first alert title.

This is not a hypothetical. AWS's December 2021 us-east-1 outage generated over 100,000 downstream alerts for a single Kinesis failure. PagerDuty's incident management data shows that mass alert storms from deploys are the number one cause of missed acknowledgments because the UI becomes unusable when hundreds of alerts arrive simultaneously.

Alert aggregation collapses related raw events into a single logical incident. One incident means one page. The underlying events stay attached for forensics so nothing is lost. The engineer gets context instead of chaos.

The core idea is simple: take N raw events that share a common cause and present them as 1 incident. The aggregator decides which events belong together using grouping keys, typically service name, region, and alert type.

The grouping key is the most important decision. Group too tightly (per host) and you miss the forest. Group too loosely (per team) and unrelated incidents merge into one confusing mega-alert. Most teams start with service plus region, then refine.

PagerDuty's Event Intelligence learns grouping keys from historical patterns using machine learning. Most teams start with hand-written rules on tags because the logic is transparent and auditable. Either approach works as long as the result is one page per actual problem.