We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Here is a pattern that kills on-call rotations. A team sets up 50 monitors during a launch sprint. Six months later, 30 of those monitors fire every week but nobody acts on them. Engineers learn to glance at the pager, see one of the usual suspects, and go back to sleep. Then a real SEV1 fires at 04:00 and sits unacknowledged for 47 minutes because the on-call engineer assumed it was another false alarm.
PagerDuty's research shows on-call engineers who receive more than 5 pages per 12 hour shift report symptoms of burnout within 3 months. That is not a survey about feelings. It shows up in attrition numbers, ack times, and error rates during incident response.
Alert fatigue is not a minor annoyance. It is the direct mechanism by which noisy alerting causes real outages. Every false positive costs trust in the system. Once engineers stop trusting the pager, you have no alerting at all. You just have noise.
Preventing alert fatigue is not one fix. It is a pipeline with four stages, each cutting the event count by a predictable factor.
| Stage | Input | Output | Reduction |
|---|---|---|---|
| Dedupe | 10,000 raw alerts/day | 3,000 unique events | ~70% |
| Grouping | 3,000 unique events | 500 logical incidents | ~83% |
| Severity filter | 500 logical incidents | 100 actual pages | ~80% |
| Weekly review | 100 pages/day | 3-5 pages/shift target | ongoing |
The math is consistent across Datadog, PagerDuty, and Splunk On-Call customer data when teams commit to tag hygiene. Google's SRE book formalizes the quality metric as the actionable alert ratio and treats anything below 4 to 1 signal to noise as a reliability bug.
The pipeline only works when all four stages are running. Dedupe without severity filtering still pages engineers for SEV4 events. Severity filtering without weekly review lets stale rules accumulate. The stages reinforce each other.