We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Observability Monitoring
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Observability Monitoring
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In 2022, Slack had a major outage. Users couldn't send messages, channels wouldn't load, and the workspace was effectively dead. The embarrassing part? Slack's own engineering team found out about the outage from Twitter. Users were posting about it before internal alerts fired.
Their alerting system had a gap. It was monitoring individual services but not the end-to-end user experience. Every reported "I'm healthy" while users stared at spinning loaders.
Alerting is the bridge between your monitoring data and human action. Metrics tell you what's happening. Logs tell you why. Alerts tell the right person at the right time that something needs fixing, before a thousand tweets do it for you.
An alert is a notification triggered when a metric crosses a threshold. That's the simple version. The useful version is more subtle:
A good alert has four properties:
Actionable. Someone can do something about it right now. "CPU at 90%" is actionable (scale up). "User signed up" is not an alert, it's a log.
Timely. It fires fast enough to matter. An alert about yesterday's disk space issue is useless.
Relevant. It goes to the person who can fix it. Database alerts go to the DBA team, not the frontend engineers.
Infrequent enough to matter. If your team gets 200 alerts a day, they stop reading them. This is called alert fatigue, and it's killed more production systems than any bug.
| Alert Quality | What Happens |
|---|---|
| Too few alerts | You miss real incidents. Users find out before you do. |
| Right amount | Team trusts alerts. Every page gets investigated. |
| Too many alerts | Team ignores alerts. Critical ones get buried in noise. |
The industry term for the sweet spot is "high signal-to-noise ratio." Every alert that fires should be worth waking someone up for, or it shouldn't exist.