We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
In short
Alerting is the part of a monitoring system that automatically notifies on-call engineers when a metric crosses a defined threshold or a condition holds true for some duration, so problems get attention before users complain. A good alert fires only when a human needs to take action right now, and it routes through a tool like PagerDuty or Opsgenie to the right person.
In short
Alerting is the part of a monitoring system that automatically notifies on-call engineers when a metric crosses a defined threshold or a condition holds true for some duration, so problems get attention before users complain. A good alert fires only when a human needs to take action right now, and it routes through a tool like PagerDuty or Opsgenie to the right person.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Alerting across a run of lessons, not one page. These 3 alone are about 92 minutes of step-by-step reading, every one with a quiz.
Getting notified when things go wrong, before your users tell you.
Design a metrics collection and monitoring system - time-series databases, aggregation pipelines, alerting, and dashboards at scale.
Design alerting to maintain signal quality, because ignored alerts are worse than no alerts.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Alertingpay once,
yours for life
Alerting is the layer that sits on top of your metrics and logs and decides when to wake somebody up. You collect numbers like error rate, request latency, queue depth, or disk usage. An alerting rule watches those numbers and fires a notification when a condition is met, for example error rate above 1 percent for 5 minutes.
The key word is action. An alert is a promise that a human should do something. If nobody needs to respond, it should be a dashboard or a daily report, not an alert. The classic failure mode is alerting on causes you cannot act on (CPU at 80 percent) instead of symptoms users feel (checkout requests failing).
Alerting is distinct from monitoring. Monitoring is recording and graphing what is happening. Alerting is the judgment call on top of that data: this specific thing, right now, is bad enough to interrupt a person.
A typical pipeline has three stages. First, a rule engine evaluates conditions on a schedule. In Prometheus this is an alerting rule that runs an expression every 15 to 30 seconds and marks a rule as pending, then firing once the condition has held for the configured `for` duration. The duration matters: it prevents a single noisy data point from paging anyone.
Second, firing alerts are sent to a router. Prometheus ships them to Alertmanager, which groups related alerts together, deduplicates copies coming from many instances, silences known issues during maintenance, and applies inhibition so that a big outage does not also fire 50 child alerts. Datadog, Grafana, and New Relic have their own equivalents built in.
Third, the router hands off to a paging tool such as PagerDuty or Opsgenie. The paging tool owns the on-call schedule, the escalation policy, and the actual phone call, SMS, push, or Slack message. If the first engineer does not acknowledge within, say, 5 minutes, it escalates to the next person, then to the team lead.
Two common rule styles are threshold alerts (latency p99 over 500ms) and rate-of-change or anomaly alerts (traffic dropped 40 percent versus the same hour last week). Threshold rules are simple and predictable; anomaly rules catch problems you did not anticipate but produce more false positives.
Alert on symptoms that map to your service level objectives, not on every internal metric. Google's site reliability practice recommends alerting on a fast-burning error budget: if you are burning a month of allowed errors in an hour, page; if you are burning slowly, open a ticket instead of waking someone at 3 AM.
The biggest trap is alert fatigue. When a team gets paged for things that are not actionable, people start ignoring the pager, and the one real alert gets lost in the noise. The fix is ruthless pruning: every alert that fires without anyone doing anything should be deleted, tuned, or downgraded.
Tuning is a balance between two errors. Make thresholds too tight and you get false positives that train people to ignore alerts. Make them too loose and you miss real incidents. Adding a `for` duration, alerting on multi-window burn rates, and grouping related alerts all reduce noise without missing real failures. A healthy team tracks how many alerts fire per week and what fraction led to real action.
Say you run a payments API with an SLO of 99.9 percent successful requests. You define a metric for failed payments and a Prometheus rule: fire if the failure rate over the last 5 minutes is high enough to burn 2 percent of the monthly error budget in one hour.
At 2 AM a downstream bank gateway starts timing out. Failed payments climb, the rule goes pending, and after holding for 5 minutes it fires. Alertmanager groups all the per-pod alerts into one, checks there is no active maintenance silence, and forwards a single alert to PagerDuty.
PagerDuty looks at the on-call schedule, calls the primary engineer, and posts to the team's Slack channel with a link to the runbook and the relevant dashboard. The engineer acknowledges, sees the gateway is the cause, fails over to the backup gateway, and the error rate drops. The alert auto-resolves when the failure rate returns to normal, and the incident is logged for the next morning's review.
PagerDuty
Owns the on-call schedule and escalation; turns a firing alert into a phone call, SMS, or push and escalates if nobody acknowledges in time.
Opsgenie
Atlassian's paging tool that routes alerts to the right on-call engineer with escalation policies and Slack and Jira integration.
Prometheus and Alertmanager
Prometheus evaluates alerting rules on metrics; Alertmanager groups, deduplicates, silences, and routes the firing alerts to paging tools.
Datadog
Built-in monitors fire on thresholds or anomaly detection across metrics, logs, and traces, then notify via Slack, email, or PagerDuty.
Monitoring is collecting and graphing data about your system over time. Alerting is the decision layer on top: it watches that data and notifies a human only when a specific condition is bad enough to need action right now.
It is actionable, urgent, and tied to something users feel. A good alert means a human needs to do something immediately, points to a runbook, and rarely fires without a real problem behind it. If it fires and nobody acts, it should be a dashboard or ticket, not a page.
Alert fatigue is when so many noisy or non-actionable alerts fire that engineers start ignoring the pager and miss real incidents. Fix it by deleting alerts that never lead to action, adding a duration so single spikes do not page, grouping related alerts, and alerting on symptoms instead of every internal metric.
Prefer user-facing symptoms like error rate and latency. High CPU is often normal and not directly actionable, while a spike in failed requests always matters. Alert on what users experience, and use resource metrics for debugging once you are already investigating.
Burn-rate alerting fires based on how fast you are consuming your error budget rather than a fixed threshold. If you are spending a month of allowed errors in an hour, page immediately; if you are spending it slowly, open a ticket instead. It reduces both false pages and missed slow-burning outages.
Design a Metrics/Monitoring System
Design a metrics collection and monitoring system - time-series databases, aggregation pipelines, alerting, and dashboards at scale
Who Notices First? Four Watchers and Four Silent Breaks
I replayed a real shop's model night by night for 162 nights, from launch on 1 July 2011, and broke it four ways that throw no error: a frozen feature table, pounds arriving as pence, a count filled with 0, and a model file that wants seven columns. Four watchers looked every night: the service, the inputs, the model's own scores, and the true answers. With nothing broken, the input checks rang on all 162 nights, because the customers grew older than any training row. With those four checks turned off, the pence, zero and seven-column breaks were caught the same night, the frozen table only after 7 to 17 days, and the true answers caught one break of four, 31 to 34 days late.
Alert Fatigue Prevention
Design alerting to maintain signal quality, because ignored alerts are worse than no alerts
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Logging
Recording discrete events with timestamps, severity levels, and context. Structured logs (JSON) are searchable; unstructured logs (plaintext) are not. Ship them to a central system.
This page explains the idea. The "Alerting" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Alerting lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Alerting in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
