We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Uber runs routing rules for more than 300 . When the checkout service throws 5xx errors at 2 AM, the payments team needs to be paged. Not the search team. Not the recommendations team. Not the platform team. The payments team, specifically the person who is on call for that service right now in whatever time zone the sun is currently up.
At small companies, routing is trivial. You have one team. All alerts go to that team. But somewhere around 15 to 20 microservices, flat routing breaks. Engineers start getting paged for services they do not own. Alert fatigue sets in. The team that actually owns the broken service does not hear about it because the wrong people are absorbing the noise. GitHub published their routing model in 2019 and saw a 40 percent drop in overnight pages for the US team after switching to time-aware routing. Shopify unified Datadog, , Sentry, and synthetic check alerts into a single routing engine specifically to stop engineers from juggling 4 separate paging apps.
Alert routing is the dispatch layer between detection and response. Get it right and the correct engineer gets a phone call within 30 seconds of the alert firing. Get it wrong and your incident response starts with 10 minutes of "who owns this?" in a Slack channel while the outage runs.
A routing engine is not a pile of if-else statements. It is a tag-driven matcher that inherits ownership from a single source of truth, usually a service catalog. When a team changes ownership of a service, the routing rules update automatically because the tag lookup picks up the new config from the catalog.
| Step | What Happens |
|---|---|
| Ingest | Alert arrives from Datadog, , Sentry, or synthetic checks |
| Tag extraction | Engine reads service, region, severity, and environment tags |
| Ownership match | Tags are matched against the service catalog to find the owning team |
| Time-of-day rule | Business hours page the local primary, off hours follow the sun |
| Channel fanout | SEV1: phone + SMS + push. SEV2: push + Slack. SEV3: Slack only |
| Fall-through | Unmatched alerts go to the platform team's catch-all rule |
The fastest production engines resolve from ingest to fanout in under 100 milliseconds. That keeps the total time from threshold breach to phone ringing under 30 seconds for most integrations.