We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Picture this. A SEV1 fires at 3:12 AM. The primary on-call engineer's phone rings. No answer. The phone rings again at 3:13. Still nothing. Without an escalation policy, that is where the story ends. The checkout page stays broken. Customers see error 500 for an hour. Revenue loss stacks up at thousands of dollars per minute for any high-traffic e-commerce platform.
Escalation policies exist because no individual person is reliable at 3 AM. The policy is a chain of fallback responders, each triggered by a timer. If the primary does not ack in 5 minutes, the secondary gets paged. If the secondary stays silent, the engineering manager is pulled in. If the manager does not respond, a VP gets a phone call. Each rung of the ladder was added because someone at some company experienced the exact failure it prevents.
PagerDuty processes over 4 million incidents per year. Their 2023 data shows that the median time to first acknowledge on SEV1 pages is 4 minutes when engineers are awake and 9 minutes when paged overnight. The gap between those two numbers is the reason escalation policies exist. You cannot assume the primary will always be available within the ack window.
An escalation policy is a list of tiers, each with a target and a timeout. The targets can be individual engineers, on-call schedules, or groups. The timeout is how long the system waits before firing the next tier.
| Tier | Target | Timeout | Purpose |
|---|---|---|---|
| L1 | Primary on-call | t=0 | First responder, determined by the schedule |
| L2 | Secondary on-call | t+5 min | Backup, usually last week's primary |
| L3 | Engineering manager | t+15 min | Organizational authority, can rally more responders |
| L4 | VP / Director | t+30 min | Last resort, authorizes emergency actions |
The ladder stops as soon as any tier acknowledges. If the primary acks at minute 3, the secondary never gets paged. The clock resets. The post-incident review later asks whether the ladder was well-tuned based on how many tiers actually fired.
Stripe, Netflix, and Google all publish variants of this ladder in their SRE playbooks. The timing varies (Netflix uses 5/10/15 minute windows) but the structure is consistent: individual, backup, manager, executive.