We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A disk-full alert fires at 3:14 AM. The on-call engineer wakes up, reads the page, opens the wiki, finds the runbook for "disk full on log volume," SSHs to the box, runs find /var/log -name '*.gz' -mtime +7 -delete, watches disk usage drop from 94 percent to 61 percent, and goes back to bed. Total time: 23 minutes of groggy human effort for a procedure that is identical every single time it runs.
Netflix measured this exact pattern across their fleet in 2016. About 60 percent of their on-call pages had a matching wiki runbook. Of those, roughly 80 percent were fully deterministic: the steps never varied based on judgement, just on the alert parameters. That is half of all pages being handled by a tired human reading a script and typing commands that a machine could execute in seconds.
The gap between a wiki runbook and an automated one is not a nice-to-have. Netflix publishes that automated runbooks cut their tier 1 MTTR from about 24 minutes to 4 minutes across services that adopted them. That is a 6x improvement in resolution time, plus it removes the 3 AM wake-up for deterministic problems entirely.