We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Google's DiRT program, Disaster Recovery Testing, has run continuously since 2006. Seventeen years of scheduled drills. In that time, Google has publicly credited DiRT with preventing at least a dozen incidents that would have been major outages. The reason is simple: incident response is a perishable skill. If you do not practice it regularly, the skill rots.
Think about what happens during a real incident. The pager fires at 2am. The on-call engineer opens a laptop, navigates to a dashboard, reads a runbook, runs commands, communicates status, escalates to the right people, and coordinates mitigation. Every one of those steps can fail: the dashboard is slow, the runbook is stale, the command syntax changed, the escalation list has someone who left the company three months ago, and the war room Slack channel is archived.
Game days test the full stack of incident response, not just the software, but the humans, the processes, the communication channels, and the documentation. They do this on a schedule, during business hours, when the stakes are low and the learning is high. The output is not a prettier runbook. The output is muscle memory: the on-call team has practiced the motions so many times that when the real incident arrives, they execute instead of figuring it out for the first time.
Game days come in two flavors. You need both.
| Format | Cost | Risk | What It Finds | Cadence |
|---|---|---|---|---|
| Tabletop drill | Low. One hour in a conference room | Zero. Nothing touches production | Stale runbooks, unclear ownership, broken escalation chains, gaps in communication plans | Monthly |
| Live drill | High. Real fault injection, real on-call response | Real but bounded. Blast radius matters | Slow dashboards, broken commands, deploy pipeline fragility, timing gaps in mitigation | Quarterly |
Tabletops find people and process problems. Live drills find tooling and timing problems. Teams that only run one format miss the entire category of bugs the other format catches. A team with beautiful runbooks from monthly tabletops discovers during a real incident that the runbook command takes 40 seconds to execute. A team that only runs live drills has fast tooling but no escalation plan when the VP calls.
Stripe and Atlassian both run tabletops monthly and live drills quarterly. Google DiRT settled on this cadence after years of experimentation. The quarterly live drill is a multi-day event with prep, execution, debrief, and follow-up. Teams rotate scenarios so over a year the entire failure surface gets touched at least once.