We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
A planned exercise where teams simulate production failures to test incident response procedures and system resilience. Like a fire drill for your infrastructure.
In short
A Game Day is a scheduled exercise where an engineering team deliberately injects failures into a system, often production or a production-like environment, to test how the system and the on-call humans respond under realistic pressure. It is a controlled rehearsal that surfaces broken runbooks, missing alerts, and recovery gaps before a real outage finds them.
In short
A Game Day is a scheduled exercise where an engineering team deliberately injects failures into a system, often production or a production-like environment, to test how the system and the on-call humans respond under realistic pressure. It is a controlled rehearsal that surfaces broken runbooks, missing alerts, and recovery gaps before a real outage finds them.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Game Day across a run of lessons, not one page. These 4 alone are about 112 minutes of step-by-step reading, every one with a quiz.
Scheduled exercises where teams practice responding to simulated incidents, building muscle memory.
Intentionally inject failures to verify resilience. Chaos Monkey, Gremlin, and Litmus.
Define who responds to incidents and when, fair rotation that prevents burnout.
Automate common operational procedures, from manual checklists to executable workflows.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Game Dayspay once,
yours for life
A Game Day is a planned event, usually a few hours long, where a team picks a failure scenario and makes it happen on purpose. Examples: kill a primary database, drop an availability zone, fill a disk to 100 percent, return 500s from a payment provider, or revoke a TLS certificate. The team then watches what the system does and what the on-call engineer does.
The point is not to break things for fun. It is to test the parts of reliability that you cannot prove with code review: do the alerts fire, does the runbook match reality, does the failover actually work, and can a tired person at 3am follow the steps. Most outages get worse because the response is slow or wrong, not because the original failure was exotic. Game Days attack that response directly.
A Game Day usually has clear roles. A facilitator designs and triggers the scenario, the on-call team responds as if it were real, and observers take notes on every gap they see. Many teams hide the exact failure from the responders so the exercise feels like a genuine incident.
It starts with a hypothesis. The team writes down what they expect: for example, if the primary Postgres node dies, the replica should be promoted within 30 seconds, traffic should reroute automatically, and PagerDuty should page within 1 minute. Writing the expectation first is what turns a random outage into a useful test.
Next comes a blast-radius plan. You decide exactly what you will break, for how long, and how you will stop it. You define an abort condition, such as error rate above 5 percent for paying customers, and a one-command rollback. Running on a real but low-traffic window, or on a staging copy carrying mirrored traffic, keeps the risk contained.
Then the team triggers the failure, often using a chaos engineering tool to inject it precisely. They observe dashboards, alerts, and human actions in real time. After recovery, the most important part is the debrief: every gap becomes a ticket. A Game Day with no follow-up action items was mostly theater.
The difference from general chaos engineering is scope. Chaos engineering is often automated and continuous; a Game Day is a scheduled, human-in-the-loop event focused as much on the team's response as on the system's behavior.
Run Game Days when the cost of a real outage is high and your recovery path is mostly untested: before a major launch, after a big architecture change, when onboarding new on-call engineers, or on a recurring cadence such as once a quarter. They are especially valuable for failover paths and disaster recovery, which tend to rot silently because they are rarely exercised.
The trade-off is real risk and real cost. You are spending senior engineering hours and accepting a chance of impacting users. Run one without an abort plan, on a Friday afternoon, with no executive awareness, and you can turn a drill into an actual incident. Start in staging, then graduate to low-traffic production windows once the team trusts the process.
The other failure mode is cultural. If a Game Day turns into blame for whoever fumbled the response, people will resist the next one. The findings have to be treated as system problems, not personal ones. The output is a list of fixes, not a list of people who did badly.
Suppose you run an e-commerce checkout service backed by a primary and replica database in two availability zones. Your hypothesis: if the primary AZ goes down, the replica is promoted, checkout keeps working, and on-call is paged within a minute.
On Game Day, the facilitator blocks all traffic to the primary AZ at 10am on a low-volume weekday. Reality differs from the hypothesis: the replica is promoted in 25 seconds as expected, but the application servers keep a stale connection pool pointed at the dead primary for 4 minutes, so checkout returns errors the whole time. Worse, the alert that should have paged on-call never fired because it was watching the load balancer, not the database.
The team aborts, restores the AZ, and walks away with three concrete tickets: shorten the connection-pool timeout, add a database-health alert, and document the manual connection-drain step in the runbook. None of those gaps would have shown up in a code review. That is the whole value of the exercise.
Amazon Web Services
Popularized internal Game Days and runs them with customers through the AWS Well-Architected program and Fault Injection Service to rehearse failures like AZ loss.
Netflix
Built the chaos engineering culture that Game Days draw on; tools like Chaos Monkey and regional failover exercises validate resilience continuously.
Runs company-wide DiRT (Disaster Recovery Testing) exercises where teams simulate large-scale outages, network cuts, and even datacenter loss to test response.
Gremlin
Commercial chaos engineering platform built specifically to design, inject, and safely abort failures during Game Days.
Chaos engineering is the broader practice of injecting failures to learn how a system behaves, and it is often automated and continuous. A Game Day is a specific scheduled event with humans in the loop, focused as much on testing the team's incident response as on the system itself.
Eventually, yes, because staging never fully reflects production. But start in staging or on a small slice of traffic, define an abort condition and one-command rollback, run it during a low-traffic window, and make sure leadership and on-call know it is happening.
A common cadence is quarterly for critical systems, plus extra runs before major launches, after big architecture changes, or to train new on-call engineers. The right frequency is whatever keeps your recovery paths and runbooks from going stale.
A written hypothesis before you start, a contained blast radius with a clear abort plan, a blameless debrief afterward, and most importantly a list of concrete fix tickets. A Game Day that produces no action items did not test anything useful.
A facilitator to design and trigger the scenario, the actual on-call responders who handle it as if it were real, and observers who document every gap. For high-stakes exercises, loop in product and leadership so nobody mistakes the drill for a real outage.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
Postmortem
A blameless analysis conducted after an incident to document what happened, why, and how to prevent it from recurring. The most important output is the list of action items.
Failure Injection
Deliberately introducing faults (latency, errors, crashes) into a system to verify that resilience mechanisms work. A specific technique within chaos engineering.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Game Days" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Game Days lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Game Day in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
