Game Day
A planned exercise where teams simulate production failures to test incident response procedures and system resilience. Like a fire drill for your infrastructure.
What is Game Day?
In short
A Game Day is a scheduled exercise where an engineering team deliberately injects failures into a system, often production or a production-like environment, to test how the system and the on-call humans respond under realistic pressure. It is a controlled rehearsal that surfaces broken runbooks, missing alerts, and recovery gaps before a real outage finds them.
What a Game Day actually is
A Game Day is a planned event, usually a few hours long, where a team picks a failure scenario and makes it happen on purpose. Examples: kill a primary database, drop an availability zone, fill a disk to 100 percent, return 500s from a payment provider, or revoke a TLS certificate. The team then watches what the system does and what the on-call engineer does.
The point is not to break things for fun. It is to test the parts of reliability that you cannot prove with code review: do the alerts fire, does the runbook match reality, does the failover actually work, and can a tired person at 3am follow the steps. Most outages get worse because the response is slow or wrong, not because the original failure was exotic. Game Days attack that response directly.
A Game Day usually has clear roles. A facilitator designs and triggers the scenario, the on-call team responds as if it were real, and observers take notes on every gap they see. Many teams hide the exact failure from the responders so the exercise feels like a genuine incident.
How a Game Day runs under the hood
It starts with a hypothesis. The team writes down what they expect: for example, if the primary Postgres node dies, the replica should be promoted within 30 seconds, traffic should reroute automatically, and PagerDuty should page within 1 minute. Writing the expectation first is what turns a random outage into a useful test.
Next comes a blast-radius plan. You decide exactly what you will break, for how long, and how you will stop it. You define an abort condition, such as error rate above 5 percent for paying customers, and a one-command rollback. Running on a real but low-traffic window, or on a staging copy carrying mirrored traffic, keeps the risk contained.
Then the team triggers the failure, often using a chaos engineering tool to inject it precisely. They observe dashboards, alerts, and human actions in real time. After recovery, the most important part is the debrief: every gap becomes a ticket. A Game Day with no follow-up action items was mostly theater.
The difference from general chaos engineering is scope. Chaos engineering is often automated and continuous; a Game Day is a scheduled, human-in-the-loop event focused as much on the team's response as on the system's behavior.
When to use it and the trade-offs
Run Game Days when the cost of a real outage is high and your recovery path is mostly untested: before a major launch, after a big architecture change, when onboarding new on-call engineers, or on a recurring cadence such as once a quarter. They are especially valuable for failover paths and disaster recovery, which tend to rot silently because they are rarely exercised.
The trade-off is real risk and real cost. You are spending senior engineering hours and accepting a chance of impacting users. Run one without an abort plan, on a Friday afternoon, with no executive awareness, and you can turn a drill into an actual incident. Start in staging, then graduate to low-traffic production windows once the team trusts the process.
The other failure mode is cultural. If a Game Day turns into blame for whoever fumbled the response, people will resist the next one. The findings have to be treated as system problems, not personal ones. The output is a list of fixes, not a list of people who did badly.
A concrete example
Suppose you run an e-commerce checkout service backed by a primary and replica database in two availability zones. Your hypothesis: if the primary AZ goes down, the replica is promoted, checkout keeps working, and on-call is paged within a minute.
On Game Day, the facilitator blocks all traffic to the primary AZ at 10am on a low-volume weekday. Reality differs from the hypothesis: the replica is promoted in 25 seconds as expected, but the application servers keep a stale connection pool pointed at the dead primary for 4 minutes, so checkout returns errors the whole time. Worse, the alert that should have paged on-call never fired because it was watching the load balancer, not the database.
The team aborts, restores the AZ, and walks away with three concrete tickets: shorten the connection-pool timeout, add a database-health alert, and document the manual connection-drain step in the runbook. None of those gaps would have shown up in a code review. That is the whole value of the exercise.
Where it is used in production
Amazon Web Services
Popularized internal Game Days and runs them with customers through the AWS Well-Architected program and Fault Injection Service to rehearse failures like AZ loss.
Netflix
Built the chaos engineering culture that Game Days draw on; tools like Chaos Monkey and regional failover exercises validate resilience continuously.
Runs company-wide DiRT (Disaster Recovery Testing) exercises where teams simulate large-scale outages, network cuts, and even datacenter loss to test response.
Gremlin
Commercial chaos engineering platform built specifically to design, inject, and safely abort failures during Game Days.
Frequently asked questions
- What is the difference between a Game Day and chaos engineering?
- Chaos engineering is the broader practice of injecting failures to learn how a system behaves, and it is often automated and continuous. A Game Day is a specific scheduled event with humans in the loop, focused as much on testing the team's incident response as on the system itself.
- Should I run a Game Day in production?
- Eventually, yes, because staging never fully reflects production. But start in staging or on a small slice of traffic, define an abort condition and one-command rollback, run it during a low-traffic window, and make sure leadership and on-call know it is happening.
- How often should a team run Game Days?
- A common cadence is quarterly for critical systems, plus extra runs before major launches, after big architecture changes, or to train new on-call engineers. The right frequency is whatever keeps your recovery paths and runbooks from going stale.
- What makes a Game Day successful?
- A written hypothesis before you start, a contained blast radius with a clear abort plan, a blameless debrief afterward, and most importantly a list of concrete fix tickets. A Game Day that produces no action items did not test anything useful.
- Who should be involved in a Game Day?
- A facilitator to design and trigger the scenario, the actual on-call responders who handle it as if it were real, and observers who document every gap. For high-stakes exercises, loop in product and leadership so nobody mistakes the drill for a real outage.
Learn Game Day hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Game Day as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Incident Response
The structured process for detecting, containing, eradicating, and recovering from security incidents. Includes communication plans, runbooks, and post-incident reviews.
Postmortem
A blameless analysis conducted after an incident to document what happened, why, and how to prevent it from recurring. The most important output is the list of action items.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.