Failure Injection
Deliberately introducing faults (latency, errors, crashes) into a system to verify that resilience mechanisms work. A specific technique within chaos engineering.
What is Failure Injection?
In short
Failure injection is the practice of deliberately introducing faults into a running system, such as added latency, dropped network packets, returned errors, or killed processes, to confirm that its resilience mechanisms (retries, timeouts, fallbacks, failover) actually work under real failure conditions instead of only on paper. It is the core technique inside chaos engineering and is run in controlled experiments with a known blast radius.
What Failure Injection Actually Is
Most outages do not come from the failure itself. They come from how the system reacts to the failure. A timeout that was never tested, a retry storm that takes down a healthy service, a fallback path that throws its own exception. Failure injection forces these reactions to happen on purpose so you can watch them and fix them before a customer ever sees them.
The idea is simple: pick a fault you expect to happen in production, inject it into the system in a controlled way, and check whether the system behaves the way you designed it to. If a dependency is supposed to be optional, kill it and verify the request still succeeds. If a database is supposed to fail over in under 30 seconds, sever the connection to the primary and measure the real recovery time.
Failure injection is the hands-on technique. Chaos engineering is the broader discipline around it: forming a hypothesis, defining steady-state metrics, running the experiment, and rolling back. You can inject faults without doing full chaos engineering, but you cannot do chaos engineering without injecting faults.
How It Works Under the Hood
Faults are injected at different layers. At the network layer, tools manipulate the kernel using utilities like tc (traffic control) and iptables on Linux to add latency, drop a percentage of packets, or partition one host from another. At the process layer, an agent can send SIGKILL to a process, peg CPU to 100 percent, or fill up the disk. At the application layer, code or a service mesh intercepts calls and returns synthetic errors, like making 10 percent of calls to the payments service return HTTP 503.
A safe injection follows a tight loop. First you measure steady state, the normal value of a metric such as successful requests per second or p99 latency. Then you state a hypothesis: "if the recommendations service goes down, checkout latency stays under 400ms." You start small, often a single host or a tiny percentage of traffic, and you keep a kill switch ready to abort instantly.
Blast radius is the central safety control. You begin in staging, then move to production on a sliver of traffic during low-risk hours, and only widen the scope once each smaller experiment passes. Service meshes like Istio and Envoy make application-layer injection easy because they can inject HTTP delays and aborts through configuration without touching application code.
When To Use It and the Trade-offs
Use failure injection when your system has redundancy and fallback logic that you are trusting but have never actually exercised. Anything with retries, circuit breakers, replicas, multi-zone deployment, or graceful degradation is a candidate. It is most valuable for distributed systems where partial failure is the normal state, not the exception.
The trade-off is real risk. A poorly scoped experiment can cause the exact outage you were trying to prevent. This is why you need good observability first: if you cannot see what is happening during the experiment, you should not run it. You also need automated rollback, run during business hours when engineers are watching, and clear communication so an injected fault is not mistaken for a real incident.
Do not start in production, and do not inject faults into a system that has no recovery mechanism to test. If you already know a dependency has no fallback, injecting its failure just causes an outage and teaches you nothing new. Fix the design first, then verify it with injection.
A Concrete Example
Netflix popularized this with Chaos Monkey, which randomly terminates virtual machine instances in production during business hours. The point was to force every team to build services that survive losing any single instance at any time, so that an unplanned EC2 failure at 3 AM is a non-event rather than a page.
A typical modern experiment looks like this. An e-commerce team suspects checkout is too tightly coupled to its non-critical recommendations service. They form the hypothesis that checkout should still complete if recommendations is down. Using their service mesh, they inject HTTP 500 responses into 100 percent of calls to recommendations, but only for 5 percent of customer sessions. They watch the checkout success rate dashboard.
If checkout success stays flat, the fallback works and they widen the test. If checkout success drops, they have found a hidden hard dependency, abort the experiment in seconds, and add a proper timeout and default response. Either way they learned something true about the system instead of guessing.
Where it is used in production
Netflix
Built Chaos Monkey and the broader Simian Army to randomly kill production instances and inject regional failures, forcing every service to tolerate loss.
AWS Fault Injection Service
Managed service that injects faults like CPU stress, API errors, and Availability Zone outages into EC2, ECS, and EKS workloads on a controlled blast radius.
Gremlin
Commercial failure-injection platform offering ready-made attacks for latency, packet loss, process kill, and disk fill with automatic rollback.
Istio and Envoy
Service mesh that injects application-layer HTTP delays and aborts through configuration, letting teams test fallback logic without changing code.
Frequently asked questions
- Is failure injection the same as chaos engineering?
- No. Failure injection is the specific technique of introducing a fault. Chaos engineering is the full discipline around it: defining steady state, forming a hypothesis, controlling the blast radius, and learning from the result. You inject faults as part of doing chaos engineering, but a quick fault injection test is not by itself a full chaos experiment.
- Should I run failure injection in production?
- Eventually yes, because staging never reflects real traffic, real data volumes, and real dependency behavior. But you start in staging, then move to production cautiously: small blast radius first, low-risk hours, full observability, an automatic kill switch, and engineers actively watching. Never start in production.
- What kinds of faults can you inject?
- Common ones are added latency, dropped or corrupted network packets, network partitions, returned error codes such as HTTP 500 or 503, killed processes, exhausted CPU, memory, or disk, and clock skew. The fault you choose should match a failure you genuinely expect to see in production.
- What is blast radius and why does it matter?
- Blast radius is how much of the system an experiment can affect, for example one host versus an entire region, or 1 percent of traffic versus all of it. Keeping it small means that if the experiment reveals a real problem, the customer impact is tiny and contained. You widen the radius only after smaller experiments pass.
- What do I need in place before injecting faults?
- Solid observability so you can see the impact, a clearly defined steady-state metric to compare against, an automated rollback or abort mechanism, and recovery logic worth testing. If a dependency has no fallback at all, fix the design first; injecting its failure just causes an outage.
Learn Failure Injection hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
See also
Related glossary terms you might want to look up next.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Game Day
A planned exercise where teams simulate production failures to test incident response procedures and system resilience. Like a fire drill for your infrastructure.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.