We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Deliberately introducing faults (latency, errors, crashes) into a system to verify that resilience mechanisms work. A specific technique within chaos engineering.
In short
Failure injection is the practice of deliberately introducing faults into a running system, such as added latency, dropped network packets, returned errors, or killed processes, to confirm that its resilience mechanisms (retries, timeouts, fallbacks, failover) actually work under real failure conditions instead of only on paper. It is the core technique inside chaos engineering and is run in controlled experiments with a known blast radius.
In short
Failure injection is the practice of deliberately introducing faults into a running system, such as added latency, dropped network packets, returned errors, or killed processes, to confirm that its resilience mechanisms (retries, timeouts, fallbacks, failover) actually work under real failure conditions instead of only on paper. It is the core technique inside chaos engineering and is run in controlled experiments with a known blast radius.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Failure Injection across a run of lessons, not one page. These 4 alone are about 130 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Intentionally inject failures to verify resilience. Chaos Monkey, Gremlin, and Litmus.
Define and measure Service Level Objectives, the practice that aligns reliability with business needs.
When network calls fail, and they will, retries turn transient failures into invisible hiccups.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Chaos Engineeringpay once,
yours for life
Most outages do not come from the failure itself. They come from how the system reacts to the failure. A timeout that was never tested, a retry storm that takes down a healthy service, a fallback path that throws its own exception. Failure injection forces these reactions to happen on purpose so you can watch them and fix them before a customer ever sees them.
The idea is simple: pick a fault you expect to happen in production, inject it into the system in a controlled way, and check whether the system behaves the way you designed it to. If a dependency is supposed to be optional, kill it and verify the request still succeeds. If a database is supposed to fail over in under 30 seconds, sever the connection to the primary and measure the real recovery time.
Failure injection is the hands-on technique. Chaos engineering is the broader discipline around it: forming a hypothesis, defining steady-state metrics, running the experiment, and rolling back. You can inject faults without doing full chaos engineering, but you cannot do chaos engineering without injecting faults.
Faults are injected at different layers. At the network layer, tools manipulate the kernel using utilities like tc (traffic control) and iptables on Linux to add latency, drop a percentage of packets, or partition one host from another. At the process layer, an agent can send SIGKILL to a process, peg CPU to 100 percent, or fill up the disk. At the application layer, code or a service mesh intercepts calls and returns synthetic errors, like making 10 percent of calls to the payments service return HTTP 503.
A safe injection follows a tight loop. First you measure steady state, the normal value of a metric such as successful requests per second or p99 latency. Then you state a hypothesis: "if the recommendations service goes down, checkout latency stays under 400ms." You start small, often a single host or a tiny percentage of traffic, and you keep a kill switch ready to abort instantly.
Blast radius is the central safety control. You begin in staging, then move to production on a sliver of traffic during low-risk hours, and only widen the scope once each smaller experiment passes. Service meshes like Istio and Envoy make application-layer injection easy because they can inject HTTP delays and aborts through configuration without touching application code.
Use failure injection when your system has redundancy and fallback logic that you are trusting but have never actually exercised. Anything with retries, circuit breakers, replicas, multi-zone deployment, or graceful degradation is a candidate. It is most valuable for distributed systems where partial failure is the normal state, not the exception.
The trade-off is real risk. A poorly scoped experiment can cause the exact outage you were trying to prevent. This is why you need good observability first: if you cannot see what is happening during the experiment, you should not run it. You also need automated rollback, run during business hours when engineers are watching, and clear communication so an injected fault is not mistaken for a real incident.
Do not start in production, and do not inject faults into a system that has no recovery mechanism to test. If you already know a dependency has no fallback, injecting its failure just causes an outage and teaches you nothing new. Fix the design first, then verify it with injection.
Netflix popularized this with Chaos Monkey, which randomly terminates virtual machine instances in production during business hours. The point was to force every team to build services that survive losing any single instance at any time, so that an unplanned EC2 failure at 3 AM is a non-event rather than a page.
A typical modern experiment looks like this. An e-commerce team suspects checkout is too tightly coupled to its non-critical recommendations service. They form the hypothesis that checkout should still complete if recommendations is down. Using their service mesh, they inject HTTP 500 responses into 100 percent of calls to recommendations, but only for 5 percent of customer sessions. They watch the checkout success rate dashboard.
If checkout success stays flat, the fallback works and they widen the test. If checkout success drops, they have found a hidden hard dependency, abort the experiment in seconds, and add a proper timeout and default response. Either way they learned something true about the system instead of guessing.
Latency injection makes a call slow, and abort injection makes it fail. These are the two HTTP faults the Istio service mesh supports. Its documentation shows a 7 second delay and an HTTP 500 abort, each applied only to requests from one test user, so real users are not touched. A latency fault is the quickest way to find out whether a caller has a timeout at all.
Request-level injection scopes a fault to a small set of requests. Netflix described this in 2014 with FIT (Failure Injection Testing), a platform that injects failures for a targeted group of customers on its production services and watches the effect. Narrow targeting keeps the blast radius small while still testing real production paths, and it shows whether a circuit breaker or fallback really takes over.
Code-level injection tests error paths that almost never run. The Linux kernel ships a fault-injection framework that can make memory allocations fail (failslab, fail_page_alloc) or return disk IO errors (fail_make_request). Teams that run injected faults as planned experiments with a hypothesis are doing chaos engineering, and the chaos engineering lesson covers how to run one safely.
Netflix
Built Chaos Monkey and the broader Simian Army to randomly kill production instances and inject regional failures, forcing every service to tolerate loss.
AWS Fault Injection Service
Managed service that injects faults like CPU stress, API errors, and Availability Zone outages into EC2, ECS, and EKS workloads on a controlled blast radius.
Gremlin
Commercial failure-injection platform offering ready-made attacks for latency, packet loss, process kill, and disk fill with automatic rollback.
Istio and Envoy
Service mesh that injects application-layer HTTP delays and aborts through configuration, letting teams test fallback logic without changing code.
No. Failure injection is the specific technique of introducing a fault. Chaos engineering is the full discipline around it: defining steady state, forming a hypothesis, controlling the blast radius, and learning from the result. You inject faults as part of doing chaos engineering, but a quick fault injection test is not by itself a full chaos experiment.
Eventually yes, because staging never reflects real traffic, real data volumes, and real dependency behavior. But you start in staging, then move to production cautiously: small blast radius first, low-risk hours, full observability, an automatic kill switch, and engineers actively watching. Never start in production.
Common ones are added latency, dropped or corrupted network packets, network partitions, returned error codes such as HTTP 500 or 503, killed processes, exhausted CPU, memory, or disk, and clock skew. The fault you choose should match a failure you genuinely expect to see in production.
Blast radius is how much of the system an experiment can affect, for example one host versus an entire region, or 1 percent of traffic versus all of it. Keeping it small means that if the experiment reveals a real problem, the customer impact is tiny and contained. You widen the radius only after smaller experiments pass.
Solid observability so you can see the impact, a clearly defined steady-state metric to compare against, an automated rollback or abort mechanism, and recovery logic worth testing. If a dependency has no fallback at all, fix the design first; injecting its failure just causes an outage.
Failure injection is breaking part of a system on purpose to see how the rest of it copes. You add a fault such as a slow call, an error response, or a stopped process, then check that timeouts, retries, and fallbacks behave as designed. It is done in small, controlled steps so a real outage does not follow.
Failure injection testing is running failure injection as a planned test with an expected outcome. You state what should happen, for example that checkout still works when recommendations fail, inject that fault for a small share of traffic, and compare the result with the expectation. Netflix's FIT platform is a well-known example.
Latency injection adds an artificial delay to calls between services. It shows whether callers have timeouts, whether those timeouts are sensible, and whether slow responses pile up into a wider outage. Service meshes such as Istio can add a fixed delay to a chosen share of requests through configuration.
People use the two names interchangeably for the practice on this page. In reliability terms, a fault is the event you introduce, such as a dropped packet, and a failure is what a user sees when the system cannot hide that fault. The goal of injecting faults is to prove they do not become failures.
The industry examples in “The next question”:
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Game Day
A planned exercise where teams simulate production failures to test incident response procedures and system resilience. Like a fire drill for your infrastructure.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
This page explains the idea. The "Chaos Engineering" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Chaos Engineering lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Failure Injection in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
