We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
In short
Chaos engineering is the practice of deliberately injecting failures into a running system (killing servers, dropping network packets, slowing down disks) to find weaknesses before they cause a real outage. You form a hypothesis about how the system should behave under a specific failure, run a controlled experiment in production or a production-like environment, and measure whether reality matches the hypothesis.
In short
Chaos engineering is the practice of deliberately injecting failures into a running system (killing servers, dropping network packets, slowing down disks) to find weaknesses before they cause a real outage. You form a hypothesis about how the system should behave under a specific failure, run a controlled experiment in production or a production-like environment, and measure whether reality matches the hypothesis.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Chaos Engineering across a run of lessons, not one page. These 4 alone are about 79 minutes of step-by-step reading, every one with a quiz.
Intentionally inject failures to verify resilience. Chaos Monkey, Gremlin, and Litmus.
Define and measure Service Level Objectives, the practice that aligns reliability with business needs.
Separating configuration from code, the key to running the same application in development, staging, and production.
Find the true cause of incidents. Five Whys, fishbone diagrams, and systemic thinking.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Chaos Engineeringpay once,
yours for life
Most reliability work assumes you already know how your system fails. Chaos engineering flips that around. You break things on purpose, watch what happens, and learn where the gaps are. The point is not to cause damage. The point is to surface the surprises while you are watching, instead of at 3 AM when a customer reports it.
The discipline was formalized at Netflix around 2010 when they moved to AWS. They could not assume any single server would stay alive, so they built Chaos Monkey, a tool that randomly terminates production instances during business hours. If killing a random server took the site down, that was a bug in the architecture, and engineers fixed it. Over time the team forced every service to tolerate the loss of any instance at any time.
A chaos experiment has four parts. First, define the steady state, a measurable signal that means the system is healthy, like requests per second or checkout success rate. Second, write a hypothesis, for example the steady state holds even if one availability zone goes dark. Third, inject the failure. Fourth, compare. If the steady state held, your confidence goes up. If it broke, you found work to do.
Failure injection happens at different layers. At the infrastructure layer you terminate VMs, drain Kubernetes nodes, or detach disks. At the network layer you add latency, drop packets, or block traffic between two services to simulate a partition. At the application layer you throw exceptions, exhaust connection pools, or pin CPU at 100 percent. Tools like Gremlin, AWS Fault Injection Service, Chaos Mesh, and LitmusChaos do this in a controlled, reversible way.
Blast radius control is the part beginners skip and regret. You start tiny: one instance, one percent of traffic, one customer cohort. You run during working hours with engineers watching dashboards, not overnight. You wire in an automatic abort, a kill switch that ends the experiment the moment the steady state degrades past a threshold. Only after a failure mode survives a small test do you widen the scope.
Mature teams automate this. Netflix later built Chaos Monkey into a broader Simian Army and then a service called ChAP that runs continuous experiments and compares a small canary group against a control group in real time, automatically stopping if the canary does worse.
Chaos engineering pays off most for distributed systems where failure is constant and the interactions are too complex to reason about on paper: microservices, multi-region deployments, anything with retries, timeouts, and fallbacks. In those systems the dangerous bugs are emergent. A retry storm or a cascading timeout only shows up when a real dependency slows down, which is exactly what an experiment reproduces on demand.
The trade-off is real risk and real cost. Running experiments in production can cause a genuine outage if your blast radius control is weak, so it demands good observability, fast rollback, and organizational trust first. If you cannot measure your steady state or you cannot abort quickly, you are not ready. It also takes engineering time to build the tooling and to fix everything the experiments reveal.
A common mistake is treating it as random destruction. Killing things with no hypothesis and no metrics is just sabotage. The value comes from the scientific loop: predict, break, measure, learn. Start in staging if production feels too risky, but know that staging rarely reproduces the load and data patterns where the interesting failures live.
Say you run a checkout service that calls a payments service, and you added a 2 second timeout plus a retry. You believe checkout stays healthy even if payments gets slow. To test it, you use a network fault injection tool to add 5 seconds of latency to 5 percent of calls from checkout to payments, during the afternoon, with one engineer watching the checkout success rate dashboard.
Within a minute you notice checkout success drops far more than 5 percent. The retries are doubling the load on an already slow payments service, the thread pool in checkout fills up waiting on slow calls, and requests that have nothing to do with payments start failing too. You found a cascading failure and a missing circuit breaker. You abort the experiment, the latency goes away, and you go fix the timeout, add a circuit breaker, and cap retries with backoff. Next week you rerun the same experiment and the steady state holds. That is the whole loop, and it caught a serious bug while you were calmly watching instead of during a real payments slowdown.
Netflix
Created Chaos Monkey and the Simian Army to randomly kill production instances; later built ChAP for automated canary-versus-control chaos experiments.
Amazon Web Services
Offers Fault Injection Service (FIS) as a managed product so customers can run controlled experiments against EC2, ECS, EKS, and RDS.
Gremlin
A commercial chaos engineering platform that injects resource, network, and state failures with built-in blast radius limits and a one-click halt.
Kubernetes ecosystem
Chaos Mesh and LitmusChaos inject pod kills, network partitions, and IO delays as native Kubernetes custom resources.
No, but production is where the real value is because staging rarely matches production load, data, and traffic patterns. Many teams start in staging to build confidence and tooling, then graduate to production with a tightly controlled blast radius once they can measure their steady state and abort an experiment within seconds.
Testing checks known behaviors against expected outputs, usually before deploy. Chaos engineering probes unknown emergent behavior in a live distributed system by injecting real failures like server loss or network latency and watching whether the system stays healthy. It complements testing, it does not replace it.
Chaos Monkey is Netflix's specific tool that randomly terminates production instances to force services to tolerate instance loss. It is the famous origin story, not the entire discipline. Chaos engineering also covers network latency, partitions, dependency failures, resource exhaustion, and region outages, all driven by hypotheses and metrics.
Three things. Solid observability so you can define and watch a steady state metric, a fast and reliable rollback or kill switch to end an experiment instantly, and a small controllable blast radius so a single experiment cannot take down the whole system. Without these, an experiment can turn into a self-inflicted outage.
No. Any team running microservices, multiple availability zones, or external dependencies with retries and timeouts benefits, because those systems have failure modes you cannot reason about on paper. Managed tools like AWS FIS and Gremlin lower the barrier so smaller teams can start with a single small experiment.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Bulkhead
A pattern that isolates different parts of a system so a failure in one part doesn't sink the whole ship. Named after the compartments in a ship's hull.
Retry
Automatically re-attempting a failed operation, usually with exponential backoff. Essential for handling transient failures in distributed systems.
Failure Injection
Deliberately introducing faults (latency, errors, crashes) into a system to verify that resilience mechanisms work. A specific technique within chaos engineering.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Chaos Engineering" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Chaos Engineering lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Chaos Engineering in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
