We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningFoundation · Core Fundamentals
A system's ability to keep operating correctly even when some of its components fail. Achieved through redundancy, replication, and graceful degradation.
In short
Fault tolerance is a system's ability to keep working correctly even when some of its parts fail, so a single broken disk, server, or network link does not take the whole system down. It is achieved mainly through redundancy (extra copies of components), replication (extra copies of data), failure detection, and graceful degradation when something does break.
In short
Fault tolerance is a system's ability to keep working correctly even when some of its parts fail, so a single broken disk, server, or network link does not take the whole system down. It is achieved mainly through redundancy (extra copies of components), replication (extra copies of data), failure detection, and graceful degradation when something does break.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Fault Tolerance across a run of lessons, not one page. These 4 alone are about 73 minutes of step-by-step reading, every one with a quiz.
Design systems that continue operating when components fail, redundancy, isolation, and graceful degradation.
When network calls fail, and they will, retries turn transient failures into invisible hiccups.
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover.
When parts of your system fail, give users a reduced experience instead of a broken one.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Fault Tolerancepay once,
yours for life
Every real system runs on hardware that fails. Disks die, machines crash, power supplies blow, network cables get unplugged, and entire data centers go dark. Fault tolerance is the property that lets a system absorb those failures and keep serving correct results to users instead of returning errors or going completely offline.
The key word is correctly. A system that stays up but starts returning wrong answers or losing data is not fault tolerant, it is just available. True fault tolerance means the visible behavior stays correct even while components underneath are broken.
There is an important distinction between fault tolerance and high availability. High availability is a target measured in nines (99.99 percent uptime is about 52 minutes of downtime per year). Fault tolerance is the set of techniques you use to hit that target. You build fault tolerance to achieve availability.
The foundation is redundancy: never have exactly one of anything on the critical path. Run multiple application servers behind a load balancer, store data on multiple disks (RAID), and replicate that data across multiple machines and availability zones. If one copy dies, another takes over.
Redundancy is useless without failure detection. Systems use health checks and heartbeats: a load balancer pings each server every few seconds, and a node that misses several heartbeats in a row is marked dead and removed from rotation. Timeouts catch the case where a component is slow rather than crashed.
Once a failure is detected, the system has to recover. Failover promotes a standby to take over (a replica database is promoted to primary). Retries with exponential backoff handle transient errors. Circuit breakers stop hammering a dependency that is already down. Graceful degradation means shedding non-essential features so the core keeps working, for example serving stale cached data when the database is unreachable.
Distributed systems add a hard constraint: to tolerate the failure of N nodes through consensus, you typically need 2N plus 1 replicas so a majority can still agree. Protocols like Raft and Paxos use this quorum rule to keep data consistent even while some nodes are down.
More fault tolerance is not always better, because it costs money and complexity. Running three copies of every server roughly triples your compute bill, and replication across regions adds network latency to every write. You match the investment to how expensive an outage is: a payment system or air traffic control needs far more than a personal blog.
The trade-offs are concrete. Active-active redundancy gives instant failover but doubles cost and forces you to handle data conflicts. Active-passive is cheaper but failover takes seconds to minutes. Synchronous replication guarantees no data loss but slows down writes, while asynchronous replication is fast but can lose the last few seconds of data if the primary dies.
The CAP theorem sets a real limit. When a network partition splits your replicas, you must choose between staying available (and risking inconsistent answers) or staying consistent (and refusing some requests). There is no design that gives you all three during a partition, so fault tolerance is always about choosing which failures you protect against and which you accept.
Picture an online store running its product catalog on a single PostgreSQL server. That server is a single point of failure: when its disk dies, the whole site goes down. To make it fault tolerant, you add a streaming replica in a second availability zone that continuously copies every change from the primary.
You put both behind automatic failover tooling. A monitor health-checks the primary every second. When it misses three checks in a row, the tool promotes the replica to primary and repoints the application connection string. Customers see a few seconds of slowness during the switch instead of an hours-long outage.
You layer in graceful degradation on top. The product pages read through a Redis cache, so even during the failover window the catalog keeps loading from cache while writes (like placing an order) briefly queue or retry. This is roughly how large e-commerce sites survive a database node dying on Black Friday without customers ever noticing.
Netflix
Runs across multiple AWS regions and deliberately kills its own servers in production with Chaos Monkey to prove the system survives instance failures.
Amazon DynamoDB
Replicates every item across three availability zones and uses quorum writes so a whole zone can fail without losing data.
Apache Kafka
Replicates each partition to multiple brokers; if the leader broker dies an in-sync follower is automatically promoted with no message loss.
Cloudflare
Anycasts traffic across hundreds of edge data centers so when one site goes offline requests are rerouted to the next healthy location.
High availability is the goal, measured as uptime percentage like 99.99 percent. Fault tolerance is the set of techniques (redundancy, replication, failover, graceful degradation) you use to reach that goal. You build fault tolerance to achieve high availability.
Not usually. A fully fault-tolerant design aims for no visible interruption, but most real systems accept a brief failover window of seconds. The point is surviving component failures without a long outage or data loss, not literally never blinking.
Any component that, if it fails, brings down the whole system because there is no backup. A lone database server, one load balancer, or a single network link are common examples. Eliminating single points of failure is the first step toward fault tolerance.
Fault tolerance handles routine failures automatically and in real time, like one server dying. Disaster recovery is the planned process for recovering from large rare events like a whole region going down, often involving backups and a recovery time measured in minutes or hours rather than seconds.
Consensus protocols like Raft and Paxos require a majority to agree before committing. With 2N plus 1 nodes you can lose N and still have a majority. An odd count avoids ties and gives you the best failure tolerance per node, which is why clusters commonly run 3, 5, or 7 nodes.
Retry Logic
When network calls fail, and they will, retries turn transient failures into invisible hiccups
Exponential Backoff
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover
Graceful Degradation
When parts of your system fail, give users a reduced experience instead of a broken one
Circuit Breaker Pattern
Prevent cascading failures in distributed systems by failing fast when a downstream service is unhealthy
Replication
Keeping copies of the same data on multiple servers. Improves read performance and provides fault tolerance if one server goes down.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Failure Injection
Deliberately introducing faults (latency, errors, crashes) into a system to verify that resilience mechanisms work. A specific technique within chaos engineering.
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
Throughput
The number of operations a system can handle per unit of time. Think of it as how many cars a highway can move per hour.
This page explains the idea. The "Fault Tolerance" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Fault Tolerance lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Fault Tolerance in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
