Fault Tolerance

Every real system runs on hardware that fails. Disks die, machines crash, power supplies blow, network cables get unplugged, and entire data centers go dark. Fault tolerance is the property that lets a system absorb those failures and keep serving correct results to users instead of returning errors or going completely offline.

The key word is correctly. A system that stays up but starts returning wrong answers or losing data is not fault tolerant, it is just available. True fault tolerance means the visible behavior stays correct even while components underneath are broken.

There is an important distinction between fault tolerance and high availability. High availability is a target measured in nines (99.99 percent uptime is about 52 minutes of downtime per year). Fault tolerance is the set of techniques you use to hit that target. You build fault tolerance to achieve availability.

The foundation is redundancy: never have exactly one of anything on the critical path. Run multiple application servers behind a load balancer, store data on multiple disks (RAID), and replicate that data across multiple machines and availability zones. If one copy dies, another takes over.

Redundancy is useless without failure detection. Systems use health checks and heartbeats: a load balancer pings each server every few seconds, and a node that misses several heartbeats in a row is marked dead and removed from rotation. Timeouts catch the case where a component is slow rather than crashed.

Once a failure is detected, the system has to recover. Failover promotes a standby to take over (a replica database is promoted to primary). Retries with exponential backoff handle transient errors. Circuit breakers stop hammering a dependency that is already down. Graceful degradation means shedding non-essential features so the core keeps working, for example serving stale cached data when the database is unreachable.

Distributed systems add a hard constraint: to tolerate the failure of N nodes through consensus, you typically need 2N plus 1 replicas so a majority can still agree. Protocols like Raft and Paxos use this quorum rule to keep data consistent even while some nodes are down.

More fault tolerance is not always better, because it costs money and complexity. Running three copies of every server roughly triples your compute bill, and replication across regions adds network latency to every write. You match the investment to how expensive an outage is: a payment system or air traffic control needs far more than a personal blog.

The trade-offs are concrete. Active-active redundancy gives instant failover but doubles cost and forces you to handle data conflicts. Active-passive is cheaper but failover takes seconds to minutes. Synchronous replication guarantees no data loss but slows down writes, while asynchronous replication is fast but can lose the last few seconds of data if the primary dies.

The CAP theorem sets a real limit. When a network partition splits your replicas, you must choose between staying available (and risking inconsistent answers) or staying consistent (and refusing some requests). There is no design that gives you all three during a partition, so fault tolerance is always about choosing which failures you protect against and which you accept.

Picture an online store running its product catalog on a single PostgreSQL server. That server is a single point of failure: when its disk dies, the whole site goes down. To make it fault tolerant, you add a streaming replica in a second availability zone that continuously copies every change from the primary.

You put both behind automatic failover tooling. A monitor health-checks the primary every second. When it misses three checks in a row, the tool promotes the replica to primary and repoints the application connection string. Customers see a few seconds of slowness during the switch instead of an hours-long outage.

You layer in graceful degradation on top. The product pages read through a Redis cache, so even during the failover window the catalog keeps loading from cache while writes (like placing an order) briefly queue or retry. This is roughly how large e-commerce sites survive a database node dying on Black Friday without customers ever noticing.