We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Airplanes have three independent hydraulic systems. If one fails, two remain. If two fail, one remains. The plane can fly and land safely with a single hydraulic system. This is fault tolerance: designing for the expectation that components will fail.
Fault-tolerant systems do not prevent failures. They accept that failures will happen and ensure the system continues operating correctly despite them. The key techniques are redundancy (have backup components), isolation (prevent failures from spreading), and graceful degradation (provide reduced functionality rather than complete failure).
Netflix pioneered chaos engineering, intentionally injecting failures into production to verify fault tolerance. Their Chaos Monkey randomly kills instances. Chaos Gorilla takes down entire availability zones. If the system survives, it is fault-tolerant. If it does not, they fix it before real failures happen.
| Technique | How It Works | Example |
|---|---|---|
| Redundancy | Multiple instances of critical components | 3 replicas of each |
| Isolation | Failures in one component do not affect others | Bulkhead pattern, circuit breakers |
| Graceful degradation | Provide reduced functionality during failures | Show cached data when DB is down |
| Failover | Automatically switch to backup on failure | Database primary-replica failover |
| Retry with backoff | Retry failed operations with increasing delays | Exponential backoff for API calls |
| Timeouts | Do not wait forever for unresponsive services | 5-second timeout on calls |
Design principle: Assume everything can and will fail. Network calls will timeout. Servers will crash. Databases will go down. Deploys will go wrong. Design for each of these scenarios explicitly.