We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Devops Cicd
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Devops Cicd
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A system was designed to "gracefully degrade" under excessive load by serving cached responses instead of querying the database. During the first real traffic spike, the team discovered what actually happened: when the database connection pool exhausted, the application threw unhandled exceptions. The error handlers allocated large error objects in a loop. The JVM ran constant garbage collection trying to free memory. Response times went from 200ms to 30 seconds. Then the health check failed and killed the pod. The new pod started, immediately hit the same traffic, and died again. Crash loop.
There was no graceful degradation. Nobody had tested it under actual stress conditions. The "graceful degradation" code path had been written but never exercised because unit tests mocked the database layer and always returned successful responses.
Stress testing is not about verifying normal performance. Load testing already does that. Stress testing is about discovering what happens when things go wrong: when traffic exceeds capacity by 2x or 5x, when a dependency fails, when resources exhaust. The goal is to find the breaking point and ensure the system fails predictably, not catastrophically.
| Discovery | Why It Matters | Example Finding |
|---|---|---|
| Breaking point | Know your hard limit | "We crash at 12,000 RPS" |
| Failure mode | Graceful or catastrophic? | "DB pool exhaustion causes crash loops" |
| Recovery time | How long to recover after overload? | "Takes 4 minutes to recover after spike" |
| Resource bottleneck | What fails first? | "Memory exhausts before CPU saturates" |
| Cascading failures | Does one failure spread? | "Auth service down takes down checkout" |
| Data integrity | Does stress cause data loss? | "Queue drops messages above 50K/sec" |
Stress test scenarios every system should run:
| Scenario | How to Test | What You Learn |
|---|---|---|
| Traffic spike | Ramp from normal to 10x in 30 seconds | Autoscaling speed, queue depth |
| Database failure | Kill the primary DB connection | Failover behavior, error handling |
| Dependency timeout | Add 10s to an external service | activation, retry storms |
| Memory pressure | Reduce container memory limit by 50% | OOM behavior, GC thrashing threshold |
| Disk full | Fill disk to 95% during write-heavy load | Write failure handling, log rotation |
| Network partition | Drop packets between service A and service B | Timeout handling, retry behavior |
Netflix invented Chaos Monkey in 2011 to stress-test their production systems. It randomly kills instances in production during business hours. Their reasoning: if the system cannot survive losing a single instance, it will not survive a real failure either. By 2024, Netflix's Chaos Engineering suite includes Chaos Monkey, Chaos Kong (region failures), and Chaos Gorilla (availability zone failures).