We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Devops Cicd
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Devops Cicd
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A team rewrote their recommendation engine. They tested it in staging with synthetic data for three weeks. All metrics looked good. They deployed to production. Recommendations were terrible.
The synthetic data in staging did not capture the patterns, edge cases, and data quality issues present in real production data. Users with empty profiles, products with missing categories, historical data with timezone bugs. None of these existed in staging.
Shadow testing (also called traffic mirroring or dark launching) sends a copy of real production traffic to the new version. The new version processes the requests, but its responses never reach users. Instead, a diff engine compares responses from both versions. You learn how the new version behaves with real-world data, at real-world scale, with zero risk to anyone.
Twitter used this approach when migrating from Ruby to Scala. Digg used it (and famously botched the validation step) during their v4 rewrite. Netflix runs shadow tests on every major algorithm change before it touches a single user's home screen.
The traffic splitter sits at the edge of your infrastructure. Every incoming request gets two destinations: the live production service, and a shadow copy of that same request.
| Step | What Happens | Who Sees It |
|---|---|---|
| 1 | Request arrives at the | User waits |
| 2 | Load balancer forwards the request to production v1 | v1 processes normally |
| 3 | Load balancer also copies the request to shadow v2 | v2 processes, but silently |
| 4 | v1 returns its response to the user | User gets their answer |
| 5 | Both v1 and v2 responses land in a diff engine | Nobody sees this yet |
| 6 | Diff engine logs differences in correctness, , errors | Engineers review later |
The user only ever sees v1's response. v2's response is discarded after comparison. If v2 crashes, returns garbage, or takes 30 seconds, users feel nothing.
What shadow testing reveals that staging cannot:
| Signal | What It Tells You |
|---|---|
| Response differences | New version produces different results for real queries |
| Error rate delta | New version errors on requests the old version handles fine |
| Latency comparison | New version is 2x slower on large payloads you never tested |
| Resource usage | New version needs 3x more memory under real traffic patterns |
| Edge case failures | Unicode names, empty arrays, null foreign keys, 10MB request bodies |