We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In the early 1900s, miners carried a caged canary into the coal mine. If the canary stopped singing, the air was toxic and the miners got out. The bird was a cheap, fast, expendable detection system. It absorbed the risk so the humans did not have to.
Canary deployments apply the same principle to software releases. Instead of pushing a new version to every server at once and hoping it works, you send it to 1% of production traffic first. If that 1% shows elevated errors, high , or degraded business metrics, the deploy is rolled back before the other 99% ever sees it.
Stripe ships every backend change through a canary that starts at 1% of production traffic. Netflix Spinnaker runs canary analysis on hundreds of deploys per day using a component called Kayenta. Google pioneered the pattern internally before it had a name, running new Search ranking code on a single data center before promoting globally.
The payoff is simple math. A bug in a 1% canary affects 1% of users for 5 to 15 minutes. The same bug in a full rollout affects 100% of users for however long it takes an engineer to notice, investigate, and roll back. Knight Capital lost 440 million dollars in 2012 because a deployment reached 100% of production with no working rollback path.
A has four moving parts, and getting any of them wrong defeats the purpose.
| Component | Role | What Goes Wrong Without It |
|---|---|---|
| Traffic splitter | Routes a percentage of requests to the new version | All traffic hits the new version at once, defeating the canary |
| Same user always sees the same version during rollout | Users bounce between v1 and v2, creating cross-version state bugs | |
| Automated judge | Compares canary metrics against stable baseline in real time | Humans miss subtle regressions or react too slowly |
| Auto rollback | Reverts the canary on metric breach without human action | A bad deploy sits at 10% for hours until someone notices |
The traffic split uses the . AWS ALB supports weighted target groups natively. Envoy supports traffic splitting via route configuration. Kubernetes supports it through Istio VirtualService weights or Argo Rollouts. The mechanism is interchangeable. The principle is not.
Consistent hashing on user ID prevents cross-version session bugs. A user who sees v1 on one request and v2 on the next will encounter cookie collisions, cache key mismatches, and state inconsistencies that look like application bugs but are actually deployment artifacts. Netflix lost hours to this exact failure mode in 2019 before switching to consistent user hashing across all Spinnaker rollouts.