We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Devops Cicd
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Devops Cicd
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A team deployed a new version that passed all tests, smoke tests, and staging validation. They did a full blue-green switch to production. Within 10 minutes, 5% of users hit a race condition that only manifested under high concurrency. Something staging's low traffic never triggered. All users were affected because the switch was all-or-nothing.
Canary deployment solves this by sending a small slice of traffic (1-5%) to the new version first. If the canary is healthy after 15-30 minutes of observation, traffic gradually increases: 5%, then 25%, then 50%, then 100%. If the canary shows elevated errors at any step, that step rolls back immediately. Only the 1-5% ever saw the problem.
The term comes from coal mining. Miners carried canary birds into tunnels. If the canary died, the air was toxic. In deployment, a small fleet running the new version is your canary. If it dies (errors spike, degrades), you pull out before the rest of the fleet is affected.
Progressive traffic shifting:
| Stage | Traffic to v2 | Observation Window | Rollback Impact |
|---|---|---|---|
| Canary start | 1-5% | 15-30 min | 1-5% of users saw the issue |
| First expansion | 10-25% | 15-30 min | Up to 25% saw the issue |
| Majority | 50% | 15-30 min | Up to 50% saw the issue |
| Full rollout | 100% | Continuous | Standard incident response |
Automated canary analysis tools:
| Tool | Platform | How It Works |
|---|---|---|
| Argo Rollouts | Native CRD, integrates with | |
| Flagger | Kubernetes + Istio/Linkerd | -aware, automated promotion |
| Spinnaker + Kayenta | Multi-cloud | Google's internal tool, open-sourced |
| AWS AppMesh | AWS | Built into the mesh, CloudWatch metrics |
These tools compare canary metrics against the baseline automatically:
| Metric | Canary (v2) | Baseline (v1) | Threshold | Decision |
|---|---|---|---|---|
| Error rate | 0.3% | 0.2% | < 0.5% delta | Continue |
| p99 | 420ms | 400ms | < 50ms delta | Continue |
| Error rate | 2.1% | 0.2% | < 0.5% delta | Exceeds. Rollback. |
The observation window matters. Five minutes is too short. Latency percentiles need time to stabilize, especially p99. A common mistake: promoting a canary after 5 minutes of "good" metrics, only to find the regression appears at minute 20 when the request mix shifts. Fifteen minutes is the minimum. Thirty is safer.