We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Devops Cicd
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Devops Cicd
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A product team spent three months redesigning the checkout flow. Internal reviews were positive. The design looked cleaner, more modern. They deployed it to all users. Conversion rate dropped 15%. The "cleaner" design had fewer visual cues, confused returning users, and hid the promo code field. Three months of work made things worse.
A/B testing would have caught this in a week. Deploy the new checkout to 50% of users, keep the old one for the other 50%, and measure conversion rate. The data would have shown the regression immediately, limited to half the user base, with instant rollback.
This is not the same as . Canary asks "is the new version safe?" A/B testing asks "is the new version better?" The distinction matters because A/B testing needs statistical rigor, not just error rate monitoring.
A/B testing infrastructure has five distinct components, and skipping any of them leads to unreliable results.
| Component | Role | Example Tools |
|---|---|---|
| Experiment service | Define experiments, variants, targeting rules | LaunchDarkly, Optimizely, in-house |
| Assignment service | Deterministically assign users to variants | Hash(user_id + experiment_id) % 100 |
| Feature flag system | Route code paths based on assigned variant | LaunchDarkly, Unleash, Flagsmith |
| Analytics pipeline | Collect metrics per variant | Segment, Snowflake, BigQuery |
| Statistical engine | Determine if results are significant | Internal stats service, Eppo, Statsig |
Deterministic assignment is non-negotiable. A user must always see the same variant. If user 12345 sees variant B on their first visit, they must see variant B on every subsequent visit. This is done by hashing:
variant = hash(user_id + experiment_id) % 100
if variant < 50: show A
else: show B
Hashing is deterministic, stateless, and does not require a database lookup per request.
Sample size requirements. You cannot call a winner after 50 users. The minimum depends on the effect size you want to detect:
| Expected Effect | Users Per Variant | Total Users Needed |
|---|---|---|
| 10% improvement | ~400 | ~800 |
| 5% improvement | ~1,600 | ~3,200 |
| 2% improvement | ~10,000 | ~20,000 |
| 1% improvement | ~40,000 | ~80,000 |
Netflix runs experiments for 2-4 weeks to get enough data. Booking.com runs over 1,000 concurrent experiments. Google ran 7,000 A/B tests on Search alone in 2019.