Is this a video course?

No. This is an interactive, slide-based learning platform. Each lesson has rich text, animated diagrams, live code editors, and quizzes. You learn by reading, interacting, and doing, not by watching videos passively.

How long do I have access?

Forever. Both pricing tiers are one-time payments with lifetime access. This includes all current 766 lessons and any future content we add.

What level of experience do I need?

None. We start from absolute basics like 'What is latency?' and build up to distributed consensus protocols. The Foundation level assumes zero prior knowledge of system design.

How much does the system design course cost?

7.99 US dollars for lifetime access globally, or 499 Indian rupees for lifetime access in India. One-time payment, no subscription, no hidden fees. 11 lessons are free with no signup required.

What technologies are covered?

Everything from DNS and load balancers to Kubernetes, Kafka, distributed databases, consensus protocols, stream processing, security architecture, and observability. We cover principles and real-world implementations used at Netflix, Google, Amazon, Uber, Stripe, and more.

Is this useful for system design interview preparation?

Yes. The lessons are structured around the exact topics asked in system design interviews at FAANG and top-tier companies. Interactive diagrams help you practice whiteboard-style explanations. Covers everything from URL shortener design to distributed payment systems.

How is this different from ByteByteGo or Educative?

766 interactive lessons (4x more than most competitors), 16 different diagram types that build step by step, real production examples from Netflix, Google, Amazon, Uber, and Stripe, and lifetime access for a one-time payment of $7.99 instead of annual subscriptions costing 100 to 200 dollars per year.

What is the difference between offline evals and online experiments?

Offline evals run a change against a fixed dataset to verify it did not regress a metric you can already measure, and they are fast, cheap, and run on every commit. Online experiments split live traffic to validate that the change actually moved a real business metric like conversion, and they are slow and expensive but the only true ground truth. You need both because a change can pass offline and still do nothing online.

Why does a change that scores higher offline sometimes do nothing online?

Because an offline metric is a proxy two steps removed from what you care about, and each step loses information. The offline score tracks an online metric, which tracks a business outcome, and a gain can fail to survive either jump. The most common concrete cause is distribution shift: the frozen golden set no longer matches live traffic, so a change tuned to the set barely touches real users.

What are guardrail metrics in an A/B test?

Guardrail metrics are secondary metrics you watch to catch a win on the target metric that quietly hurts something else, such as latency, cost per query, error rate, or retention. You declare them before the experiment launches, and a target win that trips a guardrail is not shipped automatically; it becomes an explicit trade-off decision. They exist because optimizing one metric can silently damage another.

Why can't you just A/B test every prompt change?

Because the sample size needed to detect an effect scales as one over the effect size squared, so a ten-times-smaller effect needs a hundred times more traffic. Most prompt tweaks produce small effects that would need hundreds of thousands of users per arm and months of traffic to resolve, which no team has for every change. That is why offline evals carry the small changes and A/B tests are reserved for the risky or large ones.

What is the minimum detectable effect (MDE)?

The minimum detectable effect is the smallest true difference an experiment can reliably detect given its sample size, confidence level, and power. For a 10 percent baseline with 8,000 users per arm at 95 percent confidence and 80 percent power, the MDE is about 1.3 percentage points, so any smaller real effect stays invisible. Computing the MDE before launching tells you whether an experiment can answer your question at all.

Do offline evals or online experiments matter more for shipping LLM features?

Neither replaces the other; they answer different questions and run in sequence. Offline evals gate what you are allowed to ship by catching regressions cheaply on every change, and online experiments validate that the shipped change moved a real metric. Mature teams run offline in CI as a veto and reserve online experiments for changes that are risky or large enough to justify the traffic.

A retrieval change raises the offline faithfulness score by six points and passes the CI gate. What has the offline eval actually established?

That the change did not regress a metric you can measure, which is permission to test, not proof of impact, Offline evals verify, they do not validate. A fixed dataset can only tell you a change did not make a known metric worse. The offline score is a proxy two rungs below the business outcome, so a gain there is a hypothesis about user behavior, not evidence of it. A green gate is permission to run the experiment, and only the experiment can tell you whether users actually cared.

An A/B test shows the target metric, helpfulness, up 3.1 points and significant. Latency p95 is up 310 ms and 7-day retention is down 0.8 points. What is the right move?

Treat it as an explicit trade-off: decide with the numbers whether the helpfulness gain is worth the latency and retention cost, Guardrail metrics exist to catch a target win that quietly hurts something else. A guardrail regression does not automatically kill a change, and a target win does not automatically ship it. It becomes a deliberate human trade-off with the numbers in front of you: is a 3.1 point helpfulness gain worth 310 ms of latency and 0.8 points of retention. Reading the target metric alone would have hidden the loss entirely.

Your surface gets 8,000 users per arm. A prompt tweak is expected to lift conversion by about 0.2 points on a 10 percent baseline. Should you A/B test it?

No, the effect is far below the minimum detectable effect at that sample size, so the test could never resolve it, Sample size scales as one over the effect squared, and at 8,000 users per arm the minimum detectable effect on a 10 percent baseline is about 1.3 points. A 0.2 point expected effect sits well below that floor, so the experiment would return noise no matter how long it ran. This is exactly why small low-risk changes ship behind the offline gate with guardrails rather than getting a formal experiment they cannot power.

Why do mature teams gate every change with offline evals but only A/B test some of them?

Because offline is cheap enough to run on every change while experiments cost traffic and time, so you spend experiments only on risky or large changes, It is a division of labor forced by cost and arithmetic. Offline evals run in minutes on every commit as a regression veto but only measure a proxy. Online experiments measure ground truth but are too expensive in traffic and calendar time to spend on small effects, and most changes have small effects. So offline carries the many and A/B tests validate the few that are risky enough or large enough to earn a traffic slot.

Offline vs Online: Evals Verify, Experiments Validate

Name: System Design Masterclass
Availability: InStock

in Evals

This lesson is part of the ML Platform Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.

Global

$20/lifetime

One-time payment via PayPal

Get Lifetime Access

5 Foundation lessons free, try before you buy

India Special

India

₹999/lifetime

One-time via UPI, cards, or net banking

Get Lifetime Access

5 Foundation lessons free, try before you buy

“The interactive diagrams alone are worth way more than the price.”

Anish K., Senior Engineer at a Bengaluru startup

Every ML Platform lesson, Foundation to Advanced

Real architecture diagrams with real tool logos

MLOps, LLMOps, and data engineering in depth

Quizzes with instant feedback

Built and reviewed by senior engineers

Lifetime access, all future content included

The first 5 Foundation lessons of this course are free, no payment needed. Try them now