Is this a video course?

No. This is an interactive, slide-based learning platform. Each lesson has rich text, animated diagrams, live code editors, and quizzes. You learn by reading, interacting, and doing, not by watching videos passively.

How long do I have access?

Forever. Both pricing tiers are one-time payments with lifetime access. This includes all current 766 lessons and any future content we add.

What level of experience do I need?

None. We start from absolute basics like 'What is latency?' and build up to distributed consensus protocols. The Foundation level assumes zero prior knowledge of system design.

How much does the system design course cost?

7.99 US dollars for lifetime access globally, or 499 Indian rupees for lifetime access in India. One-time payment, no subscription, no hidden fees. 11 lessons are free with no signup required.

What technologies are covered?

Everything from DNS and load balancers to Kubernetes, Kafka, distributed databases, consensus protocols, stream processing, security architecture, and observability. We cover principles and real-world implementations used at Netflix, Google, Amazon, Uber, Stripe, and more.

Is this useful for system design interview preparation?

Yes. The lessons are structured around the exact topics asked in system design interviews at FAANG and top-tier companies. Interactive diagrams help you practice whiteboard-style explanations. Covers everything from URL shortener design to distributed payment systems.

How is this different from ByteByteGo or Educative?

766 interactive lessons (4x more than most competitors), 16 different diagram types that build step by step, real production examples from Netflix, Google, Amazon, Uber, and Stripe, and lifetime access for a one-time payment of $7.99 instead of annual subscriptions costing 100 to 200 dollars per year.

How do you validate an LLM judge?

You treat it as a binary classifier: have a human expert label a sample of outputs pass or fail, run the judge on the same outputs, and measure how often they agree using TPR, TNR, and Cohen's kappa. You then sharpen the rubric and repeat until the judge sits near the human ceiling.

What are TPR and TNR for a judge?

TPR, the true positive rate, is the fraction of human-passed outputs the judge also passes, so a low TPR means the judge fails good work and blocks your team. TNR, the true negative rate, is the fraction of human-failed outputs the judge also fails, so a low TNR means the judge passes bad work through to users.

Why is accuracy a misleading metric for a judge?

Because under class imbalance it rewards laziness: if 80 percent of outputs are good, a judge that stamps PASS on everything scores 80 percent accuracy while catching zero failures. TPR and TNR look at the good and bad cases separately, so they expose that the always-pass judge has a TNR of 0.

What is the human ceiling in judge validation?

The human ceiling is how much two qualified experts agree with each other on the same set, measured with Cohen's kappa. A judge cannot honestly do better, because on cases where the humans disagree there is no single right answer to match, so you grade the judge against the ceiling rather than against perfect agreement.

What counts as a good kappa for an LLM judge?

A good kappa is one that sits close to the human-human ceiling, not one that approaches 1.0. Shopify drove a judge from 0.02, which is chance, up to 0.61 against a human ceiling of 0.69, and 0.61 just under that ceiling counts as excellent.

Why report a confidence interval with TPR and TNR?

Because each rate is computed from a sample, and TNR in particular rests only on the failing cases, so with a handful of failures the interval can run from about 0.30 to 0.95. Reporting the bare point estimate pretends to a precision you do not have, so oversample failures to tighten the interval before trusting the number.

On a validation set where 80 percent of outputs are genuinely good, a judge that marks every single output PASS will have which scores?

High accuracy but a TNR of 0, because it catches none of the failures, Because 80 percent of outputs are good, always saying PASS is right 80 percent of the time, so accuracy looks fine. But TNR = TN / (TN + FP), and an always-pass judge produces zero true negatives, so its TNR is 0. It catches no failures. This is exactly why accuracy alone is a trap under class imbalance and why you always report TPR and TNR.

A safety judge screens outputs for harmful content. Which metric matters most, and why?

TNR, because a low TNR means harmful outputs get passed through to users, TNR = TN / (TN + FP) is the fraction of truly bad outputs the judge also fails. A low TNR means the judge passes harmful content, which reaches users. That is the dangerous direction for a safety judge, so TNR is the number to defend. Accuracy would hide a low TNR under imbalance, and TPR only measures how the judge treats good outputs.

Two human experts labeling the same set agree at a Cohen's kappa of 0.69. Your tuned judge reaches kappa 0.61 against the human labels. What is the right read?

The judge is excellent, because 0.61 sits just under the human-human ceiling of 0.69, Humans do not perfectly agree with each other, so human-human agreement is the ceiling a judge can reach. At kappa 0.69 for two humans, a judge at 0.61 is close to that ceiling and is trustworthy. Chasing 1.0 would mean forcing agreement on the very cases humans themselves disagree about, which is chasing noise. This mirrors Shopify's reported result of driving a judge from 0.02 to 0.61 against a 0.69 ceiling.

Your judge's TNR came out at 0.75, but its 95 percent confidence interval runs from 0.30 to 0.95. What should you do?

Add more failing examples to the validation set so the interval tightens before you rely on the TNR, A wide interval means the estimate rests on too few cases. TNR is computed only from the failing examples, so if there are just a handful, the interval is huge and the 0.75 is not yet reliable. The fix is to oversample failures so TNR rests on more rows and the interval narrows. Reporting the bare point estimate, or falling back to accuracy, hides the uncertainty rather than reducing it.

Validating Your Judge: TPR, TNR, and the Human Ceiling

Name: System Design Masterclass
Availability: InStock

in Evals

This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.

Global

$20/lifetime

One-time payment via PayPal

Get Lifetime Access

5 Foundation lessons free, try before you buy

India Special

India

₹999/lifetime

One-time via UPI, cards, or net banking

Get Lifetime Access

5 Foundation lessons free, try before you buy

“The interactive diagrams alone are worth way more than the price.”

Anish K., Senior Engineer at a Bengaluru startup

Every ML Platform lesson, Foundation to Advanced

Real architecture diagrams with real tool logos

MLOps, LLMOps, and data engineering in depth

Quizzes with instant feedback

Built and reviewed by senior engineers

Lifetime access, all future content included

The first 5 Foundation lessons of this course are free, no payment needed. Try them now