Is this a video course?

No. This is an interactive, slide-based learning platform. Each lesson has rich text, animated diagrams, live code editors, and quizzes. You learn by reading, interacting, and doing, not by watching videos passively.

How long do I have access?

Forever. Both pricing tiers are one-time payments with lifetime access. This includes all current 766 lessons and any future content we add.

What level of experience do I need?

None. We start from absolute basics like 'What is latency?' and build up to distributed consensus protocols. The Foundation level assumes zero prior knowledge of system design.

How much does the system design course cost?

7.99 US dollars for lifetime access globally, or 499 Indian rupees for lifetime access in India. One-time payment, no subscription, no hidden fees. 11 lessons are free with no signup required.

What technologies are covered?

Everything from DNS and load balancers to Kubernetes, Kafka, distributed databases, consensus protocols, stream processing, security architecture, and observability. We cover principles and real-world implementations used at Netflix, Google, Amazon, Uber, Stripe, and more.

Is this useful for system design interview preparation?

Yes. The lessons are structured around the exact topics asked in system design interviews at FAANG and top-tier companies. Interactive diagrams help you practice whiteboard-style explanations. Covers everything from URL shortener design to distributed payment systems.

How is this different from ByteByteGo or Educative?

766 interactive lessons (4x more than most competitors), 16 different diagram types that build step by step, real production examples from Netflix, Google, Amazon, Uber, and Stripe, and lifetime access for a one-time payment of $7.99 instead of annual subscriptions costing 100 to 200 dollars per year.

Why is GPU serving so expensive?

GPU serving is expensive because the hardware costs an order of magnitude more than a CPU and does not get cheaper when idle: a card like an H100 costs the same per hour at 5 percent utilization as at 95 percent. An over-provisioned fleet running through the night with almost no traffic can burn tens of thousands of dollars a month on idle cards.

How much GPU memory does a model need?

A model's weights need roughly the parameter count times the bytes per parameter, so a 7B model needs about 28 GB in FP32, 14 GB in FP16, or 7 GB in INT8. On top of that you need room for activations and, for language models, a KV cache that grows with every token and concurrent request, and the whole thing must fit in the card's VRAM or the process dies with an out-of-memory error.

How do you push more requests through one GPU?

You push more through one GPU with four techniques in order of return on effort: dynamic batching to run many requests in one pass, quantization to store weights in fewer bits so they fit smaller and run faster, distillation to train a small student model that imitates a big teacher, and caching to answer repeats without touching a card. Batching and FP16 quantization are close to free.

What is the difference between horizontal and vertical GPU scaling?

Horizontal scaling adds more replicas of the same GPU behind a load balancer, which buys throughput and scales elastically with traffic but does not make a single request faster. Vertical scaling moves to a bigger card or splits a model across cards, which is what lets a large model fit at all and can cut per-request latency; most platforms go vertical until the model fits, then horizontal for volume.

Why do new GPU serving replicas take minutes to become ready?

A new GPU replica must find a scarce GPU node, pull a multi-gigabyte CUDA image, and load gigabytes of weights into VRAM before it can answer, commonly two to three minutes. That cold-start tax is why GPU autoscaling keeps a warm floor of replicas, scales up early on a leading signal, and waits out a cooldown before scaling down so a brief lull does not cost a cold start on the next spike.

Is the most expensive GPU always the most expensive choice?

No. What matters is cost per inference, which is the hourly price divided by how many inferences the card completes per hour. A busy H100 can be the cheapest per request because its throughput scales faster than its price, while an idle H100 is the most expensive choice of all. Pick the card by whether the model fits in VRAM and how busy you can keep it, not by hourly sticker price.

How do you share one GPU across several small models?

Two ways on modern NVIDIA cards. MIG (Multi-Instance GPU) carves one physical card into up to seven isolated instances with their own memory and cores, giving hardware isolation between tenants. Time-slicing lets several models take turns on the same card with higher packing but no memory isolation, so a noisy neighbor can interfere. Use MIG when tenants must not affect each other, time-slicing when they are friendly.

Why must a model's weights fit in the GPU's VRAM for serving?

Because the weights are loaded into VRAM once and stay resident for the life of the replica, so VRAM sets the hard limit on model size, A serving replica loads the model into VRAM when it boots and keeps it there, rather than streaming it per request. That makes each card's VRAM the hard ceiling on how large a model, plus its activations and KV cache, you can serve. Run out and the process dies with an out-of-memory error.

Why should a GPU serving autoscaler scale on GPU utilization or queue depth rather than CPU usage?

GPU work does not show up in CPU usage, so a fully saturated GPU can look idle to CPU-based autoscaling and never trigger a scale-up, Inference runs on the GPU, and a card can be pinned at 100 percent while the host CPU sits nearly idle. If you autoscale on CPU, the signal never rises, so you never add replicas even as the GPU queue backs up. You scale on real GPU utilization (for example DCGM_FI_DEV_GPU_UTIL) or on request queue depth instead.

What is the trade-off that dynamic batching makes?

It adds a few milliseconds of latency per request (from the wait window) in exchange for running many requests in one GPU pass, greatly increasing throughput, A GPU running one request leaves most of its cores idle. The batcher holds requests for a short window (a few milliseconds), forms a batch, and runs them together in one forward pass, so several requests cost almost the same GPU time as one. The price is the small added latency from waiting to fill the batch, which you tune with a max batch size and max wait.

A small model runs on an H100 that stays at about 15 percent utilization all day. Why is this often the most expensive choice per inference?

Because cost per inference is the hourly price divided by throughput, and a barely-used premium card pairs the highest hourly price with almost none of the throughput that would justify it, A busy H100 is cheap per request because its throughput scales faster than its price. But cost per inference is price divided by inferences actually served, so at 15 percent utilization the card is billing full price while completing a fraction of the requests it could. The fix is to fit the model on a smaller card, or to share the H100 with MIG or time-slicing, or to raise utilization with batching and packing.

Scaling and GPU Infrastructure: Serving Models Without Burning Money

Name: System Design Masterclass
Availability: InStock

in Core

This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.

Global

$20/lifetime

One-time payment via PayPal

Get Lifetime Access

5 Foundation lessons free, try before you buy

India Special

India

₹999/lifetime

One-time via UPI, cards, or net banking

Get Lifetime Access

5 Foundation lessons free, try before you buy

“The interactive diagrams alone are worth way more than the price.”

Anish K., Senior Engineer at a Bengaluru startup

Every ML Platform lesson, Foundation to Advanced

Real architecture diagrams with real tool logos

MLOps, LLMOps, and data engineering in depth

Quizzes with instant feedback

Built and reviewed by senior engineers

Lifetime access, all future content included

The first 5 Foundation lessons of this course are free, no payment needed. Try them now