Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · Core
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
In short
Model serving is running a trained model as a service that answers prediction requests over the network, within a time limit, for many callers at once. It covers the API, grouping requests for the GPU, scaling copies up and down, and releasing new versions safely.
In short
Model serving is running a trained model as a service that answers prediction requests over the network, within a time limit, for many callers at once. It covers the API, grouping requests for the GPU, scaling copies up and down, and releasing new versions safely.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Model Serving across a run of lessons, not one page. These 4 alone are about 260 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
How a trained model becomes a live API. Covers online vs batch serving, REST vs gRPC, dynamic batching, autoscaling and p99, with a batch simulator.
What happens when serving sends the right numbers in the wrong shape? I sent the features chapter's model 57 faults: swapped columns, a reversed order, money in pence, days as hours, a missing or extra column, ints, text and NaN. A plain numpy call raised an error for only 9 of them (20 of the 57 were random orders). All 36 wrong column orders came back with no error, and test AP fell from 0.5450 to as low as 0.1268. One customer's chance of buying went from 0.0529 to 0.9465. scikit-learn's feature names caught every order fault when the names moved with the values, and none when the right names were typed on top. An MLflow column signature put moved columns back in order and refused int64 columns and text, but let units and NaN through, like every other name or type check. A range check from the training rows fired on every clean test month. A rule check written from what each column means, after I had seen these faults, flagged no clean row and caught 44 of the 48 faults that reached it, but not money in pence.
Should a model score every customer in a nightly batch, or each request as it comes? I replayed 7,403 real visits to a UK online shop through the features chapter's model, scored four ways: online at the visit, or a stored score from a daily, weekly or monthly batch. On the 6,613 visits every way could score, with 20 seeds and a customer bootstrap, no batch could be told apart from online, even a monthly score up to 30.7 days old (-0.0027 to +0.0063 AP). An independent review found that the model, trained on 1st-of-month rows, leaned toward batch, so I trained a second model on visit-time rows: the lean went away and the verdict stayed. The costs were elsewhere: the nightly batch made 5,353 scores a day and 99.1% were never read, against 60 a day online, and 33 to 173 visits came from customers the batch had not scored yet. On a fast question, like bike rentals in the next hour, the course has already measured the opposite.
Where does the time in one prediction request go? I served the features chapter's model with FastAPI, uvicorn and SQLite on a quiet 1-vCPU EC2 box, client and service on one core, and timed every step of 20,000 real requests, five times. At the median, a request took 1,075.9 microseconds. predict_proba on one row took 583.8 of them (54%), and the work outside my own code, HTTP, the framework, the loopback and the client, took 397.0 (37%). The feature lookup, parsing, building the row and writing the answer together took less than 8%. An empty endpoint still cost 333.6 microseconds. Reading the request body had two humps, and a preview of lesson 4 found a one-row predict cost as much as 195 rows inside a 1,000-row call.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
In a notebook, model.predict(x) might return in 8 milliseconds. Put that behind a quick web app, send a few hundred requests per second, and requests start to queue while the GPU sits mostly idle. The model is fine; the serving is not. Serving means answering thousands of callers at once, inside a time budget, using an expensive GPU efficiently, surviving a crashed machine, and rolling out new versions without risk.
The first decision is the mode. Online serving, also called real-time serving, answers one request while a user or a process waits, such as a fraud check on a card payment, so latency matters most. Batch serving scores a large set of rows on a schedule with nobody waiting, such as churn scores for every account overnight, so total throughput and cost matter most. Streaming serving scores an endless flow of events and must keep up with the arrival rate.
The modes have opposite costs. Online serving keeps copies running and ready, so you pay for availability. Batch serving starts compute, uses it fully and shuts it down, so you pay only for the run. Many real systems use more than one, for example building candidate recommendations in a nightly batch and re-ranking the top few online when the app opens.
An online request rarely reaches the model directly. It first hits an API gateway or load balancer, which ends the encrypted TLS connection, checks the API key, applies rate limits and spreads traffic across identical model server copies, called replicas, often run on Kubernetes. A prediction cache, often Redis, can return a repeated answer in under a millisecond. The cache key must include the model version, or a new deployment will serve old answers.
On a cache miss, the server usually receives an id, not a ready input, so it fetches features from an online feature store. Those features must be computed exactly as they were in training, or you get train-serve skew. Then the model runs its forward pass on the GPU.
Two more choices shape the API. REST with JSON is easy to read and debug; gRPC with Protobuf sends compact binary data over HTTP/2, so a vector of 512 numbers takes about 2 kilobytes at 4 bytes each instead of several kilobytes of text. Many servers offer both. And if a prediction takes seconds, as with long text generation, serve it asynchronously: accept the job, return a job id at once, and deliver the result by polling or a callback.
A GPU runs a batch of, say, 32 inputs in about the same time as a single input, because it does the work in parallel. Feed it one request at a time and it idles between requests. Dynamic batching holds each incoming request for a short, fixed window, such as up to 5 milliseconds, collects whatever else arrives, and runs them together. The window closes when the batch reaches its maximum size or the timer runs out, whichever comes first.
The arithmetic shows why this matters. If one forward pass takes 25 milliseconds and requests are served one at a time, a single GPU can clear only about 40 requests per second. At 800 arrivals per second, the queue grows without limit. With batching, each pass carries dozens of requests, and latency settles near the batch window plus one pass.
Batching has a limit. As you push more requests through one replica, throughput rises until the GPU is saturated, then flattens while latency climbs sharply. Run each replica near that bend in the curve, and add replicas to handle more traffic. For large language models, servers such as vLLM and Triton use continuous batching, where new requests join a batch that is already running instead of waiting for the next one.
Serving quality is judged by the slow end, not the average. p50 is the time half of requests beat; p99 is the time 99 in 100 beat, so 1 in 100 is slower. If a page makes five model calls, the chance at least one hits the p99 slow path is 1 minus 0.99 to the fifth power, about 5 percent of page loads. That is why teams watch p99 closely.
Treat the latency target as a budget and assign each step a share: network, authentication, cache lookup, batch wait, preprocessing, the GPU pass, post-processing. The GPU pass is usually the largest line, so a smaller or quantized model, or better-filled batches, moves the total most.
Traffic changes during the day, so replicas are added and removed automatically. A new replica is not useful at once: it must pull a large container image, start the runtime and load the weights onto the GPU, which can take 30 to 90 seconds for a large model. Scale on remaining latency headroom so capacity arrives before p99 breaks, and keep a minimum number of warm replicas for latency-critical models instead of scaling to zero.
A model that did well offline can still fail on live traffic. Shadow deployment sends a copy of real requests to the new version and throws its answers away, so you catch crashes and slowdowns with no user impact. A canary sends a small share of real traffic, such as 5 percent, to the new version and watches error rate, p99 and the business metric, then raises the share step by step, for example to 25 and then 100 percent. Shadow shows it is safe; the canary shows whether it is better.
Log the model version with every prediction and include it in the cache key. When two versions run at once, you need to know which one produced each answer.
You rarely build this machinery yourself. NVIDIA Triton serves many frameworks with advanced batching. TensorFlow Serving and TorchServe fit teams using one framework. BentoML wraps any Python model and adds batching with a few settings. And if nobody waits on each prediction, a scheduled batch job is simpler and cheaper than any real-time endpoint.
NVIDIA Triton Inference Server
An open-source inference server that serves models from several frameworks, including TensorRT, PyTorch and ONNX, with dynamic batching over HTTP and gRPC.
vLLM
An open-source serving engine for large language models that uses continuous batching and PagedAttention to serve many requests on one GPU.
BentoML
An open-source Python framework that wraps a model as an API service, with request batching settings and container packaging.
TensorFlow Serving
Google's open-source server for TensorFlow SavedModels, with REST and gRPC endpoints and request batching.
Online serving answers each request while someone waits, so latency comes first. Batch serving scores many rows on a schedule with nobody waiting on any single row, so throughput and cost come first. Pick by asking whether anyone is blocked on this prediction right now.
The server holds each request for a few milliseconds, gathers the other requests that arrive, and runs them in one GPU pass. A GPU handles a batch in about the time it takes for one input, so this multiplies throughput for a small, bounded wait.
Because the average hides the slow tail that users feel. When a page makes several model calls, a rare slow call is hit often; with five calls, about 5 percent of page loads include at least one p99-slow response.
REST with JSON suits public and browser clients and is easy to debug. gRPC with Protobuf suits high-volume internal traffic and large inputs, because its binary format is smaller and it supports streaming. Many model servers offer both.
It is the delay before a new replica can answer, while it pulls its container image, starts up and loads weights onto the GPU. For a large model this can take 30 to 90 seconds, so teams scale early and keep some replicas always warm.
Model Signatures: The Right Numbers in the Wrong Shape, and What a Schema Check Catches
What happens when serving sends the right numbers in the wrong shape? I sent the features chapter's model 57 faults: swapped columns, a reversed order, money in pence, days as hours, a missing or extra column, ints, text and NaN. A plain numpy call raised an error for only 9 of them (20 of the 57 were random orders). All 36 wrong column orders came back with no error, and test AP fell from 0.5450 to as low as 0.1268. One customer's chance of buying went from 0.0529 to 0.9465. scikit-learn's feature names caught every order fault when the names moved with the values, and none when the right names were typed on top. An MLflow column signature put moved columns back in order and refused int64 columns and text, but let units and NaN through, like every other name or type check. A range check from the training rows fired on every clean test month. A rule check written from what each column means, after I had seen these faults, flagged no clean row and caught 44 of the 48 faults that reached it, but not money in pence.
Batch or Online Scoring: A Month-Old Score Cost Nothing I Could Measure Here
Should a model score every customer in a nightly batch, or each request as it comes? I replayed 7,403 real visits to a UK online shop through the features chapter's model, scored four ways: online at the visit, or a stored score from a daily, weekly or monthly batch. On the 6,613 visits every way could score, with 20 seeds and a customer bootstrap, no batch could be told apart from online, even a monthly score up to 30.7 days old (-0.0027 to +0.0063 AP). An independent review found that the model, trained on 1st-of-month rows, leaned toward batch, so I trained a second model on visit-time rows: the lean went away and the verdict stayed. The costs were elsewhere: the nightly batch made 5,353 scores a day and 99.1% were never read, against 60 a day online, and 33 to 173 visits came from customers the batch had not scored yet. On a fast question, like bike rentals in the next hour, the course has already measured the opposite.
Latency Anatomy: Where the Time in One Prediction Request Goes
Where does the time in one prediction request go? I served the features chapter's model with FastAPI, uvicorn and SQLite on a quiet 1-vCPU EC2 box, client and service on one core, and timed every step of 20,000 real requests, five times. At the median, a request took 1,075.9 microseconds. predict_proba on one row took 583.8 of them (54%), and the work outside my own code, HTTP, the framework, the loopback and the client, took 397.0 (37%). The feature lookup, parsing, building the row and writing the answer together took less than 8%. An empty endpoint still cost 333.6 microseconds. Reading the request body had two humps, and a preview of lesson 4 found a one-row predict cost as much as 195 rows inside a 1,000-row call.
Tail Latency: What Sits in the Slowest 1% of Requests
Why is p99 so far from the median, and what sits in the tail? I sent 100,000 real requests, one at a time, to the features chapter's model on a quiet 1-vCPU EC2 box, five times, and repeated it with four single changes. Here the p99 was only 1.08 times the median and the p99.9 1.23 times, but the slowest request of every run was the very first one, 7.7 times the median. My main guess was wrong: no garbage collection ran during any of the 500,000 measured base requests. Slow requests came in bursts in which every step was slower, and the request after a slow one was slow too 41% to 62% of the time. Because of those bursts, pages of 10 calls in a row met the tail 3.3% to 5.4% of the time, not the 9.56% of 1 - 0.99^10. A warm-up removed the slow first request; gc.freeze() and a higher gc threshold could not be told apart from run-to-run noise. Every AWS step is recorded so you can build the box yourself.
Batching Requests: Does It Help a Small CPU Model?
Does batching help a small model on a CPU, and what does it cost in latency? On a quiet 1-vCPU EC2 box, predict_proba on 1,024 rows took 5.24 times as long as on one row, so one row in a big batch was 195 times cheaper. Then I put a micro-batcher into the chapter's FastAPI service and sent it random (Poisson) arrivals from half to 2.5 times the most it answered with no batching (S, 1,098.4 a second): 17 settings, 7 rates, 5 rounds, every setting compared with no batching on the same arrivals. With no batching the service fell behind at S and gave about 1,045 answers a second at most. Batches of up to 32 with no timed wait kept up at every rate, gave 2.62 times as many answers at 2.5 x S, and even at half of S cut the p99 from 5.26 ms to 2.50 ms. Every timed wait of 1 ms or more raised the median at that low rate. All 7,179,151 scores were equal to the bit to single-row scores. Every AWS step is recorded so you can build the box yourself.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
Model Registry
A versioned catalog of trained models with their metadata, stage, and lineage, so you know exactly what is deployed and can roll back. The control plane for shipping models safely.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Train-Serve Skew
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
This page explains the idea. The "Model Serving and Inference APIs: Turning a Model File Into a Service" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Model Serving and Inference APIs: Turning a Model File Into a Service lessoncourse 2
Take models from notebook to production, with labs on real models.