May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
AI Engineering · Serving And Inference
Free preview: the first part of this 29-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Serving And Inference
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Let me start in a small office.
Three colleagues share one table and one laptop. Each of them has their own notebook with their own notes. Only one person can type on the laptop at a time, so they pass it round.

Now the manager wants more work done at the same table, with the same one laptop, and has two ideas.
Idea one: hire a fourth person. The fourth person brings a fourth notebook and takes a turn at the laptop. But the laptop does not get faster. Each person now waits a little longer for their turn, and passing the laptop round takes time too. If the work is all typing, four people finish no more than three did. They may finish a little less.
Idea two: ask three people to type one short line together. Someone has to call the other two over and wait for them to sit down. Then each gets a third of the line, and everyone waits for the last one to finish. For one short line, that costs much more than typing it alone. For a long report on a table with three laptops, it could be a good idea. On one laptop it never is.
A prediction service has the same two ideas. It can run more worker processes, like more people. Or it can let the model use more threads inside one call, like several people typing one line. In this lesson I measure both, and both together, on a rented machine with exactly one core: the one laptop of the story.