Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningEvery lab in the AI Engineering course runs a free language model on your own computer (a local LLM), through a program called Ollama. This page shows how to install it, which models to download, how to check that everything works, and how to use a different model if you prefer. It answers 21 questions, from a first install to the errors you might meet.
Save your progress and get the AI Engineering cheat sheet PDF
21 questions · 6 sections · answers from the course
These answers come from our AI Engineering course: 210 lessons, ₹999 once in India or $49 elsewhere once, with lifetime access.
Install Ollama from ollama.com. In a terminal, run ollama pull qwen2.5:3b and ollama pull nomic-embed-text. Make sure you have Python 3.9 or newer. Then run the check script on this page; when it prints "Ready for the labs." you can run any lab that uses those models.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers a local AI lab across a run of lessons, not one page. These 3 alone are about 142 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
A model reads one text, not a list of messages. Measured with Ollama and qwen2.5:3b: the message 'Hi' is 1 token as raw text and 30 tokens as a chat, because the template adds markers and a default system message. Filled in by hand, the template gave the same 39 tokens and the same reply as the chat endpoint. On 20 short instructions, its own template gave 20 clean replies; another model family's template gave 0, though 16 first answers were still right.
Every Ollama reply reports how long the model spent loading, reading the prompt and writing the answer. Measured on a laptop with qwen2.5:3b over 35 calls in a shuffled order: the median reading speed was 194 tokens a second and the median writing speed 30. Even the slowest reading of any call was faster than the fastest writing of any call. A long reply costs far more time than a long prompt.
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Chat Templates: The Text a Conversation Becomespay once,
yours for life
Read this first. It explains the tools every lab in the AI Engineering course assumes, so the rest of the page makes sense.
Ollama is a free program that downloads open language models and runs them on your own computer. Once it is running, any program can talk to it at the address http://localhost:11434. localhost means this machine, and 11434 is the port Ollama listens on.
The labs use Ollama for three reasons. It is free, so you can run every lab as often as you like. It runs on your own computer, so nothing you type leaves it. And its replies include exact timings and token counts, which is what many lessons measure.
You do not need an account, an API key or a credit card.
Each lesson names the model it used, and its numbers come from that model. The course uses seven. qwen2.5:3b (1.9 GB) is the main one for the How Models Generate chapter and appears in Agents in Production. nomic-embed-text (274 MB) and bge-m3 (1.2 GB) turn text into embeddings, for the Tokens and Embeddings, Retrieval and Evals chapters. qwen3:4b (2.5 GB) and llama3.2:3b (2.0 GB) are used in the Evals and Agents chapters. Two more versions of qwen2.5:3b, q8_0 (3.3 GB) and fp16 (6.2 GB), are used only in the lesson on quantization.
They were chosen because they are small enough for an ordinary laptop, free to download, and good enough to show the effects each lesson teaches.
You do not need all of them at once. Start with qwen2.5:3b and nomic-embed-text, which together are about 2.2 GB, and pull the others when a lesson asks for them.
What we measured: Download sizes as shown by ollama list on the author's laptop, 2026-09-27. Pulling all seven takes about 17.4 GB of disk.
No lab that you are asked to run needs one. Everything on this page is free.
A few lessons report results from paid cloud models, which you can read without an account. The error-analysis lessons in the Evals chapter used Mistral Large and Amazon Nova Pro on Amazon Bedrock to grade answers, and one comparison in the Retrieval chapter used Amazon's titan-embed-text-v2:0, also on Bedrock. Repeating those particular runs needs an AWS account with Bedrock access and is billed per token, and this page does not cover that set-up.
Ollama's documentation lists the systems it supports: macOS Sonoma (version 14) or newer on Apple M-series or Intel Macs, Windows 10 22H2 or newer (Home or Pro), and Linux.
A graphics card helps but is not required. On an Apple M-series Mac, Ollama uses the built-in graphics processor. On Windows and Linux it uses an NVIDIA or AMD graphics card if you have one with a recent driver, and runs on the main processor if not. On the processor alone everything still works, just more slowly.
For memory, qwen2.5:3b took about 2.1 GB when loaded on the author's laptop. Your operating system and browser need room too, so as a rough guide, 8 GB of memory is a practical minimum and 16 GB is comfortable. For disk, plan on at least 5 GB free for Ollama and the first two models.
What we measured: Every number in the lessons was measured on one laptop: an Apple M4 with 24 GB of memory, running Ollama 0.32.14. On your computer the timings will differ; the patterns the lessons teach should not.
Pick your operating system. The steps are taken from Ollama's own documentation at docs.ollama.com. The macOS steps were followed on the author's laptop; the Windows and Linux steps are quoted from the docs because they could not be run on that laptop.
Go to ollama.com/download and download the macOS version. Open the downloaded ollama.dmg file and drag the Ollama app into your Applications folder.
Open Ollama from Applications. The first time, it may ask for permission to create a link in /usr/local/bin; allow it, because that is what lets you type ollama in a terminal. After that, Ollama runs in the background, with a small icon in the menu bar.
To check, open the Terminal app and type ollama --version. It should print a version number, such as 0.32.14.
Go to ollama.com/download and download the Windows installer, OllamaSetup.exe. Run it. It installs for your own account and does not need administrator rights.
After installing, Ollama runs in the background and serves its API at http://localhost:11434.
If you have an NVIDIA graphics card, Ollama's documentation asks for driver version 551.61 or newer. For AMD cards it asks for a recent AMD driver.
To check, open PowerShell or Command Prompt and type ollama --version. In Windows PowerShell 5, the word curl means a different command, so where this page uses curl, type curl.exe instead.
Run Ollama's install script in a terminal: curl -fsSL https://ollama.com/install.sh | sh
The script sets Ollama up as a background service called ollama. If it is not running, start it with sudo systemctl start ollama, or run ollama serve in a terminal and leave that terminal open.
To check, type ollama -v, or sudo systemctl status ollama. For an NVIDIA card, install the CUDA drivers and check them with nvidia-smi.
Open a terminal and type curl http://localhost:11434/api/version. If Ollama is running, it answers with its version, for example {"version":"0.32.14"}. If you see a message such as "Connection refused", Ollama is not running: open the app again, or on Linux run ollama serve.
The check script further down this page does this for you, along with everything else.
Ollama downloads a model once and keeps it on your disk. After that, it loads the model whenever a program asks for it.
In a terminal, type ollama pull followed by the model's name. For the main model of the course: ollama pull qwen2.5:3b. For the first embedding model: ollama pull nomic-embed-text.
Ollama shows a progress bar and ends with the word success. Then ollama list shows every model you have, with its size.
To try a model by hand, type ollama run qwen2.5:3b followed by a question in quotes, for example ollama run qwen2.5:3b 'Say hello in five words.' Without a question, ollama run opens a chat; type /bye to leave it.
ollama ps shows which models are loaded in memory right now, how much memory each is using, whether it runs on the graphics processor, and when it will be unloaded. By default a model stays loaded for 5 minutes after it was last used.
ollama pull qwen2.5:3b
ollama pull nomic-embed-text
ollama list
ollama run qwen2.5:3b 'Say hello in five words.'
ollama ps
Ollama's documentation gives the folders: ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\<your name>\.ollama\models on Windows. On Windows you can move them by setting the OLLAMA_MODELS environment variable and restarting Ollama.
To delete a model you no longer need, type ollama rm followed by its name, for example ollama rm qwen2.5:3b-instruct-fp16. You can pull it again later.
Every lab script is Python. Most use only what comes with Python, so there is nothing else to install. Three lessons need extra packages, listed below.
Python 3.9 or newer. The labs were run with Python 3.13. Check yours by typing python3 --version (on Windows, python --version). If you do not have Python, download it from python.org.
A virtual environment keeps the course's packages apart from the rest of your computer. Create one in the folder where you keep the lab scripts with python3 -m venv .venv, then turn it on with source .venv/bin/activate on macOS and Linux, or .venv\Scripts\activate on Windows. Turn it on again each time you open a new terminal.
Tokens and Embeddings lessons 1 and 2 count tokens with OpenAI's tokenizer: pip install tiktoken.
How Models Generate lesson 4, on attention, reads the attention weights of a small model directly, which Ollama cannot show: pip install torch transformers. The first run then downloads that model, Qwen2.5-0.5B, from Hugging Face, which is about 988 MB, and keeps it for later runs.
Every other lab uses only Python's standard library, plus Ollama.
pip install tiktoken
pip install torch transformersSave the script below as check_setup.py and run it with python3 check_setup.py. It needs nothing but Python.
It checks your Python version, checks that Ollama is running, lists which of the course's models you have and prints the exact ollama pull command for each one you do not, asks qwen2.5:3b a one-line question to prove it answers, and lists the optional packages with the command to install each.
If it prints "Ready for the labs." you can run every lab that uses the models it ticked.
"""Check that your computer is ready for the course labs. Standard Python only; nothing to install.
python check_setup.py
It checks: your Python version, that Ollama is running on this computer, which of the course's
models you have pulled (and the command for each one you have not), that a model can actually
answer, and the two optional Python packages some lessons use.
"""
import importlib.util
import json
import sys
import urllib.request
OLLAMA = "http://localhost:11434"
# The models the labs use, the chapters that need them, and their download sizes (from ollama list).
MODELS = [
("qwen2.5:3b", "How Models Generate; Agents in Production", "1.9 GB"),
("nomic-embed-text", "Tokens and Embeddings; Retrieval (RAG); Evals", "274 MB"),
("bge-m3", "Tokens and Embeddings", "1.2 GB"),
("qwen3:4b", "Evals; Agents in Production", "2.5 GB"),
("llama3.2:3b", "Evals; Agents in Production", "2.0 GB"),
("qwen2.5:3b-instruct-q8_0", "How Models Generate lesson 9 only", "3.3 GB"),
("qwen2.5:3b-instruct-fp16", "How Models Generate lesson 9 only", "6.2 GB"),
]
PACKAGES = [("tiktoken", "Tokens and Embeddings lessons 1 and 2"),
("torch", "How Models Generate lesson 4 (attention)"),
("transformers", "How Models Generate lesson 4 (attention)")]
def get(path, body=None):
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(OLLAMA + path, data=data, headers={"Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req, timeout=120).read())
ok = True
v = sys.version_info
print(f"Python {v.major}.{v.minor}.{v.micro}: {'ok' if v >= (3, 9) else 'too old, install 3.9 or newer'}")
ok &= v >= (3, 9)
try:
version = get("/api/version")["version"]
print(f"Ollama {version}: running at {OLLAMA}")
except Exception as e:
print(f"Ollama: NOT reachable at {OLLAMA} ({e.__class__.__name__}).")
print(" Start the Ollama app (macOS, Windows) or run 'ollama serve' (Linux), then run this again.")
sys.exit(1)
have = {m["name"] for m in get("/api/tags")["models"]}
have |= {n.split(":")[0] for n in have if n.endswith(":latest")}
print("\nModels:")
for name, where, size in MODELS:
if name in have:
print(f" [x] {name:<27} used in {where}")
else:
print(f" [ ] {name:<27} used in {where}")
print(f" get it with: ollama pull {name} ({size})")
if "qwen2.5:3b" in have:
reply = get("/api/generate", {"model": "qwen2.5:3b", "prompt": "The capital of France is", "raw": True,
"stream": False, "options": {"num_predict": 3, "temperature": 0}})
print(f"\nTest answer from qwen2.5:3b: {reply['response'].strip()!r} (it should mention Paris)")
print("\nOptional Python packages:")
for pkg, where in PACKAGES:
found = importlib.util.find_spec(pkg) is not None
print(f" [{'x' if found else ' '}] {pkg:<13} for {where}")
if not found:
print(f" get it with: pip install {pkg}")
print("\nReady for the labs." if ok else "\nFix the items above, then run this again.")
You are free to use any model you like. The labs are written so that changing the model is one line. Here is what to change and what to expect.
Every lab script names its model once, near the top or in the request, for example "model": "qwen2.5:3b". Pull the model you want with ollama pull, then change that one name. Nothing else in the script needs to change.
Browse the models Ollama offers at ollama.com/library. Each model page shows the exact name to pull and the sizes available.
No, and that is expected. The numbers in each lesson, such as timings, probabilities, token counts and scores, come from the named model on the author's laptop. A different model, or a different computer, gives different numbers.
What should hold is the pattern the lesson teaches: for example that reading a prompt is faster per token than writing a reply, that a prompt longer than the window is cut, or that a smaller file writes faster. If your result goes the other way, that is worth a closer look, and worth telling us.
Most labs work with any chat model Ollama can run. A few need a particular kind of model.
Embedding labs, in the Tokens and Embeddings and Retrieval chapters, need an embedding model, such as nomic-embed-text or bge-m3. A chat model is not a replacement.
The chat template lesson shows Qwen's own turn markers, <|im_start|> and <|im_end|>. Other model families use different markers; ollama show <model> --template prints them.
Labs that compare sizes or formats of one model, such as the quantization lesson, need all the versions they compare.
Larger models give better answers but need more memory and write more slowly. On a computer with little memory, a 1B or 3B model is the sensible choice.
These are the problems a first-time setup usually runs into, with the exact messages you will see.
The full message from Python is URLError: <urlopen error [Errno 61] Connection refused> (the number can differ on Windows and Linux). It means nothing is listening at localhost:11434, so Ollama is not running.
Open the Ollama app, or on Linux run ollama serve or sudo systemctl start ollama, and try again. curl http://localhost:11434/api/version tells you when it is back.
Ollama answers {"error":"model 'qwen2.5:3b' not found"} with the HTTP status 404, and Python reports HTTP Error 404: Not Found. The model is not on your computer yet. Pull it with ollama pull and the exact name from the error.
Check the spelling too: qwen2.5:3b and qwen2.5:3b-instruct-q8_0 are different models.
The message is Error: listen tcp 127.0.0.1:11434: bind: address already in use. It means Ollama is already running, usually the app in the background. You do not need a second one; close the terminal and use the one that is running.
Run ollama ps while a lab is running. The PROCESSOR column shows whether the model runs on the graphics processor (GPU) or the main processor (CPU). On the CPU, expect it to be several times slower than the timings in the lessons.
Close other large programs to free memory, or try a smaller model. The first request after a pause is also slower, because Ollama has to load the model again.
Ollama writes a log. On macOS: cat ~/.ollama/logs/server.log. On Linux: journalctl -u ollama --no-pager --follow --pager-end. On Windows: open the folder %LOCALAPPDATA%\Ollama and read server.log.
One lesson uses this log to show a warning that never reaches your program: when a prompt is too long for the window, Ollama cuts it and writes "truncating input prompt" only in its log.
Every answer on this page comes from our AI Engineering course: 210 lessons on RAG, evals, agents, serving, security and MLOps. Many of them are built around a real experiment. You learn why the answer is right, which is what an interviewer checks with the second question. 10 lessons are free to read, with no card needed.
course 2
Take models from notebook to production, with labs on real models.
These questions are a start. The interview asks more.
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
