LiveAI Engineering Live: RAG, Agents and Evals in Production
Build RAG and agents that work in production, and prove it with evals
Seven weekends of live classes on what happens after the demo: retrieval that finds the right text, agents that stop and ask a human, evals you can trust, and models served at a cost you can explain.
Schedule
Weekends only. Pick the slot that fits your day.
7 weekends from Saturday, 31 October 2026 to Sunday, 13 December 2026. Each class is two hours, live on Google Meet, and recorded.
Morning slot
Small batchSaturdays and Sundays, 10:00 AM to 12:00 PM IST
Times in other countries
- From Sat 31 Oct: Sat + Sun, 12:30 AM to 2:30 AM; From Sat 7 Nov: Fri + Sat, 11:30 PM to 1:30 AM US Eastern
- Sat + Sun, 4:30 AM to 6:30 AM London
- Sat + Sun, 5:30 AM to 7:30 AM Central Europe
- Sat + Sun, 8:30 AM to 10:30 AM Dubai
- Sat + Sun, 12:30 PM to 2:30 PM Singapore
Evening slot
Small batchSaturdays and Sundays, 8:30 PM to 10:30 PM IST
Times in other countries
- From Sat 31 Oct: Sat 11:00 AM to 1:00 PM, Sun from 10:00 AM; From Sat 7 Nov: Sat + Sun, 10:00 AM to 12:00 PM US Eastern
- Sat + Sun, 3:00 PM to 5:00 PM London
- Sat + Sun, 4:00 PM to 6:00 PM Central Europe
- Sat + Sun, 7:00 PM to 9:00 PM Dubai
- Sat + Sun, 11:00 PM to 1:00 AM Singapore
Neither time works? Tell me what would.
Outcomes
What you will be able to do
- 1Explain what one model call costs in tokens and money, and get structured output you can trust from it.
- 2Build a RAG pipeline and measure its retrieval with recall@k, then improve it with better chunking, hybrid search and a reranker.
- 3Build a LangGraph agent with tools, safe loop limits, a checkpointer and a human approval step.
- 4Write an MCP server and connect its tools to an agent, with least-privilege rules.
- 5Run error analysis on real traces, build an eval set, and know when an LLM judge can and cannot be trusted.
- 6Serve an open model with vLLM, read time to first token and tokens per second, and work out the KV cache memory a request needs.
- 7Add guardrails, tracing and cost tracking before an AI feature goes live.
Curriculum
Week by week
Every weekend has the same rhythm: Saturday is the concept and a guided lab, Sunday is a build on your own machine and a review together. Every class ends with a hands-on lab.
Saturday
One model call, end to end
- Tokens, context windows and what a call really costs
- Messages, roles and structured output with Pydantic
- Calling OpenAI and a local model through LangChain's init_chat_model
Lab: A first app that prints the token count and cost of every call.
Sunday
Build: the tool-calling loop by hand
- How a model asks for a tool and how you return the result
- The same loop with LangChain's create_agent
- Review: what to log from day one
Lab: Write the agent loop yourself in about 40 lines, then replace it with create_agent and compare.
Saturday
Chunking, embeddings and vector search
- Fixed, recursive and structure-aware chunking
- Embedding models and what the vector really holds
- Vector stores: pgvector and Qdrant
Lab: Measure recall@k for three chunking strategies on the same documents.
Sunday
Hybrid search, reranking and context order
- BM25 plus vectors, merged with reciprocal rank fusion
- A cross-encoder reranker on the top results
- Where to put the best text in the prompt, and why the middle gets lost
Lab: Add hybrid search and a reranker, and measure recall again.
Saturday
Measuring retrieval and answers
- A small golden question set, written by hand
- Retrieval metrics (recall@k, MRR) vs answer metrics (faithful, complete)
- Reading failures one by one before choosing a fix
Lab: Build a 30-question test set for your own documents.
Sunday
Build: 2-step RAG vs agentic RAG
- Query rewriting and multi-step retrieval
- When an agent helps retrieval and when it only adds cost
- Review of everyone's numbers
Lab: Run both designs on the same questions and compare quality, latency and cost.
Saturday
Graphs, state and control
- StateGraph, state and reducers, nodes and edges
- Conditional routing and loops with a recursion limit
- Command and Send for hand-offs and map-reduce
Lab: A support agent that routes, looks up an order and drafts a reply.
Sunday
Build: durable agents
- Checkpointers on SQLite and Postgres
- interrupt for human approval before a risky action
- Short-term and long-term memory, scoped per user
Lab: Crash the agent mid-task and resume it from the checkpoint.
Saturday
The Model Context Protocol
- MCP servers, tools and resources
- Writing a server with FastMCP from the official Python SDK
- Tool output is untrusted input: prompt injection and least privilege
Lab: Write an MCP server for a small internal API and connect it to your agent.
Sunday
Build: when to use more than one agent
- Subagents, hand-offs and a router
- When a single agent or a fixed workflow is the better design
- Review: cost per task for each design
Lab: Split one agent into a router and two specialists, and measure whether it got better.
Saturday
Error analysis and LLM-as-judge
- Read traces first, then name the failure categories
- Code checks vs LLM judges
- Judge pitfalls: position bias, length bias, a judge that grades its own model kindly
- From our lab: two model graders read the same 40 traces and sorted the failures into different categories
Lab: Grade 40 traces with a judge, then check how often it agrees with your own labels.
Sunday
Build: an eval gate
- Datasets and evaluators in LangSmith
- How many examples a failure category needs before its rate means anything
- Running evals in CI so a prompt change cannot ship a regression
Lab: Add an eval step to a pipeline that fails the build when quality drops.
Saturday
Serving models with vLLM
- PagedAttention, continuous batching and prefix caching
- KV cache memory: 2 x layers x KV heads x head size x bytes, per token
- Time to first token, time per output token and throughput
Lab: Serve an open model with vLLM's OpenAI-compatible server on a rented cloud GPU and load-test it (or follow my recorded run at no cost).
Sunday
Going live safely
- Guardrails: input and output checks, PII masking, tool allow-lists
- Tracing with OpenTelemetry and LangSmith or Langfuse
- Cost dashboards and alerts
- Final project demos and review
Lab: Add guardrails and tracing to your final project and show its cost per request.
Labs and projects
What you build
All labs are our own and run on your laptop. The code is in a labs repo you get before the first class.
RAG you can measure
A document assistant with hybrid search and a reranker, and a test set that shows recall before and after each change.
LangChain · pgvector or Qdrant · Python
A durable support agent
A LangGraph agent that looks things up, asks a human before refunds, and survives a crash.
LangGraph · Postgres checkpointer · OpenAI or Ollama
Your own MCP server
An MCP server for a small API, connected to the agent with clear limits on what it may do.
MCP Python SDK · FastMCP · LangChain
An eval gate in CI
A dataset, code checks and an LLM judge whose agreement with you is measured, run on every change.
LangSmith · pytest · GitHub Actions
Who it is for
Is this for you?
This is for
- Software engineers moving into AI engineering.
- ML engineers and data scientists who now ship LLM features.
- Engineers whose RAG or agent demo works on the laptop but not for real users.
- Tech leads who need to judge AI work by numbers, not by demos.
You need
- Python you write without looking up the basics: functions, classes, virtual environments and pip.
- Comfort with a terminal and Git.
- An OpenAI API key helps (labs cost a few dollars in total); a free local model with Ollama works for most labs.
- The vLLM lab needs a GPU for about an hour. Renting one is optional: my recorded run is shared with everyone.
Not for
- Someone new to programming.
- Someone looking for prompt tips only, with no code.
Careers
Roles you can target after this
AI Engineer
Build and run LLM features end to end: retrieval, agents, evals and serving.
GenAI / RAG Engineer
Own the retrieval layer and prove its quality with numbers.
Applied LLM Engineer
Turn a model into a reliable product feature with guardrails and monitoring.
ML Platform Engineer
Run model serving, cost and observability for many teams.
Companies that hire for these skills include:
Check each company's careers page for current openings. We have no hiring arrangement with any of them. This programme does not promise a job, a placement or a salary.
Your instructor
Roni Das
18 years of building and running enterprise systems. He writes and teaches the courses on systemdesign.academy: 770 system design lessons and 212 AI engineering lessons, with labs he runs himself.
What is included
Everything you get
14 live classes, 28 hours on Google Meet, with time for your questions in every class.
Every recording, with lifetime access on systemdesign.academy.
The labs repo: every lab and project, runnable on your laptop.
A certificate of completion from systemdesign.academy.
A doubt-clearing group for your batch, between classes.
Included once your fee is fully paid: lifetime access to the self-paced AI Engineering course on systemdesign.academy (212 lessons).
How paying works
- Full fee: ₹15,000 / $300 in one payment.
- Or reserve your seat for ₹2,000 / $40 now, and pay the remaining ₹13,000 / $260 before the batch starts. The deposit is credited in full.
- Every payment follows our refund and cancellation policy.
Questions
Frequently asked questions
Which versions of LangChain and LangGraph do we use?
The current 1.x releases of langchain and langgraph, pinned in the labs repo so every lab runs the same way on your machine.
How much will the API calls cost me?
Most labs cost cents. The whole programme is usually a few dollars of OpenAI usage. Every lab also runs on a free local model with Ollama, and every lab prints its own cost.
Do I need a GPU?
Only for the vLLM lab, and only if you want to run it yourself. You can rent a cloud GPU for about an hour, or follow my recorded run at no cost.
When does the batch start, and when are the classes?
The batch starts on Saturday, 31 October 2026. Classes are on Saturdays and Sundays for 7 weekends, two hours each, in the morning slot (10:00 AM to 12:00 PM IST) or the evening slot (8:30 PM to 10:30 PM IST). You choose one slot when you enrol. The last class is on Sunday, 13 December 2026.
What if I miss a class?
Every class is recorded. The recording appears in My live classes on systemdesign.academy after the class, and you keep it for life. Bring your questions to the next class or to the batch group.
How does the seat reservation work?
Pay the reservation deposit to hold your seat in the slot you choose. The deposit is credited in full against the fee, so you pay only the rest. Pay the remaining amount before the batch starts; your reservation email has a link to do it.
Can I get a refund?
Every payment follows our refund and cancellation policy. Please read it before you pay: systemdesign.academy/refund-policy.
How many people are in a batch?
It is a small batch: at most 30 people in each slot, so there is time for your questions.
How do I pay?
In India: UPI, cards or netbanking through Razorpay, in rupees. Everywhere else: PayPal, or a Visa or Mastercard card, in US dollars.
Is a self-paced course included?
Included once your fee is fully paid: lifetime access to the self-paced AI Engineering course on systemdesign.academy (212 lessons).
Do I get a certificate?
Yes. You get a certificate of completion from systemdesign.academy at the end of the programme.
Where do the classes happen?
On Google Meet. Your joining link and the class calendar reach your email before the first class.
Still unsure? Ask on our WhatsApp group, or book a short call on topmate.io/roni.
Need a different time?
Tell me which days and times work for you
If enough people ask for a similar time, I will open a new slot and email you. This does not hold a seat; a seat is held only by enrolling or reserving.