Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Recording what an LLM application actually did on each request, the prompts, model calls, tool calls, tokens, cost and latency, so you can explain and fix a bad answer.
In short
LLM observability is recording what an LLM application actually did on each request: the exact prompt sent, the model that answered, the tokens, the cost, the time of each step, the tool calls and the final output, all joined by one trace id. You need it because LLM apps usually fail with a healthy 200 OK and a wrong answer.
In short
LLM observability is recording what an LLM application actually did on each request: the exact prompt sent, the model that answered, the tokens, the cost, the time of each step, the tool calls and the final output, all joined by one trace id. You need it because LLM apps usually fail with a healthy 200 OK and a wrong answer.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LLM Observability across a run of lessons, not one page. These 4 alone are about 230 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
An LLM app fails behind a 200 OK: right status, wrong answer. Capture the rendered prompt, token split, and per-step latency on one trace id.
A language model does not write sentences. At every step it gives odds for every possible next token, and something else picks one. Measured with qwen2.5:3b on 24 prompts: for 12 facts the top next token had a median probability of 0.70; for 12 open prompts only 0.17. The same five words with and without the hidden chat template gave ' Paris' at 0.62 or 'The' at 0.999.
Most of an LLM bill is tokens you resend, not tokens users type. Decompose the spend, then cut it with caching, slimming, output caps, and batching.
Forcing a model to reply with one JSON answer field is great for programs. On 70 maths problems with known answers, two small models collapsed from 46 and 70 right to almost none when the only room was the answer box. A steps field before the answer won most of it back, but not all. When only asked nicely for JSON, one model left the calculation unworked in the answer field, and the other ignored the rule and wrote its working anyway.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
An LLM, a large language model, is the kind of model behind a chatbot: text goes in, text comes out. An LLM application wraps that model with other steps. It may search your documents, build a prompt, call the model, call tools, and return an answer.
Observability means being able to explain what a system did from the data it leaves behind. LLM observability applies that idea to these applications. For every request, you record enough to answer one question later: a user got a bad answer at 14:32, what happened?
It is not a product you install. The lesson says this plainly: every field is something you record on purpose. A tracing library gives you the pipes. What goes through them is your decision.
Classic systems fail loudly. You see a 500 error, a timeout or a stack trace. Your dashboards are built to catch those signals.
An LLM fails by succeeding. It returns a well-formed answer, on time, with status 200. The failure is inside the words. A status code describes the transport, the network layer that moved the bytes. Nobody told the transport the answer was wrong.
The lesson opens with an example. A support bot told a customer they could return a mattress after 180 nights. The real policy was 30. The logs showed POST /ask, status 200, 950 ms, no exception. What had happened: a dependency update changed the embedding model used for search. Search began returning zero documents for some questions. The prompt was built with an empty context block. The model filled the gap with a plausible guess.
Every step returned cleanly. The only record that could have shown the empty context was the exact prompt sent on that request, and nobody was saving it.
Work backwards from the incident question and the record falls out. The lesson lists seven fields.
The structure that holds this record already exists. It comes from distributed tracing. A request becomes a trace. Each piece of work becomes a span: a record with a start time, an end time and a parent. The spans form a tree.
LLM apps fit this tree well. The user request is the root span. Retrieval, each model call and each tool call are its children. Smaller steps nest under those.
The tree shape shows problems at a glance. An agent that calls the same tool three times shows three identical sibling spans, so a wasteful loop is easy to see. A retrieval span that returned zero results sits right next to the model span that had to guess without it.
One boundary matters. The model provider does not join your trace. So the span you record around the API call, on your side, is the only record of that step. That is why it must hold the model, the token split and the prompt. A practical habit from the lesson: return the trace id to the caller and show it in error and feedback screens, so support can paste it into the trace store.
With spans, latency becomes a picture called a waterfall. Each span is a bar, indented by depth and placed by start time.
There is a trap in reading it. A parent span's time includes its children. In the lesson's example, the retrieve span reads 180 ms, but 170 ms of that belong to its two children, the embedding call and the vector search. The retrieve step's own work is only 10 ms.
The number that tells you where work happened is self time: a span's total minus the totals of its children. Rank spans by self time, not total time. In this example the model call owns 1195 ms of the 1420 ms request. Tuning retrieval could save 180 ms at best.
Based on a figure in the lesson LLM Observability and Tracing: Seeing Inside a Nondeterministic System.
You could name your fields anything, like myapp.tokens_in. Teams that do this regret it when they change vendors, because every dashboard and alert was written in a private vocabulary.
OpenTelemetry defines standard names for LLM telemetry, called the GenAI semantic conventions. They all start with gen_ai. Examples are gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.tool.name. A model span is named by operation and model, such as chat gpt-4o.
Two cautions from the lesson. First, prompt and output text is not captured by default, because it often contains personal data; turning it on is a choice. Second, the conventions are still marked as in development and have renamed fields between versions. Pin the version you emit, record it, and treat upgrades as migrations.
Classic observability has three pillars: traces, metrics and logs. For LLM apps, traces stay. Metrics stay with new content. Logs change.
Metrics are the summary view you alert on: latency percentiles, tokens, cost per request, error rate, and a refusal rate, the share of questions the model declined. Use percentiles, never the mean. In the lesson's 20-request example, the median latency was 1050 ms, the mean about 1508 ms and the 95th percentile 3570 ms. The two most expensive requests, 10 percent of the batch, carried about 43 percent of the cost.
Raw logs of full model outputs are unreadable at scale. The lesson replaces them with sampled outputs that carry quality scores. This is where observability meets evaluation. Observability records what happened. Evaluation judges whether it was good. A judge scores a sample of stored traces later, not inline, and writes the score back on the trace id. See the LLM evaluation lesson for how judges work.
Full traces are large and can hold private data, so you do not keep all of them. The lesson's policy decides per trace after it ends, which is called tail-based sampling. Keep errors, refusals, timeouts, slow outliers, user-flagged traces and a small random sample in full. Keep only the timing and token skeleton for the rest. Remove personal data in the collector before anything is stored.
To build the full pipeline, including the waterfall and dashboard code and the reference architecture, the next step is the LLM Observability and Tracing lesson.
Langfuse
An open-source, self-hostable trace store built for LLM apps, with prompt bodies, cost accounting and eval scores keyed by trace, and OpenTelemetry ingestion.
Arize Phoenix
An open-source LLM tracing tool built on OpenTelemetry, with strong support for debugging retrieval steps.
OpenLLMetry
A set of instrumentations that emit gen_ai-style spans from the OpenAI and Anthropic SDKs, LangChain, LlamaIndex and many vector stores to any OTLP backend.
Datadog LLM Observability
A managed product that folds LLM spans into the same application performance monitoring the infrastructure team already uses.
For AI and LLM applications, it means recording what the system did on each request so you can explain a bad answer later: the exact prompt, the model that served it, tokens and cost, the time of each step, tool calls and the final output, linked by one trace id.
The lesson names Langfuse and Arize Phoenix as open-source trace stores, OpenLLMetry for instrumentation, and LangSmith and Datadog LLM Observability as managed options. It does not rank them. All of them store a trace tree with gen_ai-style attributes, so following the OpenTelemetry conventions matters more than the vendor.
LLM stands for large language model: a model trained on a large amount of text that takes text in and produces text out. ChatGPT is built on one. LLM observability is about watching the applications built around these models.
Observability records facts on every request: what was sent, what came back, how long it took and what it cost. It makes no judgement. Evaluation judges quality, such as whether an answer was grounded, on a sample. They meet when eval scores are written back against the trace id.
Checked against these on .
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
Span
A single unit of work in a distributed trace, representing one operation (e.g., an HTTP call or database query). Spans are nested to form a trace tree.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "LLM Observability and Tracing: Seeing Inside a Nondeterministic System" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the LLM Observability and Tracing: Seeing Inside a Nondeterministic System lessoncourse 2
Take models from notebook to production, with labs on real models.