Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
The rules for splitting documents into pieces before they are embedded for search. Chunk size and boundaries decide what retrieval can find.
In short
Chunking strategies are the rules for cutting long documents into smaller pieces, called chunks, before they are embedded and stored for search. The cut points matter because search can only return whole chunks, so a fact split across two chunks may never reach the model.
In short
Chunking strategies are the rules for cutting long documents into smaller pieces, called chunks, before they are embedded and stored for search. The cut points matter because search can only return whole chunks, so a fact split across two chunks may never reach the model.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Chunking Strategies across a run of lessons, not one page. These 4 alone are about 207 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Chunk boundaries set the ceiling on RAG answer quality. How too-large and too-small chunks fail, why overlap helps, and how to size chunks against your own eval.
Should you embed a whole document as one list of numbers, or cut it into small pieces first? Measured on this course's 112 AI lessons with 470 of their own quiz questions and bge-m3 reading up to 8,192 tokens. With one embedding per lesson, the right lesson came first for 286 questions. With 200-word pieces, for 356. Pieces won on short lessons too, not only long ones, and whole lessons still won 31 questions.
Lesson 1's sorting rules with 2,000 words of unrelated text pasted in four places: nowhere, before the rules, between the rules and the message, or after the message. On qwen2.5:3b the scores were 40, 40, 40 and 36 of 40; on llama3.2:3b 36, 38, 35 and 34. No placement differed from the no-document control by more than luck. Every prompt fitted the 8,192-token window. A document that does not fit is the real danger: in the demo, a cut prompt with the document last lost both the rules and the question.
Send an embedding model a long text and it may read only the start, with no error. Measured with Ollama on this laptop: nomic-embed-text and bge-m3 both stopped at 2,048 tokens by default, although bge-m3's own page says 8,192. A sentence hidden past that point changed the score by exactly 0.000. Raising two settings let bge-m3 read 8,192 tokens, and then the hidden sentence was read but barely moved the score: 0.332 against 0.328.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Chunking: The First Lever on Retrieval Qualitypay once,
yours for life
In a RAG system, a language model answers questions from your own documents. RAG stands for retrieval-augmented generation. Before the model answers, a search step finds the few passages that match the question, and the model reads only those.
Your documents are usually too long to search as one piece. So you cut them into smaller pieces called chunks. Each chunk is turned into one embedding, a list of numbers that stands for its meaning. At question time, search compares the question with those embeddings and returns the closest chunks.
A chunking strategy answers three questions. Where do you cut? How big is each piece? Do neighbouring pieces share some text? The answers look like a small setting. They are not small.
Search returns whole chunks. Nothing smaller than a chunk can come back, and nothing split across two chunks comes back in one piece. So the moment you choose where to cut, you fix what can ever be found together.
The lesson opens with a failure that feels impossible. A user asks a RAG assistant for the maximum file size an upload accepts. The upload guide states it in one clear line. The assistant says it does not know. No error was thrown. The sentence had been cut in half: its subject ended one chunk, and the number started the next. Neither half read like an answer, so neither half was returned.
The team tried a bigger model, then a reranker, a second step that reorders the results. Neither helped, because neither touches the cut. A better model reasons better over the text it receives. It cannot reason over text it never received. That makes chunking the first lever you control on retrieval quality once the text exists.
The two failures pull in opposite directions, which is why sizing is hard.
The target is roughly one complete idea per chunk: big enough to answer a question alone, small enough that its embedding points at a single topic.
The seam is the point where one chunk ends and the next begins. The lesson has a small runnable example of a fact falling into it. The text says that the maximum file size an upload accepts is 512 megabytes. The code below is copied from the lesson's playground and shortened to the chunker and the two runs. It counts words instead of tokens to stay readable, and the problem is the same.
Python, copied from the Chunking lesson's playground (shortened)
text = (
"Uploads are handled by the ingest service. It validates the request, "
"then stores the object. The maximum file size an upload accepts is "
"512 megabytes. Larger files are rejected with an error. Contact support "
"if you need a higher limit for your account."
)
words = text.split()
QUERY_PHRASE = "maximum file size an upload accepts is 512 megabytes"
def chunk(words, size, overlap):
"""Slide a window of 'size' words forward, stepping back 'overlap' each time."""
step = size - overlap
chunks = []
start = 0
while start < len(words):
piece = " ".join(words[start:start + size])
chunks.append(piece)
if start + size >= len(words):
break
start += step
return chunks
def query_hits(chunks, phrase):
"""A chunk 'answers' the query only if it contains the whole fact phrase."""
return [i for i, c in enumerate(chunks) if phrase in c]
no_overlap = chunk(words, size=10, overlap=0)
with_overlap = chunk(words, size=10, overlap=5)
print(query_hits(no_overlap, QUERY_PHRASE)) # [] : the fact is split
print(query_hits(with_overlap, QUERY_PHRASE)) # a chunk holds the whole factWith ten-word chunks and no overlap, the cut lands inside the fact, and no chunk holds the whole phrase. With the same size and a five-word overlap, the window steps back, the sentence lands whole in one chunk, and the search finds it. The document did not change. Only the boundary did.
Overlap means each new chunk starts a few tokens before the previous one ended, so it repeats the tail of the chunk before it. A sentence that sat on the old seam now appears whole in at least one chunk. The familiar pairing of 512 tokens with 50 of overlap comes from a common starting point of about ten to fifteen percent overlap.
The lesson counted the damage on real text. Every lesson in the course, 59,660 sentences, was cut at five sizes and three overlap settings. Then each sentence was checked: does any single chunk hold all of it?
At 500 characters with no overlap, 22.6% of sentences were cut. A 20% overlap brought that down to 7.8%, for 1.25 times as many chunks to store and embed. At 300 characters, more than a third of sentences were cut with no overlap, and even 20% overlap left 20.5% cut. Size buys the most here. Overlap cleans up what is left.
One limit matters. Nothing was embedded and no question was run, so these numbers do not say whether search finds the right chunk. They count one failure: the fact cut in half before search began.
Based on a figure in the lesson Chunking: The First Lever on Retrieval Quality.
Overlap softens a bad cut. A better cut point helps more. There are four common ways to choose where the boundaries go, from simple to advanced.
Content type matters too. Prose splits well on paragraphs. Code should keep a whole function or class together. A table row cut away from its header is just numbers, so the header must travel with each group of rows. Transcripts should split on speaker turns, keeping each question with its answer. A mixed collection often needs a different chunker for each type.
There is no single best one, and no universal best size. The right choice depends on how your documents are written, how long one complete idea runs in them, and what your users ask. Short FAQ entries and long legal clauses do not want the same size.
The lesson's honest order is this. Start with fixed-size at 512 tokens and 50 overlap. Use structural or recursive splitting when your documents have real headings. Adopt semantic splitting only when a measurement on your own data shows it wins. Complexity you cannot justify with a measurement is just risk.
Size is also a cost dial. For a corpus of one million tokens with ten percent overlap, returning the top five chunks, 128-token chunks give about 8,700 embeddings and about 640 tokens per prompt. 1024-token chunks give about 1,090 embeddings but about 5,120 prompt tokens. Smaller chunks grow the index. Larger ones grow every prompt.
Two patterns ease the trade. Store metadata on every chunk: the document id, its position and its source. Position lets you fetch the neighbouring chunk, and it powers small-to-big retrieval, where you search small chunks for sharp matches and then hand the model the larger parent section. Anthropic's Contextual Retrieval adds a short generated sentence of context to each chunk before embedding. In their testing that step alone cut retrieval failures by roughly a third.
Whatever you pick, confirm it with RAG evaluation: a set of real questions with known source passages, checked after every change. The Chunking lesson walks through each strategy with diagrams and the full runnable chunker. Two other lessons measure chunking directly: splitting text into pieces and whether there is really a best chunk size.
Anthropic Contextual Retrieval
Prepends a short, chunk-specific context sentence to each chunk before embedding, so a chunk that only says "the limit is 512 megabytes" also says what the limit is for.
LangChain
Ships several splitters, including fixed-size, recursive and structure-aware ones, and a parent-document retriever for small-to-big retrieval.
LlamaIndex
Offers several node parsers for splitting, and auto-merging retrieval, its version of searching small chunks and returning the larger section.
OpenAI text-embedding-3
Accepts at most 8191 tokens of input, a hard upper limit on chunk size when these models do the embedding.
It is the set of rules for splitting documents into the pieces you embed and search: where to cut, how big each piece is, and how much neighbouring pieces overlap. It matters because search returns whole chunks only, so the cut decides which sentences can ever be found together.
No single strategy is best for every collection. Start with fixed-size chunks of 512 tokens and 50 overlap, move to structural or recursive splitting when your documents have clear headings, and use semantic splitting only if a test on your own questions shows it wins.
Fixed-size splitting counts tokens and cuts. Structural splitting cuts on headings and paragraphs. Recursive splitting tries big boundaries first and falls back to smaller ones to stay under a size limit. Semantic splitting embeds each sentence and cuts where the meaning changes.
A common start is about ten to fifteen percent of the chunk size, such as 50 tokens on 512. In the lesson's count over 59,660 sentences, 20% overlap at 500 characters cut split sentences from 22.6% to 7.8%, for 1.25 times as many chunks. Measure on your own data.
Checked against these on .
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
Semantic Search
Search that matches meaning instead of exact words. Text is turned into embeddings, and the results are the documents whose embeddings sit closest to the query's.
RAG Evaluation
Measuring a retrieval-augmented generation system in two parts, whether retrieval found the right text and whether the answer used it, so a failure can be traced to one of them.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
HNSW
Hierarchical Navigable Small World, a graph-based nearest neighbor index that gives high recall at low latency by navigating layers of links, at the cost of more memory than IVF.
This page explains the idea. The "Chunking: The First Lever on Retrieval Quality" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Chunking: The First Lever on Retrieval Quality lessoncourse 2
Take models from notebook to production, with labs on real models.