Document Ingestion: The Layer That Quietly Caps Everything

What was revenue in Q3 for the EMEA region? A finance lead asks a RAG assistant that exact question and gets a confident number that is wrong, or a flat "I cannot find one." The number is right there, in a table on page four of a PDF the system indexed weeks ago. The retriever, the chunker, the embeddings, all the machinery earlier lessons tuned, never had a chance. The number stopped being a number the moment the parser read the page.

It missed because of a step nobody on the team thought about. Before any chunking, before any embedding, before the clever hybrid search from the last lesson, something had to open that PDF and turn it into text. That step read the table on page four and flattened it into a line of loose numbers. The value the finance lead wanted was still technically present, but it was no longer tied to the word EMEA or to the label Q3. Retrieval never had a chance, because the fact it needed had already been broken apart.

This is the layer called document ingestion, and it is the most under-respected part of a RAG system. Enterprise knowledge does not arrive as clean markdown. It lives in PDFs, slide decks, spreadsheets, and scans, and those carry multi-column layouts, tables, headers and footers, figures, and images of text. Practitioners hit this wall over and over. Public parser comparisons and open bug reports keep circling the same failures: multi-column reading order scrambles, and tables and charts are the hardest case of all.

The claim this lesson defends is blunt: your retrieval quality cannot exceed the quality of the text you extracted. Ingestion is the ceiling. Everything the rest of this chapter taught sits below it. By the end you will be able to name the six stages of an ingestion pipeline, explain why the second one caps the rest, choose the right extraction tool for a given document, and put the three guards in place that keep the whole thing honest.