Forward Index

The previous lesson covered inverted indexes, mapping terms to documents. That's how search engines find which documents match your query. But after finding the matching documents, the engine needs to do something else: score them.

To score a document, the search engine needs to know what's inside it. How many total terms does it contain? How many times does the search term appear? What's the document's length?

This is where the forward index comes in. While the inverted index maps terms to documents (for finding matches), the forward index maps documents to their terms (for scoring, highlighting, and field retrieval).

Think of it this way: the inverted index is the phone book that helps you find Alice's number. The forward index is Alice's contact card that tells you everything about her.

A forward index maps each document to the terms it contains, along with metadata:

Doc 1 → {
 terms: ["cat", "sat", "mat"],
 term_frequencies: { "cat": 1, "sat": 1, "mat": 1 },
 total_terms: 6,
 field_lengths: { "title": 3, "body": 3 }
}

Doc 2 → {
 terms: ["dog", "sat", "log"],
 term_frequencies: { "dog": 1, "sat": 1, "log": 1 },
 total_terms: 6,
 field_lengths: { "title": 2, "body": 4 }
}

Where this data gets used:

1. BM25 scoring. The relevance scoring formula needs term frequency per document and document length. The forward index provides both without re-scanning the original text.

2. Highlighting. When Google bolds your search terms in the result snippet, it needs to know where those terms appear in the document. The forward index (or stored fields) provides the original text for snippet extraction.

3. More Like This queries. "Find documents similar to this one." The engine reads the forward index of the source document to extract its significant terms, then queries the inverted index with those terms.

4. Field-level boosting. A match in the title should score higher than a match in the body. The forward index tells the scoring engine which field a term came from.