Vector Database

A vector database is built to store and search embeddings. An embedding is a list of numbers, often 384, 768, or 1,536 of them, produced by a model that turns text, an image, or audio into a point in high-dimensional space. Items that mean similar things land close together in that space, even when they share no words. The sentence "how do I reset my password" sits near "forgot my login" because the model placed them close, not because the strings overlap.

A normal database answers exact and range queries: find the row where email = x, or price between 10 and 20. A vector database answers a different question: given this query vector, return the k nearest vectors. Nearness is measured with cosine similarity or Euclidean distance. This is called k-nearest-neighbor (k-NN) search, and it is the core operation everything else is built on.

Most vector databases store more than the raw vector. Each record carries the original text or a reference to it, plus metadata like tenant ID, document source, or timestamp. That lets you combine similarity search with normal filters, for example "the 5 most similar paragraphs, but only from documents this user owns, written after January."

Comparing a query to every stored vector is exact but slow. With 100 million vectors of 768 dimensions, a brute-force scan touches tens of billions of floating-point operations per query, which is far too slow for an interactive app. So vector databases use Approximate Nearest Neighbor (ANN) indexes that trade a tiny bit of accuracy for huge speed gains.

The most common index is HNSW (Hierarchical Navigable Small World), a layered graph where each vector links to its near neighbors. A search starts at the top sparse layer, greedily hops toward the query, then drops into denser layers to refine. It typically reaches the right neighborhood in a few hundred comparisons instead of millions, giving sub-10ms queries with 95 to 99 percent recall. Other approaches include IVF (inverted file lists that cluster vectors and only scan nearby clusters) and product quantization, which compresses vectors to shrink memory.

Recall is the metric that matters: of the true top-k neighbors, how many did the index return? You tune index parameters like ef_search in HNSW or nprobe in IVF to slide between faster-but-fuzzier and slower-but-exact. Distance metric, vector dimension, and index type are usually fixed when a collection is created because changing them means rebuilding the whole index.

Reach for a vector database when relevance is fuzzy and language varies: semantic search over docs, product or content recommendations, deduplication, image search by example, and Retrieval-Augmented Generation, where you fetch the most relevant chunks to feed an LLM as context. If your queries are exact lookups, joins, or aggregations, a regular SQL or document database is the right tool.

The big trade-offs are memory and accuracy. HNSW keeps the graph in RAM for speed, so a billion 768-dimension float32 vectors need roughly 3 TB before index overhead, which pushes teams toward quantization or disk-backed indexes. ANN also means results are approximate; you accept the occasional missed neighbor in exchange for speed, and you tune that balance per workload.

You do not always need a dedicated system. Postgres with the pgvector extension handles millions of vectors well and keeps your embeddings next to your relational data, which simplifies filtering and transactions. Dedicated engines like Pinecone, Milvus, Qdrant, or Weaviate earn their place at hundreds of millions of vectors, high query volume, or when you need built-in sharding and replication. A common pattern is hybrid search: blend vector similarity with traditional keyword (BM25) scoring so exact terms like product codes still rank correctly.

Say you build a support assistant over 50,000 help articles. Offline, you split each article into ~500-token chunks, run every chunk through an embedding model, and store each resulting vector plus its text and article ID in the vector database. This is a one-time ingestion job that you re-run when content changes.

At query time a user asks "my card was charged twice." You embed that question into one vector, ask the database for the 5 nearest chunks filtered to the user's language, and get back the most relevant passages in a few milliseconds. You paste those passages into the LLM prompt as context and the model answers grounded in your real docs instead of guessing.

That retrieval step is what keeps the answer accurate and current. Update an article, re-embed its chunks, and the bot's answers change immediately, with no model retraining. The vector database is doing the heavy lifting of finding the right needle in the haystack.