Embedding Storage

Your AI application generates embeddings using OpenAI's text-embedding-3-large model. Each embedding has 3,072 dimensions. Each dimension is a 32-bit float (4 bytes). That's 12,288 bytes per vector, about 12 KB.

You have 100 million documents. The raw embedding data alone is 1.2 terabytes. Add the index structures (HNSW graphs, IVF centroids) and you're looking at 2-3 TB of memory needed to keep everything searchable at low latency.

RAM costs roughly $5-10 per GB per month on cloud providers. Keeping 3 TB in memory runs $15,000-30,000 per month, just for embeddings.

This is the embedding storage problem. The vectors themselves are large. The indexes needed for fast search are even larger. And the most common access pattern. ANN search, requires the index to be in memory (or at least on fast SSD) for acceptable latency.

Understanding how embeddings are stored, compressed, and served is critical for anyone building AI applications at scale. The difference between a naive approach and an optimized one can be 10-50x in cost and latency.

How vectors are physically stored:

Flat arrays. The simplest format. Each vector is stored as a contiguous array of floats. For 1,536 dimensions at 32-bit precision, that's 6,144 bytes per vector. Fast to read, easy to work with, but takes the most space.

Memory-mapped files. Store vectors in files on disk and use the OS's virtual memory system to map them into the process's address space. The OS handles paging, keeping hot data in RAM and evicting cold data to disk. This lets you work with datasets larger than available RAM, at the cost of potential page faults and higher latency.

Columnar storage. Store each dimension as a separate column (like Parquet). Useful for analytics on embeddings (e.g., computing the mean value of dimension 42 across all vectors), but terrible for the primary use case of loading a complete vector for distance computation.

In practice, vector databases use flat arrays with memory-mapped files as a fallback. Pinecone, Qdrant, and Weaviate all keep the hot index in memory while overflow goes to disk.

Metadata alongside vectors:

Every embedding has associated metadata, the original text, document ID, timestamps, tags. This metadata needs to be stored and filterable alongside the vectors. Vector databases typically store metadata in a separate structure (often an inverted index or B-tree) and combine metadata filters with vector search at query time.

{
 "id": "doc_42",
 "vector": [0.12, -0.34, 0.56, ...],
 "metadata": {
 "source": "engineering-wiki",
 "department": "platform",
 "updated_at": "2025-03-23",
 "category": "authentication"
 }
}

A query might be: "Find the 10 vectors most similar to my query, WHERE department = 'platform' AND updated_at > '2025-01-01'." The metadata filter narrows candidates before or after the vector search.