Vector Indexes for RAG: HNSW, IVF, Quantization, and Scale

An embedding is an array of numbers. An index is the data structure and operating model that make searching millions or billions of those arrays practical. Treating the two as interchangeable leads to poor capacity plans and unexplained recall loss.

Establish an exact-search reference

Exact nearest-neighbor search compares the query with every eligible vector. It is expensive at scale, but it is the ground truth for evaluating an approximate nearest-neighbor (ANN) index.

Build a representative sample and record the true top K with exact search. ANN recall is then:

ANN Recall@K = |exact_top_K intersect approximate_top_K| / K

This measures index approximation, not business relevance. You need both ANN recall against exact neighbors and retrieval recall against human-labeled evidence.

Choose the similarity function that matches model training: cosine similarity, dot product, or Euclidean distance. If cosine similarity is implemented with dot product, normalize document and query vectors consistently.

HNSW: strong recall with memory cost

Hierarchical Navigable Small World graphs connect each vector to nearby vectors across layers. They provide excellent recall and low query latency for many medium-to-large corpora.

Important controls are:

• M: graph connections per node; higher values improve recall and memory cost
• efConstruction: candidate effort while building; higher values improve graph quality and slow writes
• efSearch: candidate effort at query time; higher values improve recall and increase latency

Tune efSearch online or by request class. An exact identifier query handled well by BM25 may need less vector effort; a difficult semantic query may justify more.

Memory is not just vectors × dimensions × bytes. Budget for the vector payload, graph edges, identifiers, metadata, allocator overhead, replicas, and temporary migration capacity. Measure actual resident memory with production-shaped metadata.

HNSW trade-offs:

• fast queries and strong recall
• relatively expensive memory
• graph construction cost during bulk loads
• updates and deletes can fragment performance depending on implementation
• restrictive filters can damage recall if filtering occurs after graph traversal

Prefer engines that integrate filtering during traversal, or maintain partitions that align with stable high-selectivity attributes.

IVF: partition first, search selected lists

Inverted File (IVF) indexes cluster vectors into nlist partitions. At query time, the system searches the closest nprobe partitions.

• more nlist values create finer partitions but require enough training data
• higher nprobe improves recall and increases work
• a poorly trained index performs badly even when queries are fast

IVF is useful for large, relatively stable collections and combines naturally with compression. Train centroids on representative vectors across tenants, languages, and document types—not the first batch available.

Product quantization and disk-based indexes

Product Quantization (PQ) compresses vectors into short codes. It can reduce memory dramatically at the cost of distance accuracy. A common pattern retrieves candidates with compressed vectors and reranks them using full-precision vectors.

Scalar quantization, such as float16 or int8, is simpler and may preserve enough recall. Benchmark it before adopting more complex PQ configurations.

Disk-oriented graph indexes such as DiskANN-style approaches keep much of the index on SSD and use memory for compressed navigation structures and caches. They trade storage access and operational complexity for lower RAM requirements at very large scale.

The decision table is workload-specific:

Index	Strength	Main cost	Good fit
Flat/exact	Perfect neighbor recall	Linear compute	Small corpora and benchmark truth
HNSW	High recall, low latency	RAM and build cost	Interactive search at millions of vectors
IVF-Flat	Tunable scan fraction	Training and parameter tuning	Large, stable corpora
IVF-PQ	High compression	Approximation loss	Very large memory-constrained corpora
Disk graph	Scale beyond RAM	SSD latency and complexity	Hundreds of millions to billions of vectors

Filters change the benchmark

Benchmark with the same tenant and ACL filters used in production. Post-filtering 100 global neighbors may leave zero authorized results. Pre-filtering can create tiny candidate pools or disconnected graph paths.

Possible strategies include:

• dedicated indexes for large, isolated tenants
• shared indexes with filter-aware ANN traversal
• partition keys for region, tenant class, or security domain
• over-fetching followed by filtering only when leakage into the retrieval service is acceptable
• lexical fallback when a highly selective filter makes ANN ineffective

Never use post-filtering outside the trusted retrieval boundary for access control.

Multi-tenancy and sharding

Choose between:

• shared index: efficient for many small tenants, but requires rigorous filters and noisy-neighbor controls
• index per tenant: strong isolation and tunability, but operationally expensive for thousands of small tenants
• tiered model: dedicated indexes for large or regulated tenants and shared shards for the long tail

Shard on an attribute that supports routing without destroying recall. Hash sharding spreads load but requires fan-out to every shard unless a routing key is known. Domain or tenant sharding reduces fan-out but can create imbalance.

Replicas provide availability and read capacity. They do not replace backups or the ability to rebuild from the source corpus. Plan for one replica being unavailable during peak load.

Estimate capacity from measurements

Begin with:

vector bytes = number_of_chunks × dimensions × bytes_per_dimension

For 50 million 1,024-dimensional float32 vectors, the raw vectors alone are about 205 GB before index, metadata, replicas, and headroom. Float16 halves that raw figure; quantization can reduce it further.

Then measure actual index size and memory with a one-to-five-percent production sample. Include:

• growth in chunks per day and retention period
• lexical index and stored source text
• metadata cardinality and filter structures
• replicas and failure headroom
• background compaction and rebuild space
• dual indexes during embedding migrations
• query concurrency and tail-latency targets

Capacity test at expected peak QPS plus safety margin while ingestion and compaction are active. Average latency on an idle index is not a production result.

Tune with a Pareto curve

Sweep index parameters and plot recall against p50, p95, and p99 latency, CPU, RAM, and cost. Select a point on the Pareto frontier instead of declaring one parameter set “best.”

For every configuration, keep constant:

• corpus and embedding version
• query set and filters
• concurrency and hardware
• warm-up procedure and cache state
• K and candidate count

Report cold and warm behavior separately. Include difficult filters, recently inserted documents, deletions, and skewed tenants.

Version and migrate indexes

An index name should encode corpus, embedding, and index configuration versions. A safe migration is:

1. Build the new index from an immutable corpus snapshot.
2. Apply changes that arrived after the snapshot.
3. Validate counts, ACL distributions, and sampled nearest neighbors.
4. Run offline evaluation and shadow production queries.
5. Shift a small percentage of traffic through an alias.
6. Expand while monitoring quality, errors, and latency.
7. Retain the old index for a declared rollback period.

The index is an artifact, not the source of truth. If it cannot be rebuilt deterministically, it cannot be operated safely.