Multi Vector Retrieval

Multi-vector retrieval represents documents and queries as multiple token-level vectors rather than a single embedding, then scores relevance through late interaction for more precise results than traditional dense search.

What It Is

Standard dense retrieval compresses an entire document into a single vector — one point in high-dimensional space that tries to capture everything the document means. This works for simple queries, but it has a core limitation: one vector cannot preserve the distinct meaning of every word and phrase in a passage. When a query combines multiple concepts, the single-vector approach blends them into one averaged representation, and nuance gets lost.

Multi-vector retrieval takes a different path. Instead of collapsing a document into one vector, it keeps a separate vector for each token — roughly each word or word-piece produced by the model’s tokenizer. A query gets the same treatment: each query token becomes its own vector. The system then compares query tokens against document tokens individually, looking for the best match for each part of the question.

Think of it like grading an essay with a rubric instead of a gut feeling. Single-vector retrieval gives one overall “relevance score” for the whole document. Multi-vector retrieval checks each rubric criterion separately: “Did this document address concept A? What about concept B?” Then it combines the individual scores into a final ranking.

The dominant scoring method behind this is called MaxSim. For each query token, the system finds the document token with the highest cosine similarity — the best local match. Then it sums those maximum scores across all query tokens to produce the final relevance score. According to Khattab & Zaharia, this approach was introduced in the ColBERT model at SIGIR 2020, demonstrating that preserving per-token representations outperforms single-vector models on standard retrieval benchmarks.

This is called “late interaction” because the query and document encodings don’t interact until the final scoring step. The encoder processes them independently, so all document vectors can be pre-computed and stored. At search time, only the query needs encoding — the comparison happens against pre-stored document vectors.

The trade-off is storage. One vector per token means a single document produces dozens or hundreds of vectors instead of just one. According to Santhanam et al., ColBERTv2 addresses this through residual compression, reducing the storage footprint by six to ten times compared to the original ColBERT while maintaining retrieval quality.

How It’s Used in Practice

The most common place you encounter multi-vector retrieval today is in retrieval-augmented generation (RAG) pipelines — specifically, the retrieval step that decides which documents get fed to a language model before it generates an answer. When a user asks a complex question with multiple facets, multi-vector retrieval finds passages that match across all parts of the query, not just the dominant keyword.

Consider a product manager searching an internal knowledge base with the query “how does our billing system handle refunds for annual subscriptions.” A single-vector search might return documents about billing OR refunds OR annual plans — whichever meaning dominates the averaged embedding. Multi-vector retrieval scores each concept independently: it looks for passages that address billing AND refunds AND annual subscriptions, checking each aspect separately before combining scores.

Search teams in academic paper retrieval, legal document search, and enterprise knowledge bases have adopted multi-vector approaches for the same reason — their queries tend to be long and multi-faceted.

Pro Tip: If your RAG pipeline returns results that feel “close but not quite right” — especially for multi-part questions — the bottleneck is often single-vector retrieval averaging away the details. Swapping in a late interaction model like ColBERT can improve answer quality without changing anything else in your pipeline.

When to Use / When Not

Common Misconception

Myth: Multi-vector retrieval is always slower than single-vector search because it stores more data. Reality: Storage requirements are higher, but query latency depends on implementation. Pre-computation and compression techniques mean search speed can approach single-vector levels. Better retrieval also means the language model receives more relevant context on the first pass, reducing retries.

One Sentence to Remember

Multi-vector retrieval trades storage space for matching precision — instead of asking “does this document roughly match my overall query,” it checks whether the document addresses each part of the question individually, then combines those scores for a final ranking.

FAQ

Q: How does multi-vector retrieval differ from standard dense retrieval? A: Dense retrieval compresses each document into one vector. Multi-vector retrieval keeps a separate vector per token, enabling token-level matching that captures more of the query’s distinct concepts.

Q: Does multi-vector retrieval require training a model from scratch? A: No. Pre-trained multi-vector models like ColBERT encode documents and queries out of the box. Fine-tuning on domain-specific data improves results but is not required to start.

Q: What is the main downside of multi-vector retrieval? A: Storage cost. Each document produces many vectors instead of one. Compression methods reduce this significantly, but the index remains larger than a single-vector approach for the same document set.

Sources

• Khattab & Zaharia: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT - foundational paper introducing multi-vector late interaction for passage retrieval
• Santhanam et al.: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction - improved compression and training for practical multi-vector deployment