Bi-Encoder

A bi-encoder encodes queries and documents separately into fixed-size vectors, so you can precompute document embeddings once and search them in milliseconds — the architecture behind every vector database and dense retrieval system.

What It Is

The core idea is independence: one transformer encodes the query, another (or the same with shared weights) encodes the document, and the two never see each other during encoding. Each input becomes a fixed-length vector — typically 384 to 1024 dimensions — and relevance is measured by cosine similarity or dot product between the two vectors. According to Sentence Transformers Docs, this separation is what makes bi-encoders fast: document embeddings can be precomputed, indexed in a vector database like FAISS or Qdrant, and searched in sub-millisecond time over millions of entries.

The architecture traces back to Siamese networks, but the modern version was established by Sentence-BERT (Reimers & Gurevych, 2019). The key insight was that BERT’s cross-attention — powerful for pair classification — was computationally prohibitive for retrieval at scale. By training two BERT towers with a contrastive objective (pulling matching pairs closer, pushing non-matching pairs apart in embedding space), Sentence-BERT achieved semantic similarity search that scaled linearly with corpus size instead of quadratically.

The tradeoff is precision. Because the query and document never interact during encoding, a bi-encoder cannot capture fine-grained token-level dependencies. It sees that two texts are “about the same topic” but can miss that one answers a question the other merely mentions. A query like “does Python support async generators” and a document about “Python’s generator protocol” will land close in embedding space, even though the document never discusses async. This is exactly the gap that cross-encoder rerankers fill in a two-stage retrieve-then-rerank pipeline.

How It’s Used in Practice

Every vector search system runs a bi-encoder under the hood. When you call an embedding API — OpenAI’s text-embedding-3-small, Cohere’s embed-v4, or a local Sentence Transformers model — you are running a bi-encoder. The document side is encoded at ingestion time and stored in the index. At query time, only the query needs encoding, and the nearest-neighbor search over precomputed vectors returns candidates in single-digit milliseconds.

In RAG pipelines, the bi-encoder is the first stage: it retrieves the top 50 to 200 candidates by vector similarity. A cross-encoder reranker then rescores that shortlist with full query–document attention, and only the top 5 to 10 documents are passed to the LLM as context. The bi-encoder handles recall (find everything plausibly relevant), the cross-encoder handles precision (keep only what actually answers the question).

Beyond search, bi-encoders power semantic deduplication (find near-duplicate documents by embedding similarity), clustering (group documents by topic without labels), and recommendation systems (embed users and items into the same space).

Pro Tip: The choice of pooling strategy matters more than model size for most retrieval tasks. Mean pooling over all token embeddings usually outperforms CLS-token pooling, and Matryoshka embeddings let you truncate vectors to smaller dimensions at search time without retraining.

When to Use / When Not

Common Misconception

Myth: A bi-encoder and a cross-encoder are interchangeable — just pick the more accurate one. Reality: They produce fundamentally different outputs. A bi-encoder outputs a reusable vector that can be stored, indexed, and searched without the original text. A cross-encoder outputs a score for one specific query–document pair and stores nothing. You cannot index a cross-encoder, and a bi-encoder cannot match cross-encoder precision on short lists. Production systems use both: bi-encoder for recall, cross-encoder for precision.

One Sentence to Remember

A bi-encoder turns text into a vector once and searches it forever — fast and scalable, but blind to the fine-grained match that only a cross-encoder can see.

FAQ

Q: What is the difference between a bi-encoder and a cross-encoder? A: A bi-encoder encodes query and document independently into vectors that can be precomputed and indexed — fast but coarse. A cross-encoder processes the pair jointly through shared attention — slow but precise. Most retrieval systems use a bi-encoder first, then a cross-encoder to rerank the shortlist.

Q: How do I choose which bi-encoder model to use? A: Match the model to your domain and dimension budget. According to the MTEB Leaderboard, multilingual models like gte-multilingual-base work well across languages, while English-only models like all-MiniLM-L6-v2 are smaller and faster. Check MTEB retrieval benchmarks for your language and task, and test on your own data — benchmark rankings do not always transfer.

Q: Can I fine-tune a bi-encoder on my own data? A: Yes, and it usually helps significantly. According to Sentence Transformers Docs, contrastive fine-tuning with domain-specific query–document pairs is the single highest-impact improvement for retrieval quality. Even a few thousand labeled pairs can close the gap between a general-purpose model and a domain-specific one.

Sources

• Sentence Transformers Docs: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — canonical library and documentation for bi-encoder training and inference.
• arXiv (Reimers & Gurevych, 2019): Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — foundational paper establishing bi-encoder architecture for sentence similarity.
• MTEB Leaderboard: Massive Text Embedding Benchmark — benchmark for comparing bi-encoder model quality across retrieval, clustering, and classification tasks.