Embedding

What an embedding is

Before a neural network can do anything with text, it needs numbers. Computers don’t understand the word “cat.” They understand arrays of floating-point values. An embedding solves this translation problem: it converts discrete items, like words or tokens, into dense numerical vectors positioned in a continuous mathematical space.

Think of it like assigning GPS coordinates to concepts. Just as “Paris” and “Lyon” sit closer together on a map than “Paris” and “Tokyo,” the embedding for “king” sits closer to “queen” than to “refrigerator” in vector space. The distance between vectors carries meaning. Similarity in position reflects similarity in usage and semantics.

The process works through learned parameters. During training, a neural network adjusts millions of weights so that each token’s vector captures patterns from the training data. Early approaches like Word2Vec and GloVe learned one fixed vector per word. Modern transformer-based models go further: they produce contextual embeddings, where the same word gets a different vector depending on its surrounding sentence. The word “bank” in “river bank” and “bank account” receives two distinct representations.

For transformers specifically, the embedding layer is the entry point of the entire architecture. Raw tokens, which are integers representing subword units, pass through an embedding matrix that maps each token to a high-dimensional vector, typically with hundreds or thousands of dimensions. These vectors then flow into the self-attention mechanism, where the model compares every token with every other token. This is where the quadratic scaling cost originates: attention computation grows with the square of the sequence length, and it all starts with the embedding vectors that feed into it.

Embeddings also appear in multimodal systems. Images, audio clips, and code snippets can all be projected into the same vector space, enabling cross-modal search and comparison. The underlying principle stays the same: represent structured data as points in a space where geometry encodes meaning.

How embeddings are used in practice

The most common place you encounter embeddings is in search and retrieval. When you type a question into an AI-powered search tool, your query gets converted into an embedding vector. The system then compares it against pre-computed embeddings of documents, FAQs, or code snippets, returning the closest matches based on how near the vectors sit to each other in that space.

Recommendation systems rely on the same mechanism. Streaming services, e-commerce platforms, and content feeds embed both users and items into a shared space, then recommend whatever sits nearest to the user’s vector. AI assistants use embeddings internally at every inference step: each token in your prompt gets embedded before the model’s attention layers process it.

When evaluating embedding quality for a retrieval task, test with adversarial queries. Search for questions that share vocabulary but differ in meaning, like “How to train a dog” versus “How to train a model.” If results mix up the two, the embedding space isn’t capturing semantic distinctions well enough for your use case.

The embedding reading path: geometry first, market last

Start with how neural networks encode meaning into vectors — it builds the geometric intuition every later decision leans on, from why similar meanings land close together to what a coordinate in semantic space actually is. Then read the prerequisites and technical limits of vector representations in the same sitting: it explains what each similarity ruler measures and where the geometry stops working, which is exactly the knowledge that prevents metric-by-gut-feel mistakes.

When you are ready to build, the semantic search pipeline guide walks the four surfaces every pipeline must specify — ingest, embed, store, query — and gives the model-selection framework that benchmark tables cannot. For the market context behind that choice, the open-source surge reshaping the embedding market tracks how open-weight models reached benchmark parity and what the price collapse means for lock-in. Close with the ethical risks of embedding models in high-stakes decisions — if your vectors will ever rank people rather than paragraphs, read it before you ship, not after.

How embeddings differ from tokens, indexes, and frameworks

Three neighbours get mistaken for this topic, and each confusion sends debugging in the wrong direction.

• Embeddings are not token IDs. A tokenizer assigns arbitrary integers; nothing about token 4021 is closer to token 4022 in meaning. An embedding is a learned coordinate — proximity carries meaning, which is what makes similarity search possible at all.
• The embedding is not the index. The vector is data; vector indexing is the search structure built over that data. If results are wrong, suspect the model; if results are slow or recall drops at scale, suspect the index. Teams that swap one to fix the other burn weeks.
• The model is not the framework. Sentence Transformers is a framework for producing and fine-tuning sentence-level embeddings; the embedding itself is the representation any such tool outputs. Choosing a framework does not answer the model question — the guide’s selection criteria still apply.

When embeddings fit — and when they don't

Common questions about embeddings

Q: Are open-weight embedding models good enough to replace proprietary APIs? A: On major benchmarks, yes — open-weight models from NVIDIA, Alibaba, and BAAI now match or exceed proprietary APIs at a fraction of the cost. Proprietary services still lead some of the newest English benchmarks, so the market split is about cost and lock-in, not a single winner.

Q: Should I just pick the embedding model with the highest MTEB score? A: No — benchmark scores alone do not predict production performance, because the benchmark measures a different corpus than yours. Test candidate models on your actual documents before committing; the pipeline guide turns that into a step-by-step selection framework.

Q: Can I switch embedding models after my documents are already indexed? A: Not without re-embedding everything. Vectors from different models live in different geometric spaces — often with different dimensions — so index-side and query-side must always use the same model. The limits of vector representations explains why the spaces are incompatible.

Q: Does embedding bias matter if I only rank documents, not people? A: The stakes scale with what the ranking decides. For internal document search the encoded bias mostly skews relevance; the moment the same vectors rank résumés, applicants, or claims, the bias becomes a decision about people with no single line of code to hold accountable.

Part of the retrieval-augmented generation theme · closest neighbour: vector indexing. Coming to vectors from classical software? Start with the story: Debugging RAG Failures: Why Developers Need a New Diagnostic Model.