Encoded Bias, Opaque Geometry: The Ethical Risks of Embedding Models in High-Stakes Decisions

There is a particular kind of harm that occurs when discrimination moves from something a person does to something a system calculates. It becomes automatic, ambient, distributed across infrastructure so thoroughly that even the engineers maintaining it cannot trace where the bias entered. Embedding models sit at the center of this shift — and almost nobody is examining what that means.

The Geometry Nobody Audits

The question worth asking is not whether AI systems can be biased. That debate is settled. The question is what happens when bias becomes geometric — when prejudice is encoded not as a rule someone wrote but as a proximity relationship in high-dimensional space, a function of Cosine Similarity that no compliance officer will ever inspect.

Word2vec made this visible a decade ago. In 2016, researchers demonstrated that the vector for “computer programmer” sat closer to “man” than to “woman,” while “homemaker” clustered near female-associated terms (Bolukbasi et al.). The finding was treated as a curiosity — a quirk of early models trained on biased corpora. But the deeper implication, that Semantic Search systems built on these vectors would inherit and amplify those associations, received far less attention.

The bias was not a mistake in the data pipeline. Caliskan and colleagues showed that word embeddings replicate the full spectrum of human implicit biases as measured by the Implicit Association Test (Caliskan et al.). The model did not invent prejudice. It absorbed prejudice from the text we collectively produced, then rendered it as mathematics — portable, scalable, and silent.

The Promise of Mathematical Neutrality

The conventional defense rests on a reasonable premise: vectors are numbers, and numbers do not discriminate. A Vector Database stores coordinates, not opinions. The system retrieves documents by proximity, ranks candidates by similarity, surfaces content by relevance — all without explicit rules about race, gender, or class.

This is the steelman, and it is not frivolous. Embeddings do remove certain forms of explicit bias. There is no “if applicant is female, rank lower” instruction. There is no human screener having a bad morning, no interviewer making snap judgments based on accent. The geometry appears clean, the process reproducible, the outcomes auditable in the narrow sense that every query returns the same ranked list.

The outcomes, however, are not clean. A 2025 Brookings study tested three embedding models — E5-Mistral-7b-Instruct, GritLM-7B, and SFR-Embedding-Mistral — in resume screening tasks. For those specific models, white-associated names were preferred in 85.1% of tests; equal selection occurred only 6.3% of the time (Brookings). The math was neutral; the output was not.

Where Training Data Becomes Inherited Prejudice

The hidden assumption inside the neutrality argument is that mathematical operations sanitize their inputs. They do not. Every embedding model is a distillation of its training corpus, and every training corpus is a record of human language — with all its accumulated prejudice and structural inequality.

How do embedding models encode and amplify gender, racial, and cultural biases? Not through malice, and not through negligence in the ordinary sense. The mathematical architecture — Dimensionality Reduction compressing millions of textual relationships into vector coordinates — treats prejudice and meaning as the same kind of signal. The model cannot distinguish between “these words co-occur because of genuine semantic relationship” and “these words co-occur because of a century of systemic discrimination.” Both register as proximity. Garg and colleagues traced this inheritance across a century of English text, showing that embeddings faithfully reproduced the stereotypes of each decade — the occupational segregation, the ethnic hierarchies, the gendered assumptions that defined their era (Garg et al.). The model does not merely learn language. It learns the power structures embedded in language. And when that model sits inside a Dense Retrieval pipeline powering a hiring platform, those historical power structures become operational again.

The Bureaucracy of Coordinates

There is a useful parallel in the history of institutional discrimination. Redlining did not require individual loan officers to be personally racist. The system worked because discrimination was encoded into maps — into the administrative geometry of colored zones that determined who received credit and who did not. Individual decisions appeared neutral, because they followed the map. The map was the problem, but nobody “owned” the map in any accountable way.

Embedding models have become the new maps. When a Matryoshka Embedding powers the similarity function in a recruitment tool, it carries the inherited biases of its training data into every ranking decision. The employer did not choose to discriminate. The vendor did not design discrimination. The model developer did not intend it. And yet the outcomes systematically disadvantage certain groups — not because of any single decision, but because the coordinate system itself encodes historical inequality.

Who is accountable when biased embeddings affect hiring, lending, or content moderation systems? The courts are beginning to test this question. In Mobley v. Workday, a federal court ruled that AI-powered hiring tools can be treated as an “agent” of the employer (FairNow). That ruling gestures toward an answer, but it also reveals the depth of the problem: accountability requires locating a decision, and embedding bias is not located anywhere.

The Distance Between Regulation and Reality

Thesis: The ethical risk of embedding models is not that they are biased — it is that they make bias unlocatable, distributing discrimination across mathematical transformations that no existing accountability structure can trace.

Regulation is arriving, but aimed at the wrong abstraction layer. The EU AI Act, with full high-risk obligations taking effect in August 2026, classifies hiring and credit-scoring AI as high-risk — requiring bias audits and transparency. But the Act regulates deployed systems, not the embedding layers beneath them (EU AI Act). Colorado’s AI Act, effective June 2026, requires “reasonable care” to prevent algorithmic discrimination — but what constitutes reasonable care when the bias lives in a coordinate system that operates below the threshold of institutional visibility?

The technical remedies are similarly incomplete. Research has shown that debiasing methods can reduce surface-level associations while bias re-emerges in downstream tasks. Recent work on anonymization strategies shows promise for mitigating name bias across embedding models, but it addresses one symptom of a deeper condition: we are using mathematical representations of human language to make consequential decisions about human lives, and we have not yet developed the institutional capacity to audit what those representations encode.

Questions That Cannot Be Delegated

This is not a problem that better algorithms alone will solve, though they may help at the margins. It is a problem of category. We built decision-making infrastructure on a foundation — vector similarity — that inherits the prejudices of its training data by design. And we placed that infrastructure inside systems that determine who gets hired, who receives credit, and whose content surfaces in search.

The reflection that matters here is not “how do we fix embedding bias?” but “what kind of governance do we need for systems where the discriminatory mechanism is a distance calculation?” No major embedding provider publishes standalone bias audits for their production models. The testing burden falls on downstream teams who often lack the tools to evaluate fairness across demographic categories. What does informed consent mean when the person being evaluated cannot inspect the geometry that sorted them?

Where This Argument Loses Its Footing

The vulnerability of this position is real, and it should be named. If debiasing techniques mature, if training data becomes more representative, if auditing tools gain the capacity to inspect high-dimensional space for proxy discrimination — the argument that embedding bias is structurally unaddressable weakens considerably. The claim here is not that the problem is permanent. It is that the problem is currently invisible to the institutions tasked with preventing discrimination, and that the gap between technical capability and governance capacity is where harm accumulates.

If someone builds a reliable, scalable method for auditing bias in production embedding systems — one that works across models, languages, and downstream tasks — this essay becomes a historical document. That would be a welcome outcome.

The Question That Remains

We built systems that convert language into geometry, then used that geometry to sort human beings. The sorting reproduces the prejudices of the language it was trained on, at a layer of abstraction our legal and ethical frameworks cannot yet reach. The question is not whether we will close that gap. The question is how much damage it will produce before we do.