Frozen Bias, Invisible Harm: The Ethical Risks of Sentence Embeddings in Automated Decision Systems

Every time a resume passes through an automated screening system, a sentence is converted into a numerical array. That array lands somewhere in a high-dimensional space, and its position — its proximity to other arrays — determines what happens next. The mathematics are elegant. The consequences are not, and the gap between the two is where the ethical reckoning begins.

The Geometry Nobody Audits

When an Embedding model processes the sentence “experienced nurse with ten years of patient care,” the output is a fixed-length vector. That vector carries no explicit label for gender, ethnicity, or age. It appears neutral — a translation from language into geometry. But the Sentence Embedding Association Test, which extends word-level bias benchmarks to full sentences, has demonstrated that these vectors systematically associate occupations with demographic attributes (May et al.). “Nurse” and “secretary” cluster toward female pronouns. “Engineer” and “programmer” cluster toward male. The Universal Sentence Encoder produced measurable gender-occupation association scores ranging from -2.0 to 2.0 on the WEAT scale (Google Developers Blog).

These are not edge cases. They are the structural output of training on web-scale data that reflects centuries of social stratification. The model did not invent the bias. It froze it — and made it computationally efficient to apply at scale.

The Case for Mathematical Innocence

The conventional defense is reasonable, and it deserves to be heard at its strongest. Sentence Transformers and similar frameworks are mathematical tools. They learn statistical regularities from data through Contrastive Learning objectives, encoding sentences via Siamese Network architectures and Mean Pooling operations into dense vectors. The Similarity Search Algorithms and Vector Indexing structures built on top of these embeddings serve retrieval, not judgment.

Proponents argue that bias is a data problem, not a model problem. Fix the data, fix the outcome. And debiasing research offers genuine promise — FairFil, a contrastive framework presented at ICLR 2021, demonstrated that minimizing mutual information between debiased embeddings and sensitive attributes is technically feasible. The argument holds that the mathematics are correctable.

This defense is not wrong. But it rests on an assumption that is.

When Proximity Becomes Prejudice

The hidden assumption is this: geometric proximity is socially neutral until a human makes it otherwise. That framing treats the embedding as a passive artifact, waiting for a decision layer to give it moral weight. But in practice, no separate moment of human intervention exists. When a Multi Vector Retrieval system surfaces candidates for a hiring pipeline, the embedding’s bias is the decision. The distance metric is the judge.

Healthcare embeddings illustrate this starkly. BioBERT and BioGPT — models trained on biomedical corpora — showed statistically significant gender-linked and ethnicity-linked bias at P<0.01 (Gray & Wu). When a clinical decision support system uses biased embeddings to retrieve similar patient cases, the bias does not sit dormant in a vector space. It shapes which precedents the system considers relevant, which treatment patterns it surfaces, which patients it treats as comparable.

The measurement instruments themselves reveal the depth of the problem while simultaneously falling short of it. SEAT and WEAT can only measure bias between two demographic groups at a time — they cannot cleanly capture the intersectional, multi-group reality of how discrimination actually operates. We can measure that something is wrong, but our instruments undercount how wrong it is.

Redlining in Higher Dimensions

There is a historical parallel worth sitting with. In the 1930s, the Home Owners’ Loan Corporation drew maps of American cities, coloring neighborhoods by perceived lending risk. The red lines were not based on individual assessment. They encoded aggregate assumptions — about race, about property values, about who belonged where — into a system that appeared objective because it was spatial. The maps were geometric. The consequences were human.

Sentence embeddings operate through a similar structural logic. They encode aggregate assumptions from training data into geometric relationships, and those relationships are then applied to individuals as if they were neutral measurements. The sophistication is greater, the dimensionality higher, but the mechanism is recognizable: aggregate prejudice, applied as individual judgment. As of March 2026, the sentence-transformers library at v5.3.0 includes no built-in fairness evaluation or audit module. The geometry arrives without a mirror.

The Accountability Vacuum

Thesis: The ethical failure is not that embeddings contain bias — it is that no institutional mechanism exists to hold anyone accountable for what they encode and how that encoding is applied.

NIST’s framework on AI bias identifies three categories — computational, systemic, and human — and warns explicitly against techno-solutionism, the belief that technical fixes alone can resolve socially embedded problems (NIST). Debiasing an embedding addresses the computational layer. It leaves the systemic and human layers untouched. And the systemic layer is where the damage compounds — in hiring systems where a study of roughly 361,000 resumes found intersectional gender-racial biases affecting hiring probability by one to three percentage points across LLM-based screening pipelines (PNAS Nexus). That study examined complete systems rather than isolated embeddings, but the embedding layer is where the initial geometric encoding of bias begins. The system amplifies what the embedding already contains.

Legislation is arriving — but slowly, and with ambiguity. The EU AI Act’s full obligations for high-risk systems take effect August 2, 2026, with potential extension to December 2027 under the Digital Omnibus draft (LegalNodes). Yet the Act does not explicitly classify embedding models themselves as high-risk; the classification depends on the context in which they are used. An embedding in a music recommendation engine and an embedding in a criminal sentencing tool rely on the same mathematics. Only the stakes differ. Who is responsible for ensuring the classification matches the consequence?

Questions We Cannot Afford to Defer

This is not a call for prohibition. Sentence embeddings are among the most useful constructs in modern information retrieval, and debiasing research shows that intervention is possible. The questions that matter are structural: who audits the embeddings before they enter high-stakes pipelines? What standard of fairness applies, and who defines it? When an automated system produces a discriminatory outcome traceable to an embedding’s geometry, who bears liability — the model trainer, the system integrator, or the organization that chose to automate the decision?

The OWASP AI Testing Guide offers technology-agnostic methodology for fairness evaluation, but methodology without mandate is aspiration without consequence. The gap is not technical. It is institutional.

Where This Argument Is Weakest

This position is most vulnerable to two developments. First, if scalable, multi-group bias evaluation methods emerge that genuinely capture intersectional discrimination in embedding spaces, the measurement problem weakens — and with it, the argument that current instruments are insufficient for accountability. Second, if major embedding frameworks integrate fairness auditing as a default rather than an afterthought, the claim that the geometry arrives unexamined loses its empirical basis. Both developments are plausible. Neither has arrived.

The Question That Remains

Sentence embeddings compress language into distances. Those distances carry the biases of the world that generated the training data — frozen, scaled, and applied without deliberation. The mathematics work. The question is whether we will build the institutions to govern what the mathematics encode, or whether we will continue to let geometry make decisions that belong to democratic accountability.