When Nearest Neighbors Are Wrong: Bias Propagation and Accountability Gaps in Similarity Search Systems

We tend to think of Similarity Search Algorithms as neutral infrastructure — plumbing that connects queries to results. But plumbing decides where the water flows. And what flows through these systems, increasingly, are consequential decisions about human beings, routed through a mathematics of proximity that nobody elected and few audit.

The Quiet Sorting Machine

Every time a hiring platform retrieves candidates “most similar” to a successful employee profile, it is making an assertion about human equivalence. Every time a surveillance system matches a face to a watchlist, it is answering — with false confidence — the question of who belongs near whom. These are acts of classification with real consequences, performed by systems that present their outputs as distances rather than decisions.

The troubling part is not that these systems exist. It is that we treat their outputs as findings rather than judgments. A hiring algorithm that ranks candidates by Euclidean Distance or Dot Product similarity appears to be measuring something objective. The appearance of objectivity is the danger, because it discourages exactly the scrutiny these systems require.

The Case for Mathematical Innocence

The strongest defense of similarity search in consequential domains: the algorithm has no opinions. It measures distance in vector space. It does not know what race is, what gender means, or what a “good employee” looks like. If two candidates are equidistant from a reference profile, they receive equal treatment. The math is indifferent — and indifference, the argument goes, is the closest thing to fairness we can engineer.

This defense captures something real. Algorithmic decision-making emerged as a response to documented human bias in hiring and criminal justice. Remove the human from the loop, remove the prejudice. A system that operates on vectors cannot discriminate — or so the reasoning runs.

The reasoning is elegant. It is also wrong, because it confuses the neutrality of a function with the neutrality of the space it operates in.

The Prejudice Encoded in the Geometry Itself

The flaw is not in the distance function. It is in the space where distance is measured. An Embedding does not emerge from first principles — it is learned from data that carries every structural inequality of the society that produced it. When Word2Vec, trained on Google News text, encodes “man is to computer programmer as woman is to homemaker,” it is not inventing a stereotype (Bolukbasi et al.). It is compressing one into geometry. The vector space is the prejudice, made computationally efficient.

This is not a historical curiosity. Research on retrieval-based resume screening found white-associated names preferred in 85.1% of tests, while Black-associated names were preferred in only 8.6% (Brookings). The intersectional disparity was starker — in bias tests, Black men’s resumes were selected at a rate of 0% compared to white men’s. The system was doing exactly what similarity search does: finding nearest neighbors in a space where “nearest” had already been shaped by structural inequality.

The same pattern surfaces in surveillance. Commercial facial analysis systems misclassified darker-skinned females at error rates up to 34.7%, while lighter-skinned males experienced a maximum error of 0.8% (Buolamwini & Gebru). Six documented wrongful arrests from facial recognition in the United States — every one involving a Black individual (Innocence Project). A former Detroit police chief publicly acknowledged a misidentification rate of roughly 96% when the technology was used independently. That is a system trusted despite its own demonstrated failure.

The bias enters before the search begins. Who looks at these numbers and still calls the geometry neutral?

Approximation Adds a Layer We Cannot See

Now consider what happens when we add Locality Sensitive Hashing or Product Quantization to accelerate search. Approximate nearest neighbor methods trade exactness for speed — a reasonable tradeoff when searching for similar product images. When the items being sorted are people, that tradeoff acquires a dimension no benchmark captures.

Approximation means some true neighbors are missed and some false neighbors included. The error is not random — it follows the structure of the index, the quantization boundaries, the hash collisions. If the embedding space already clusters certain demographic groups into denser or sparser regions, approximation may compound the inequality. A qualified candidate near a quantization boundary might vanish from results entirely. No log would record it. The system returns fewer results and calls them complete.

Whether quantization introduces its own bias in vector search specifically remains under-studied. We are operating consequential infrastructure on the assumption that approximation errors are benign — an assumption nobody has proven. What does it mean to optimize for speed when you have not determined whether the errors fall equally on everyone?

Accountability Cannot Be Approximate

Here is the uncomfortable conclusion: similarity search algorithms in consequential domains are governance systems. They determine access, visibility, and opportunity. They sort populations. And they present their outputs as technical measurements rather than institutional decisions.

Thesis: When a system sorts people, the obligation to explain and correct that sorting does not diminish because the mechanism is mathematical — it intensifies, precisely because the mechanism looks objective.

The accountability gap is structural. External audits typically arrive only after a system is already in use, and organizations struggle to identify harms before that point (Raji et al.). The EU AI Act classifies AI in recruitment as high-risk, with core requirements enforceable as of August 2, 2026. But regulation that arrives after the infrastructure has sorted millions of people is remedial at best.

No single actor owns the bias. The team that trained the embedding model is not the team that built the index, which is not the team that integrated it into a hiring platform. Responsibility diffuses across a supply chain, and diffused responsibility is functionally no responsibility at all. Who is accountable — the data, the embedding, the approximation, or the institution that chose not to ask?

The Audit We Owe the Sorted

This essay does not arrive with a solution, because the problem admits of obligations, not fixes. We owe the people being sorted — candidates, suspects, borrowers — commitments the current infrastructure does not provide.

We owe them legibility: the ability to understand why a system placed them where it did. We owe them contestability: the ability to challenge a ranking that governs their access to opportunity. And we owe them a recognition that distance in vector space is not distance in moral space — that proximity is constructed, not discovered.

What would it mean to treat every similarity search system with the same rigor we expect from a courtroom identification? Not because the technology is identical, but because the stakes are.

Where This Argument Breaks Down

The most honest objection: even perfectly accurate, unbiased search would not eliminate inequality, because the data reflects a world that is already unequal. Debiasing embeddings risks flattening real differences in the name of statistical parity.

There is also the pragmatic counterargument that imperfect algorithmic screening may produce less biased outcomes than the human processes it replaces. If the alternative is a biased hiring manager, the calculus is not straightforward.

These objections deserve sustained research and honest debate — neither of which is possible when we pretend the systems are neutral to begin with.

The Question That Remains

Similarity search is becoming the invisible architecture through which institutions sort and grant access to human beings. The mathematics are precise. The consequences are profound. The governance is almost entirely absent.

Who will audit the geometry — and what happens to the people it has already sorted?