Jaccard Similarity

Jaccard similarity is a measure of how much two sets overlap, calculated by dividing the number of elements they share by the total number of distinct elements across both sets.

What It Is

When you build a training dataset for a language model, the same passage can show up dozens of times across scraped web pages, mirrored articles, and boilerplate footers. Feeding those near-identical copies to a model wastes compute and pushes it to memorize repeated text instead of learning general patterns. To clean the data, you first need to answer a precise question: “how similar are these two documents?” Jaccard similarity is one of the oldest and most reliable answers.

The idea is to stop treating a document as a string of words and start treating it as a set — a bag of distinct pieces. In deduplication, those pieces are usually short overlapping word sequences called shingles or n-grams (for example, every run of three consecutive words). Jaccard similarity then compares two documents with a simple ratio: the number of shingles they share, divided by the number found in either one.

The formula is intersection over union. The intersection is the elements both sets share; the union is every element that appears in at least one of them. If two documents share 80 shingles and have 100 between them in total, their Jaccard similarity is 0.8. A score of 1 means the sets are identical; a score of 0 means they share nothing. Because the result is always a clean fraction between 0 and 1, you can set a threshold — say, treat anything above 0.8 as a near-duplicate — and apply it consistently across an entire corpus.

What makes Jaccard well-suited to this job is that it ignores order and frequency. It does not care where a phrase appears or how many times a word repeats — it asks only what fraction of the distinct pieces is shared. That makes it stable against the small edits — a changed date, an inserted ad, a reordered paragraph — that distinguish “near-duplicate” content from truly original writing.

How It’s Used in Practice

The most common place people encounter Jaccard similarity is inside data deduplication pipelines that clean training corpora before a model ever sees them. Tools that prepare large text datasets convert each document into a set of shingles, then use Jaccard similarity as the definition of “too similar.” Documents that cross the threshold collapse to a single copy, so the model trains on variety instead of redundancy.

There is a catch, though: computing the exact Jaccard similarity between every pair of documents in a billion-document corpus is impossibly slow, because comparing every pair grows quadratically — double the documents and you roughly quadruple the work. This is why Jaccard rarely travels alone in production. It pairs with MinHash, which produces compact fingerprints whose collision rate approximates Jaccard similarity, and Locality-Sensitive Hashing (LSH), which buckets likely matches together so you only compare pairs that stand a real chance of being duplicates. Jaccard is the target the whole system estimates; MinHash LSH is the shortcut that makes estimating it feasible at scale.

Pro Tip: Your shingle size quietly controls everything. Shingles that are too short (single words) make almost every document look similar, because common words are shared everywhere. Shingles that are too long make the metric brittle, since one inserted word breaks a long sequence. Most setups land on shingles of a few words — test a sample of your own data before committing to a threshold.

When to Use / When Not

Common Misconception

Myth: A high Jaccard similarity means two documents mean the same thing.

Reality: Jaccard measures shared exact pieces, not shared meaning. Two passages can express the identical idea in different words and score near zero. For catching reworded or translated duplicates you need a meaning-aware approach such as semantic deduplication, which compares embeddings instead of raw word sets. Jaccard is excellent at finding copies; it is blind to paraphrase.

One Sentence to Remember

Jaccard similarity turns the fuzzy question “are these two documents basically the same?” into a single number between 0 and 1 — which is why it sits at the heart of the deduplication systems that keep training data clean. Just remember it counts shared words, not shared meaning.

FAQ

Q: How is Jaccard similarity calculated?

A: Divide the number of elements two sets share (their intersection) by the total distinct elements across both sets (their union). The result is always between 0 and 1.

Q: What is the difference between Jaccard similarity and cosine similarity?

A: Jaccard compares sets by shared membership, ignoring frequency. Cosine similarity compares vectors by angle and accounts for how often elements appear, suiting it to weighted or embedding-based comparisons.

Q: Why use MinHash instead of computing Jaccard directly?

A: Exact Jaccard requires comparing every pair of documents, far too slow for large corpora. MinHash estimates the same score from compact fingerprints, making deduplication at scale practical.