Term Frequency

Term frequency is the count of how often a word appears in a document — the foundational signal that lexical retrieval models like BM25 and learned sparse systems re-weight to rank relevance.

What It Is

When a search engine ranks documents for a query, it needs a numeric signal of how strongly each document is about the query terms. The simplest workable answer is in plain sight: count how often each query word appears in each document. That count is term frequency. It’s the original quantitative footing of information retrieval, and despite decades of more sophisticated models built on top, it remains the input that nearly every lexical retriever starts from.

In its rawest form, term frequency is an integer count per document, per term. According to Wikipedia, Salton and McGill formalized this as a coordinate of a document vector in early vector-space retrieval, and Karen Spärck Jones added inverse document frequency in 1972 to penalize words appearing in too many documents to be discriminative. Together they formed TF-IDF, which scored relevance as the dot product of query and document vectors built from these counts.

Modern retrievers almost never use raw term frequency directly. According to Stanford IR Book, BM25 normalizes the count through a saturation function, so each additional occurrence of a word adds less score than the previous one — the fiftieth mention adds far less weight than the fifth. Learned sparse models go further. According to arXiv (SPLADE), SPLADE replaces the raw count entirely with a per-token weight predicted by a transformer, and ELSER does the same on the Elastic stack. The concept hasn’t changed; what’s changed is that the score is no longer a function of how many times you wrote a word, but of how much weight the model thinks that word should carry in this document.

How It’s Used in Practice

Most teams encounter term frequency through Elasticsearch or OpenSearch, even when they don’t realize it. The default scoring in these systems is BM25, which takes term frequency as a core input alongside inverse document frequency and document length. When you index a corpus and run a query, the engine computes per-term term frequencies, applies saturation, weights by IDF, and ranks. The same holds on the learned-sparse side: ELSER and SPLADE produce per-token weights that are functionally a learned version of term frequency, scored in the same lexical-retrieval framework.

For a developer integrating retrieval into a RAG pipeline, term frequency rarely appears as a parameter to set directly. The decisions you actually make are upstream and downstream: how to chunk documents (chunk size affects how often a term appears), and how to tune the saturation and length-normalization parameters of BM25. Get chunking right and term frequency mostly takes care of itself.

Pro Tip: Don’t try to game term frequency by stuffing keywords into your documents. BM25 saturation and learned sparse models both heavily discount repeated occurrences past the first handful, and downstream cross-encoder rerankers punish keyword stuffing further. Write naturally and let the retrieval math do its job.

When to Use / When Not

Common Misconception

Myth: More occurrences of a query word in a document mean a proportionally higher relevance score. Reality: Modern retrievers apply saturation — additional occurrences add diminishing score, so the tenth mention barely moves the ranking. Learned sparse models like SPLADE and ELSER replace the raw count with a model-predicted weight, where some tokens carry no weight and others much more than their raw count would suggest.

One Sentence to Remember

Term frequency is the simplest signal in retrieval — count the words — but every modern retriever transforms that count before scoring with it, because language doesn’t reward repetition the way arithmetic does.

FAQ

Q: What is the difference between term frequency and TF-IDF? A: Term frequency is the raw count of a term in a document. TF-IDF multiplies that count by inverse document frequency, which penalizes words that appear in many documents and so carry little discriminative power.

Q: Does BM25 use term frequency directly? A: No. BM25 takes term frequency as input but normalizes it through a saturation function, so additional occurrences add diminishing score, and long documents are penalized for matching by sheer length.

Q: How do learned sparse retrievers like SPLADE and ELSER handle term frequency? A: They replace the raw count with a transformer-predicted weight per token. The model learns which tokens deserve weight in each document, expanding to related terms and discounting filler.

Sources

• Stanford IR Book: Okapi BM25 — Introduction to Information Retrieval - Reference treatment of BM25 and how it normalizes term frequency through saturation and length normalization.
• arXiv (SPLADE): SPLADE: Sparse Lexical and Expansion Model - Original paper introducing the learned sparse retriever that replaces raw term frequency with a per-token weight.