Pyserini

Pyserini is a Python toolkit from the University of Waterloo that runs reproducible information retrieval experiments with BM25, learned-sparse models like SPLADE, and dense neural retrievers over the same Lucene-backed index.

What It Is

If you have read a paper claiming a new search algorithm beats BM25 by three points on BEIR, you have probably wondered how to actually try it on your own data. Pyserini exists to close that gap. It is the practical toolkit that turns retrieval research papers into runnable Python — so a product manager evaluating search options, or an engineer building a RAG prototype, can reproduce a published result without reading Java source code or compiling a search engine from scratch.

According to the Pyserini paper, the project provides Python bindings to Anserini, a Java toolkit built on Apache Lucene — the same indexing engine that powers Elasticsearch and Solr. That means a sparse keyword index built with Pyserini is structurally identical to one used in production search engines; you are not running a slower research-only stand-in. On top of those Lucene indexes, Pyserini layers two main entry points. According to the Pyserini Docs, LuceneSearcher handles classic BM25 keyword retrieval, and LuceneImpactSearcher handles learned-sparse models such as SPLADE++ EnsembleDistil and SPLADE-v3, where neural networks predict which terms to expand a query with.

Pyserini also ships prebuilt indexes for the standard benchmarks researchers use, such as MS MARCO and the BEIR suite. You install one Python package, point it at a prebuilt index, and run the same query format that a Castorini researcher used in a SIGIR paper. According to the Pyserini PyPI page, the current release requires Python 3.12 or newer. For dense retrieval, the toolkit integrates with Hugging Face encoders, so embeddings from models like BGE-large-en-v1.5 can be indexed and searched alongside the sparse representations — which is why Pyserini is often the first stop for teams designing a hybrid pipeline that fuses BM25 with a learned-sparse or dense retriever.

How It’s Used in Practice

The most common reason a team picks up Pyserini is to prototype a hybrid search pipeline before committing to a production architecture. A typical workflow looks like this: index your document collection once with LuceneSearcher to get BM25 scores, index it again with LuceneImpactSearcher using a SPLADE-v3 model, then run both retrievers on the same evaluation queries and fuse the result lists with reciprocal rank fusion. Because both indexes live in the Lucene format, query latency stays low enough to evaluate on real benchmarks like BEIR overnight rather than over a weekend.

Researchers use the same toolkit to publish reproducible baselines. Engineering teams use it to answer the question “would adding a learned-sparse retriever actually move our retrieval metrics, or are we fine on BM25?” — before they redesign a production stack around the answer.

Pro Tip: Start with a Pyserini prebuilt MS MARCO or BEIR index for your first experiments. You can validate your evaluation harness against the published numbers from the Pyserini repository before pointing it at your own corpus — that catches scoring bugs you would otherwise blame on the model.

When to Use / When Not

Common Misconception

Myth: Pyserini is a toy research library — for serious search you need Elasticsearch or a vector database. Reality: Pyserini queries run on Apache Lucene, the same engine that powers Elasticsearch and Solr. The reason teams move from Pyserini to those systems for production is operational — distributed indexing, replication, monitoring, and access control — not search quality.

One Sentence to Remember

Pyserini is the shortest path from a retrieval paper to a working search prototype on your own data, and it covers BM25, learned-sparse retrievers like SPLADE-v3, and dense models from a single Python API.

FAQ

Q: Is Pyserini free to use? A: Yes. According to the Pyserini GitHub repository, it is open-source software maintained by the Castorini research group at the University of Waterloo and freely available through the standard Python package index.

Q: What is the difference between Pyserini and Anserini? A: Anserini is the underlying Java toolkit built on Apache Lucene. Pyserini is the Python package that wraps Anserini, exposing the same indexes and retrievers through Python classes such as LuceneSearcher and LuceneImpactSearcher.

Q: Can Pyserini run dense vector search the way FAISS does? A: Yes. Pyserini supports dense retrievers using Hugging Face encoders such as BGE-large-en-v1.5, and can run hybrid pipelines that combine dense embeddings with sparse BM25 or SPLADE results in a single experiment.

Sources

• Pyserini GitHub: castorini/pyserini repository - Source code, documentation, and prebuilt index registry for the Pyserini Python toolkit.
• Pyserini paper: Pyserini: A Python Toolkit for Reproducible IR Research with Sparse and Dense Representations - SIGIR 2021 paper introducing the toolkit and its design goals.