Reranking in AI: what it is, explained simply

In plain words

When an AI looks things up in your documents (RAG), the first search has to be fast, because it scans thousands or millions of pieces. Fast means rough: the right passage might come in at number 37, behind others that only look similar.

Reranking fixes that with a second pass. Take the top 100 or so results, and have a slower, sharper model read your question and each result together, scoring how well that passage actually answers it. Then reorder, and keep only the best few.

Those few go to the AI to write its answer. Since the AI can only read a limited amount, and does best with relevant text, a good reranker means fewer wrong or made-up answers. The cost is a little extra time on every search.

Everyday example

Shortlisting for a job opening

A company gets 2,000 applications. HR first filters by keywords and experience in minutes, leaving 80. That's fast but crude. Then the hiring manager actually reads those 80 CVs against the job, and picks the best 5 for interviews. The keyword filter is the first search. The manager's careful read is the reranker. Only the 5 reach the final round, the AI.

Where the example breaks: A hiring manager can call a candidate to check. A reranker only judges the text in front of it, and can't rescue a good passage the first search never found.

How it actually works

Most retrieval systems work in two stages: a cheap search that casts a wide net, then a costly model that scores each candidate properly.

Stage one: fast recall

Keyword search, vector search over embeddings or hybrid search pulls a long list of candidates, often 50 to 200. Embeddings are computed ahead of time, separately for the question and each document, which is what makes this quick.

Stage two: read them together

A reranker, usually a cross-encoder, takes the question and one candidate as a single input and outputs a relevance score. Because it sees both at once, it can weigh how the two actually relate, word by word. The Sentence-Transformers docs note cross-encoders score better than separate embeddings.

Why not use it for everything?

Cross-encoders can't be precomputed: every question needs a fresh pass over every candidate. The Sentence-Transformers docs estimate that comparing 10,000 sentences pairwise this way would take about 65 hours, against 5 seconds with embeddings. So it is used only on the shortlist.

Keep the top few

Results are sorted by the new score and the top handful go into the context window. Some systems use a large language model itself as the reranker, which is slower and costlier but flexible.

Breakthrough paper

"Passage Re-ranking with BERT" by Rodrigo Nogueira and Kyunghyun Cho (January 2019) topped the MS MARCO passage benchmark with a 27% relative gain

Measured effect in RAG

In Anthropic's September 2024 tests, adding a reranker lifted the cut in failed retrievals from 49% to 67%

Typical setup

Retrieve about 150 candidates, rerank, keep about 20 (Anthropic's example)

Also called

Re-ranking, reranker, cross-encoder scoring

Questions people ask

What is a reranker in RAG?

A model that sits between the search and the AI. It rescores the retrieved passages against the question so only the most relevant few are passed to the language model.

What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder turns the question and each document into separate number lists (vectors) and compares them: fast, can be stored ahead of time. A cross-encoder reads both together and gives one score: slower, more accurate.

Does reranking slow down search?

A little. It is an extra model call on every query. Anthropic calls the added delay small, and you control it by how many candidates you rerank.

Do I always need a reranker?

No. For small, clean document sets, the first search may be good enough. Test with your own questions: if the right passage is often in the top 50 but not the top 5, a reranker will help.