Reranking

Reranking is a second-stage retrieval step that takes a shortlist of already-retrieved candidates — typically the top 20 to 100 from a first-pass search — and rescores them with a slower, more accurate model, reordering the list before it's handed to the generation step.

First-pass retrieval (vector search, keyword search, or hybrid search) is built for speed: it has to rank across an entire corpus, often millions of chunks, in milliseconds. That speed comes from a shortcut — comparing a query embedding to pre-computed document embeddings independently — which is fast but throws away information about how the query and document interact with each other. Reranking spends more compute on far fewer candidates to recover that lost precision.

How it works

The dominant reranking architecture is a cross-encoder: unlike an embedding model, which encodes the query and each document separately and compares the results afterward, a cross-encoder feeds the query and one candidate document into the model together, letting it attend across both simultaneously and output a single relevance score. That joint attention is what makes cross-encoders more accurate than embedding similarity — and also why they can't run over an entire corpus: scoring every document this way for every query is too slow at scale. The standard pipeline is therefore first-pass retrieval to narrow millions of candidates down to dozens, then a cross-encoder reranker to get the ordering right on that much smaller set.

Why it matters

Retrieval quality is the ceiling on RAG quality, and rerankers reliably raise that ceiling further than tuning the first-pass retriever alone can. First-pass search optimizes for recall — don't miss the relevant document among millions — while reranking optimizes for precision at the very top of the list, which is exactly the part of the ranking a language model actually reads. A relevant document sitting at position 15 out of a 20-candidate shortlist might as well not exist if only the top 3 make it into the prompt; reranking is what gets it there.

A worked example

A query retrieves 20 candidates by vector similarity, and the one truly relevant document lands at rank 4 — reciprocal rank 1/4 = 0.25 for that query. A cross-encoder reranker rescores all 20 candidates jointly against the query and moves the relevant document to rank 1, lifting its reciprocal rank to 1/1 = 1.0 and its precision@1 from 0 to 1 for this query. Across a batch of queries where this pattern repeats, that shift in individual reciprocal ranks is exactly what drives MRR upward when a team adds reranking to a retrieval pipeline.

How miatz teaches it

The reranking lab has learners retrieve a fixed candidate list with plain vector search, record MRR and precision@1 by hand, then run the identical candidates through a cross-encoder reranker and recompute both metrics — so the value of the second stage is a number they watched move, not a claim about cross-encoders they took on faith.