Reranking is a second retrieval stage that reorders an initial set of candidate passages by true relevance to the query before they go to the model…
See why fast first-stage retrieval leaves the best passage buried.
First-stage retrieval — vector or keyword search — has to be fast enough to scan a whole corpus, so it uses a cheap comparison: for vectors, it embeds the query and each document separately and compares them. That speed comes at a cost in precision, so the single most relevant passage often lands at rank 5 or 15 rather than rank 1.
That matters because a RAG model only sees the few passages you pass it. If the best one is buried below your cutoff, or outranked by an on-topic-but-wrong passage, the answer suffers. Reranking exists to fix the order before the model reads it.
Understand the cross-encoder that scores query and passage together.
The retriever is a bi-encoder: it turns the query and each document into vectors independently, so comparisons are just fast dot products — but the model never sees the query and document at the same time. A reranker is a cross-encoder: it feeds the query and one passage into the model together and outputs a single relevance score.
Because the cross-encoder can attend to how the query and passage interact — which words answer which — it judges relevance far more accurately. The price is speed: it must run the model once per candidate, so it cannot scan the whole corpus, only rerank a shortlist.
Combine the two stages into the standard high-precision pattern.
The standard pattern plays to each stage's strength. First-stage retrieval pulls a wide candidate set — say the top 50 to 100 passages — cheaply, prioritizing recall so the right passage is somewhere in the set. The reranker then scores those candidates precisely and keeps only the top few (often 3 to 5) to send to the model.
Wide-then-narrow means you rarely miss the right passage in stage one and rarely bury it in stage two.
candidates = vector_search(query, top_k=50) # stage 1: fast, wide scored = [(reranker(query, p), p) for p in candidates] # stage 2: precise top = [p for _, p in sorted(scored, reverse=True)[:5]] # keep best 5 answer = llm(build_prompt(query, top))
Stage one retrieves 50 candidates for recall; the cross-encoder reranker scores each against the query; the best 5 go into the prompt. The model now reads a small, high-relevance context instead of a long, noisy one.
Weigh reranking's cost and avoid the ways it goes wrong.
Reranking adds a model call per candidate, so it costs latency and compute. It is usually worth it: passing fewer, better passages improves answers and can even cut generation cost by shrinking the context. Managed reranker APIs and small local cross-encoders both work; pick based on your latency budget and candidate count.
The usual errors: retrieving too few candidates in stage one (the reranker can only reorder what it's given — if recall is poor, reranking can't save it); reranking too many candidates (latency balloons); and skipping evaluation (measure answer or retrieval quality with and without reranking on real queries). Reranking improves ordering, not recall — fix first-stage recall separately.
Reranking adds a precise second stage to RAG retrieval. Fast first-stage search (a bi-encoder) is coarse and often buries the best passage; a cross-encoder reranker reads the query and each passage together to score relevance accurately. The standard pattern retrieves a wide candidate set for recall, then reranks to the top few for the model. It costs latency per candidate but usually lifts answer quality — just remember it improves ordering, not recall.
Take a RAG app that sometimes cites an on-topic but wrong passage. Add reranking: choose how many candidates to retrieve and how many to keep, explain why a cross-encoder helps here, and describe the before/after measurement that would prove it worked.
What is reranking in RAG?
Reranking refines the order of an initial candidate set so the most relevant passages reach the model, improving answer quality.
Why is a cross-encoder reranker more accurate than the first-stage retriever?
The cross-encoder attends jointly to query and passage, giving a much better relevance judgment at the cost of running once per candidate.
What is the standard two-stage retrieval pattern?
Retrieve-wide, rerank-narrow ensures the right passage is captured in stage one and surfaced to the top in stage two.
What is a key limitation of reranking?
A reranker only reorders what it receives, so poor first-stage recall must be fixed separately; reranking also adds latency per candidate.