Evaluate Retrieval and Reranking Separately

September 2026 · 14 min read

Before you start: nDCG and the full-catalog ideal. This chapter uses those scores to locate a failure inside a search pipeline.

A final ranking score tells you how the page performed, but not which component to change. The embedding score can prefer unhelpful items. An approximate index can miss the score’s best neighbors. Filtering can leave too few candidates. A reranker can bury the useful items it receives.

We will keep one eligible, fully judged catalog: A: 3, B: 2, C: 1, D: 0, E: 0. Thus A, B and C are relevant. Our final page has three slots. The numbers below are constructed examples, not measurements from a trained search system.

First ask what the index is trying to reproduce

Exact vector search scores every eligible item under the chosen similarity function and returns its top neighbors. An approximate nearest-neighbor (ANN) index tries to recover those neighbors with less search work. The exact score is its reference, even when that score disagrees with human relevance.

Suppose exact search returns D, A, C, while ANN returns D, C, E. Compare the approximate results with two targets:

Here ANN overlap recall@K means the number of shared IDs between exact and approximate top-K sets, divided by K. Relevance recall@K divides the number of retrieved relevant IDs by all known relevant IDs. Use unique result IDs and the same eligible corpus, vectors, similarity function and cutoff for the exact–approximate comparison. Resolve score ties consistently.

Replacing ANN with exact search recovers A. Relevance recall rises from 1/3 to 2/3, but B is still missing at this cutoff. Improving approximation alone cannot fix that miss. It may require a different representation, a complementary retriever or a larger candidate set.

Read benchmark definitions carefully. The Faiss benchmark notes also use a different measure called 1-NN recall at R: whether the single nearest neighbor appears anywhere in the first R results. That is not the top-K set overlap defined here.

Separate missing candidates from poor ordering

A reranker can only reorder the candidates it receives. To measure its available opportunity, sort those candidates by their known relevance grades and score that list against the full eligible catalog’s ideal. Call this the candidate oracle. It uses evaluation labels, so it is a diagnostic ceiling rather than a deployable ranking rule.

For our original candidates D, C, E, the oracle is C, D, E. Only C has positive gain. With exponential gain and K=3, the actual order earns about 0.0672 nDCG, and the oracle earns about 0.1065. The ideal denominator is still the score of A, B, C, approximately 9.3928, as derived in the previous chapter.

The distance from the actual score to the oracle is the room to improve ordering. The distance from the oracle to 1 is the loss caused by the available candidate set. The figure keeps these two gaps on the same scale.

Try Include A, then apply the oracle. A becomes available, but B is still missing, so even the best order remains below 1. With All three relevant, the oracle reaches 1; a poor actual order can still lose credit.

These gaps describe headroom under the judgments. They do not prove which change is cheapest, feasible or causally responsible for a production regression. An almost-oracle reranker has little ordering headroom on that candidate set; use controlled component swaps before choosing the engineering intervention.

from retrieval_metrics import ranking_metrics

qrels = {"A": 3, "B": 2, "C": 1, "D": 0, "E": 0}
runs = {"original": ["D", "C", "E"],
        "include_A": ["D", "C", "A"],
        "all_relevant": ["C", "B", "A"]}
diagnostics = {}
for name, run in runs.items():
    oracle = sorted(run, key=lambda item: (-qrels[item], item))
    actual = ranking_metrics(run, qrels, 3)["ndcg"]
    ceiling = ranking_metrics(oracle, qrels, 3)["ndcg"]
    diagnostics[name] = (actual, ceiling, ceiling - actual, 1 - ceiling)
    print(name, *(round(x, 4) for x in diagnostics[name]))
# Columns: actual, ceiling, ordering gap, candidate gap.
# The two gaps plus the actual score sum to 1.

Match the exact baseline to the eligibility rule

Now suppose the vector index also contains X and Y, which are unavailable for this query. They are outside our eligible catalog. The global exact score order is X, Y, D, A, C, B, E; exact search within the eligible catalog returns D, A, C.

If we fetch the global top three and filter afterward, only D survives. This failure happens even with exact vector scores. The problem is where filtering occurs and how many items were fetched.

Fetching five global neighbors recovers the eligible top three. Fetching still more does not change this three-slot output. Its index overlap is now perfect, yet relevance recall remains 2/3 because the embedding score still puts D above B.

Oversampling means fetching extra items before filtering and trimming. It can help fill the list, but offers no universal multiplier that works for every filter. In-query filtering or partitioned indexes are other options; their recall and cost depend on the implementation. Faiss documents ID-based subset selection and index-specific behavior. Benchmark the actual path your application uses.

Change one component at a time

Freeze the queries, catalog snapshot, relevance judgments and eligibility rules. Save the candidate IDs, scores, filter decisions and final order for every run. Then make comparisons that answer a specific question:

Finally test the proposed complete stack. Component changes can interact: a new candidate generator can expose cases the old reranker handles poorly. A larger candidate set can raise its oracle ceiling while the actual output gets worse.

Sweep candidate count and search effort against end-to-end quality and measured latency. Include query embedding, index lookup, filtering, transfer and reranking. Measure p95 on complete query requests; adding separate components’ p95 values does not generally give the request’s p95. Keep workload, concurrency, hardware, warmup and failure counts with the measurements.

Use the failure cases to choose the next experiment

Break out exact identifiers, paraphrases, rare entities, languages, new items and strict filters. Keep query counts visible. A lexical retriever may recover identifiers missed by a dense retriever; test whether their union adds relevant items at the candidate budget you can afford.

Check judgment coverage at each candidate boundary. Otherwise a new retriever can look weak simply because its new results have not been assessed. Use representative data for quality and performance comparisons; synthetic vectors are useful for implementation checks but cannot establish product relevance.

After locating a promising change, the next question is whether its measured gain is stable across queries. Continue to paired comparisons and uncertainty.

References and further reading

• Faiss: evaluating vector indexes — exact-neighbor references, distinct recall definitions and measured speed/accuracy comparisons on specific workloads.
• Faiss: per-query search parameters and subset selection — search effort, ID selectors and filtering behavior that depends on the index and selector.
• Manning, Raghavan and Schütze: ranked retrieval evaluation — the relevance-based metrics used to evaluate candidates and final rankings.