Search & Retrieval · Evaluation: Reading Guide
“Search improved” can mean several things: the index found more exact neighbors, the ranking contains more useful results, or users completed more tasks. Each claim needs different evidence. This series teaches how to build that evidence and how to locate a failure when the scores disagree.
You only need to know what a query, a ranked result list and a relevance label are to begin. The first three chapters develop ranking evaluation; the next two compare systems and diagnose stages. The final two connect these measurements to clicks, experiments and generated answers. Equations are introduced inside the worked examples where they are needed.
Choose the claim before choosing the metric
| What you want to know | Evidence to inspect | Start here |
|---|---|---|
| Are useful results easy to find? | Judged relevance, result order and the visible cutoff | Binary metrics and graded ranking credit |
| Did approximation lose the right candidates? | ANN output against exact search under the same vectors, distance and eligibility rules | The two meanings of recall |
| Is the new system’s gain convincing? | Matched query scores, uncertainty and regressions within important slices | Paired model comparison |
| Did the product help users? | A randomized comparison of a declared user outcome | Online experiments |
| Did the answer use and cite sufficient evidence? | The actual context, answer claims and resolved citations | RAG evaluation |
If the relevance labels themselves are uncertain, begin with the test-collection chapter. A metric cannot resolve an undefined intent or turn an unjudged result into a known negative.
Seven chapters, each with one main job
Build a relevance rubric, see how incomplete judgment pools affect a comparison, and distinguish diagnostic query samples from traffic-weighted estimates.
Follow each relevant hit into precision, recall, AP and reciprocal rank. Inspect the denominators before averaging over queries.
Build nDCG from grades, gains and positions. Compare the full judged ideal with an ideal restricted to what the system happened to return.
Separate exact-neighbor overlap from relevance, calculate a candidate ceiling, and inspect filtering and reranking with controlled changes.
Inspect all 256 resamples of a four-query example, then group repeated users correctly and distinguish an interval from a hypothesis test.
Swap result positions, reweight action choices, and see why missing support, clipped weights and changing experiment denominators matter.
Follow missing context, inspect individual claims and citations, and evaluate abstention with both answer quality and coverage visible.
Build an evaluation you can explain one query at a time
On a first pass, read the main example and predict one change before touching a control. Skip the optional code and test details until the distinction is clear. On a second pass, reproduce the calculation and connect it to your own data. The figures use small constructed examples; they are not benchmarks of real systems.
Carry four connected artifacts through an evaluation. A run is the ordered list of IDs a system returned for each query. Relevance judgments, often called qrels, associate query–document pairs with grades. Keep these identities through scoring so you can return from an aggregate to an actual failure.
Question, intent, document revision, relevance grade and eligibility
The same queries, ordered IDs and a recorded configuration for each system
Scores, differences, inclusion counts and the units used for uncertainty
Which useful item disappeared, which order changed, or which evidence never reached the answer
Before comparing two scores, align the cutoff, judged collection, gain convention, unknown-label policy and query weighting. Preserve missing and undefined cases with their counts. Otherwise a change in the evaluation population can look like a change in the ranker.
For a first small exercise, calculate one query from chapter 2, reorder a result, then explain why some metrics move while others do not. After that, add a second query and inspect how averaging changes the question. Broaden to representative queries and paired uncertainty before drawing a system-level conclusion.
Run the standard-library metric lab
Download retrieval_metrics.py and run these commands from the directory containing it:
python retrieval_metrics.py
python retrieval_metrics.py --self-test
The first command prints chapter 2’s example: precision 0.60, recall 0.75, AP about 0.4417 and reciprocal rank 0.50, along with graded scores and counts. The second runs 11 checks, including short rankings, duplicate IDs, unjudged results and undefined metrics. No third-party packages are required.
The helper defaults to all known relevant documents in the AP denominator and exponential gains for nDCG. It rejects unjudged returned IDs unless a policy is explicitly supplied. Use the chapters to understand these choices before changing them. Its paired bootstrap assumes independent paired units; repeated users require a suitable cluster analysis.
Take the next step that matches your system
For the mechanics and workload cost of vector search, continue with HNSW and Annoy and ANN benchmarking. Keep index approximation measurements separate from human relevance judgments.
For a generated-answer product, bring the saved retrieval trace into RAG system design and chapter 7’s claim audit. For a search product, bring the query-level diagnosis into chapter 6’s user experiment. The next measurement should resolve the uncertainty behind the next engineering decision.
References and further reading
- Manning, Raghavan and Schütze: evaluation of ranked retrieval — the metric foundations developed in chapters 2 and 3.
- Smucker, Allan and Carterette: significance tests for IR evaluation — primary research behind the paired-comparison discussion in chapter 5.
- Joachims, Swaminathan and Schnabel: learning from biased feedback — the click and propensity assumptions examined in chapter 6.
- Gao et al.: ALCE — the answer and citation evaluation distinctions developed in chapter 7.