Search Evaluation Starts with Relevance, Not a Metric

September 2026 · 14 min read

Before you start: basic training, validation and test splits. No ranking-metric knowledge is required.

A search score is only meaningful relative to a clearly defined need and a set of judgments. For “waterproof hiking shoes,” popularity or attractive images do not establish usefulness. A highly ranked result must satisfy the conditions the evaluator agreed to judge.

We will build one small example, then see how missing judgments and the query mix can change a comparison without changing either system.

State the need before assigning grades

The unit here is a query–item pair. The same shoe can be relevant to one request and irrelevant to another. For document search, decide whether the judged unit is a passage or a whole document; for recommendations, include the available user context and decision time.

Separate eligibility from graded usefulness. In this example, stock, region and required size determine whether an item can be served. Among eligible items, waterproofing and suitability for hiking are essential requirements. Price and color are declared preferences. Returning an ineligible item is an eligibility failure to report, not a new candidate to add to the evaluation’s ideal answer.

This rubric is illustrative. A price cap, for example, might be a hard requirement in another task. Write such choices before annotation. For the binary calculations below, grades 1, 2 and 3 count as relevant; grade 0 does not. More demanding tasks can choose a different threshold.

Give annotators the same query context and boundary examples. Hide the producing system and its rank when practical. Have a sample judged independently twice, investigate disagreements and retain both original labels when producing an adjudicated judgment. Agreement helps diagnose ambiguity; it does not prove that the rubric measures user value.

A missing judgment can change which system appears better

Judging every query against every catalog item is often too expensive. Pooling collects candidate results from multiple retrieval systems, deduplicates them for each query, and sends that union for judgment. Include different approaches and inspect newly retrieved items when the comparison expands.

Our old pool has grades A: 3, B: 0, C: 2. The baseline returns [A, B, C]. A new run returns [D, A, E], where D and E are not yet judged. All five IDs are eligible; their order remains fixed throughout the figure.

To keep the arithmetic small, count the fraction of relevant items in the first three slots, called precision@3. A common provisional scoring rule gives unjudged items zero credit. Also report judged coverage: how many of the returned items actually have a label.

With the old judgments, the zero-credit rule reports 2/3 for the baseline and 1/3 for the new run. But the new run’s actual binary precision could lie anywhere from 1/3 to 1, depending on its two unknown labels. In the expanded example, both turn out relevant, so it scores 1.

These are bounds from missing labels, not statistical confidence intervals. They assume the known labels are correct and concern these fixed three results. They do not quantify uncertainty about future queries or annotator disagreement.

Deleting unjudged results is another policy, with another consequence: the new run would shrink to [A], which looks perfect among its judged items. That no longer evaluates the original three-slot page. Record the scoring convention and coverage together; a numeric fallback does not convert “unknown” into a factual negative.

from retrieval_metrics import ranking_metrics  # Download the linked helper.

runs = {"baseline": ["A", "B", "C"], "new": ["D", "A", "E"]}
qrels_v1 = {"A": 3, "B": 0, "C": 2}  # D and E have no judgment yet.
qrels_v2 = {**qrels_v1, "D": 3, "E": 1}
for version, judgments in [("v1", qrels_v1), ("v2", qrels_v2)]:
    for name, ranking in runs.items():
        result = ranking_metrics(ranking, judgments, 3, unjudged="zero")
        lower = result["retrieved_relevant"] / 3
        upper = (result["retrieved_relevant"] + result["unjudged_count"]) / 3
        print(version, name, result["judged_fraction_of_returned"],
              result["precision"], (lower, upper))
# v1 baseline: coverage=1, precision=2/3, possible precision=(2/3, 2/3)
# v1 new:      coverage=1/3, precision=1/3, possible precision=(1/3, 1)
# v2 baseline: coverage=1, precision=2/3, possible precision=(2/3, 2/3)
# v2 new:      coverage=1, precision=1,   possible precision=(1, 1)

The query mix determines what an average means

A traffic sample estimates behavior on that traffic distribution. A balanced collection of common and difficult queries can be better for diagnosing failures. These are different summaries, even when they use the same per-query metric.

Suppose the baseline’s mean precision@3 is 0.90 on common queries and 0.20 on difficult queries. The new system scores 0.80 and 0.60. With a 90% common-query share, the baseline averages 0.83 and the new system 0.78. With a 50/50 mix, the averages are 0.55 and 0.70.

At a 90% common-query weight, the baseline calculation is $0.9\times0.90+0.1\times0.20=0.83$. A diagnostic set with equal numbers from each group estimates the 50/50 average unless you deliberately reweight it. Preserve query slice tags and report the weighting; do not present the balanced average as the traffic estimate.

Useful slices may include spelling errors, rare intents, languages, strict filters and new items. Keep their sample sizes visible. A small slice can expose a failure without supporting a precise estimate of its frequency in production.

Save enough information to replay the comparison

Qrels are query relevance judgments: at minimum, a query ID, an item/document ID and a grade. A run is the ranked list produced by one system for a query. Give their versions separate identities so expanded judgments can be applied to the same saved runs.

For our example, a record such as (query=q17, item=A, grade=3, judgments=v1) is distinct from (query=q17, item=A, rank=1, run=baseline-v1). Preserve the query’s text, filters and time, the eligible catalog snapshot, rubric version, retrieval configuration and evaluator settings alongside these records.

When the pool expands, evaluate both saved runs under the new judgments. Keep the old report too. This lets a reviewer distinguish a better system from a different reference set.

Freeze what was knowable at the decision time

Separate training, validation and final testing before learning representations or tuning retrieval choices. For a future-traffic evaluation, feature values, graph edges, inventory and user history must reflect what was available for each query. A label created later can describe the eventual outcome; it cannot justify using future features in the prediction.

Group repeated users, query variants or near-duplicate items when the intended generalization requires keeping them together. A random row split can otherwise test memorized examples. The splitting guide explains how to match the split to the task.

Clicks introduce exposure and position effects; “not clicked” need not mean “not relevant.” Read click bias and online experiments before converting logs into labels. If an LLM assists judgment, preserve its model version, rubric, prompt and parser, and assess it against independent human judgments.

Once the task, judgments and query set are fixed, continue to precision, recall, AP and reciprocal rank. The metrics will then answer questions about a defined experiment.

References and further reading

• Manning, Raghavan and Schütze: information retrieval system evaluation — test collections, information needs and separation of development from final testing.
• Manning, Raghavan and Schütze: assessing relevance — human judgments, pooling and inter-assessor disagreement.
• NIST: the TREC evaluation process — participating systems return ranked results that are pooled and judged.
• ir-measures: measure definitions — relevance thresholds and judged-only evaluation options; check a measure’s exact convention.