Precision, Recall, MAP, and MRR: One Ranking, Four Questions

September 2026 · 14 min read

Before you start: relevance judgments and test collections. Here every item is judged and eligible; we can focus on what the metrics count.

A search engine returns B, A, C, E, D, F. The useful items in this six-document catalog are A, C, D and F. The first result is wrong, three useful results are in the first five slots, and one sits just beyond them.

We will keep that example throughout the chapter. Precision asks how much of the displayed list is useful. Recall asks how much useful material we recovered. Average precision rewards finding useful items early; reciprocal rank looks only for the first one.

One overlap, two denominators

Choose a cutoff $K$: the number of ranked slots being evaluated. At $K=5$, we inspect B, A, C, E and D. Three are relevant. That same count answers two different questions:

For the equation, let $r_i$ equal 1 when the item at rank $i$ is relevant, and 0 otherwise. Let $R$ be the total number of relevant items in the eligible catalog’s judgments. Here $R=4$. Adding the first $K$ indicators counts the hits:

Moving A to rank one keeps the top-five membership unchanged, so both scores stay the same. They cannot tell which useful item appears first. Increasing $K$ can recover more items and increase recall; precision can rise or fall depending on the newly included items.

Choose $K$ to match the stage being evaluated: five visible results and a hundred candidates sent to a reranker are different questions. Compare systems at the same cutoff before attributing a score difference to better retrieval.

Average precision gives credit at each relevant hit

Imagine scanning the ranking from the top. Every time you find a relevant item, record the precision of the list you have seen so far. This prefix precision rewards finding the hit with fewer irrelevant items ahead of it.

In our example, A is the first hit at rank 2: its prefix precision is 1/2. C is the second hit at rank 3: 2/3. D is the third hit at rank 5: 3/5. F is outside the cutoff and gets zero credit. Average these four contributions, one for each relevant item: (1/2 + 2/3 + 3/5 + 0) / 4 = 53/120 ≈ 0.4417.

This is average precision at K, or AP@K. In the compact formula, $P@i$ is the prefix precision at rank $i$. Multiplying by $r_i$ keeps that term only when rank $i$ contains a relevant item:

We divide by all $R$ relevant items, including those not found by the cutoff. That makes missing F matter. At $K=6$, F adds a prefix precision of 4/6; AP becomes approximately 0.6083. Irrelevant ranks contribute no direct term, but placing them before useful items lowers those items’ prefix precisions.

Reciprocal rank stops at the first useful result

For a task where one acceptable answer is enough, ask how far down the user must go before encountering any relevant item. Call that first relevant rank $j$. Reciprocal rank assigns 1 for rank one, 1/2 for rank two, 1/3 for rank three, and so on. If no relevant item appears by $K$, the score is zero:

Our first hit is A at rank two, so RR@5 is 0.5. Put A first and RR becomes 1. It stays 1 even if every later result is poor. AP can distinguish those later changes because it also counts the remaining relevant items.

Move a result and see which scores react

Start with Promote A: precision and recall stay fixed, while AP and RR improve. Then choose Delay later hits: A stays first, but C moves down. RR stays perfect while AP falls. The controls change only the ranking and cutoff; the relevance labels stay fixed.

MAP and MRR average over queries

AP and RR describe one query. Mean average precision (MAP) averages the AP values of the included queries; mean reciprocal rank (MRR) averages their RR values. Calculate each query’s score before taking the mean.

Add a second query with one relevant item, found at rank one. It has AP=1 and RR=1. Give the two queries equal weight:

The distinction also explains macro versus micro recall. Macro recall averages the two query recalls: (0.75 + 1)/2 = 0.875. Micro recall pools the counts first: (3 + 1)/(4 + 1) = 0.80. The latter gives more influence to queries with more relevant items. Neither number is a mistake; they summarize different units. For traffic weights and diagnostic query sets, use the previous chapter’s query-mixture example.

Keep the evaluation contract explicit

For a short run, this series still divides precision by the requested $K$: returning two useful items for five slots gives P@5=0.4. Empty slots receive no credit. If your product intentionally evaluates only the returned list, name that different denominator.

Require unique document IDs. Repeating A cannot recover another relevant document; our helper rejects duplicates. Also declare the unjudged-item policy. This chapter’s complete labels let us avoid treating missing assessments as known negatives.

If $R=0$, recall and AP involve division by zero. Our helper returns None, excludes those values from their means, and reports the counts. RR is zero when there is no hit, but that does not evaluate whether abstaining was correct. Report no-answer queries separately and follow a benchmark’s stated convention when comparing with it.

from retrieval_metrics import ranking_metrics, evaluate

qrels = {"A": 3, "B": 0, "C": 2, "D": 1, "E": 0, "F": 1}
run = ["B", "A", "C", "E", "D", "F"]
scores = ranking_metrics(run, qrels, 5)
print({key: round(scores[key], 4)
       for key in ("precision", "recall", "ap", "rr")})
# {'precision': 0.6, 'recall': 0.75, 'ap': 0.4417, 'rr': 0.5}

print(ranking_metrics(run, qrels, 2)["ap"])  # 0.125
print(ranking_metrics(run, qrels, 2, ap_denominator="min")["ap"])  # 0.25

report = evaluate({"q1": run, "q2": ["X"]},
                  {"q1": qrels, "q2": {"X": 1}}, 5)
print(report["macro"]["ap"])  # mean=0.720833..., included=2, excluded=0
print(report["macro"]["rr"])  # mean=0.75, included=2, excluded=0

These metrics treat all positive grades alike. If an excellent answer should earn more credit than a marginally useful one, continue to graded gain and nDCG.

References and further reading

• Manning, Raghavan and Schütze: ranked retrieval evaluation — set-based cutoffs, precision at relevant hits, missing relevant items and query-level means.
• ir-measures: reciprocal rank — the first-hit measure, relevance thresholds and cutoff options.
• NIST trec_eval: AP at cutoffs — an implementation using all relevant documents in the denominator.
• Recommenders: map_at_k source — the alternative denominator min(number relevant, K).