nDCG: How Much Does the Order of Useful Results Matter?

September 2026 · 13 min read

Before you start: precision, recall, AP and reciprocal rank. Those metrics use binary relevance; this chapter keeps the differences between positive grades.

Suppose two useful documents exchange places: an excellent match moves below a marginally useful one. Binary precision cannot detect that change. Both are still “relevant.” Normalized discounted cumulative gain, or nDCG, gives stronger matches more credit and gives credit near the top more weight.

Use a fully judged, eligible catalog with four documents: A has grade 3, B grade 2, C grade 1 and D grade 0. The system’s order is B, D, A, C. At cutoff $K=3$, we evaluate grades [2, 0, 3]. Here grades 1–3 represent increasing usefulness under the declared rubric; all three meet the task’s essential requirements.

Turn a grade into credit, then account for its position

A grade is an annotation. A gain specifies how much evaluation credit that grade earns. Start with exponential gain: $g(r)=2^r-1$, where $r$ is a relevance grade. Grades 0, 1, 2 and 3 therefore earn 0, 1, 3 and 7 units of credit.

Next apply a rank discount. At rank $i$, multiply the gain by $1/\log_2(i+1)$. The first position keeps all its gain; the second keeps about 0.631; the third keeps 0.5. This is a chosen way to value prominence, not a measured probability that a user examines the result.

Discounted cumulative gain (DCG) adds these contributions over the first $K$ positions. In the equation, $r_i$ is the grade of the document at rank $i$, and $g$ is the gain mapping we just chose:

For B, D, A at the top, the three contributions are 3 × 1 = 3, 0 × 0.631… = 0 and 7 × 0.5 = 3.5. Thus DCG@3 = 6.5. D earns nothing, but occupying rank two pushes a valuable result farther down.

Compare that credit with the best available list

A raw DCG of 6.5 has little meaning without knowing what the query could achieve. Build the best top-three list from the catalog’s judgments: A, B, C, with grades [3, 2, 1]. Its ideal DCG, abbreviated IDCG, is 7 + 3/log₂3 + 1/2 ≈ 9.3928.

Divide the actual credit by this ideal credit, using the same cutoff, gain mapping and discount in both:

Our ranking scores 6.5 / 9.3928… ≈ 0.6920. It earns about 69.2% of the ideal discounted gain under this rubric. With nonnegative gains and a positive ideal, the score lies between 0 and 1. It is not the fraction of relevant documents retrieved and does not mean that 69.2% of users will succeed.

The gain mapping changes what improvement is worth

We could instead use linear gain, $g(r)=r$. Grade 3 would be worth three times grade 1, rather than seven times. Exponential gain makes putting an excellent match near the top especially valuable. Neither ratio follows automatically from calling a result “excellent.”

In the figure, first move A upward with the arrow buttons. Then switch the gain mapping without changing the order. Both the actual and ideal gain change; the normalized score can change too. Best order reaches 1 under either mapping because actual and ideal agree.

Choose a gain mapping with the rubric, before comparing systems. Preserve it in the report. A rise of 0.05 nDCG is an increase on that evaluation scale; it does not establish a 5% improvement in revenue or task completion. The online-evaluation chapter addresses measured user outcomes.

An ideal built only from returned items can hide a miss

Select Miss A at K=3 above. The evaluated list is B, C, D: it omits the highest-grade item. Those three documents are already in their best internal order, but that is a weaker achievement than returning the best available documents.

With exponential gain, B, C, D earns 3 + 1/log₂3 ≈ 3.6309. Keep this same numerator and compare two denominators:

A reranker cannot introduce A if retrieval never supplied it. Sorting candidates by their known grades gives an achievable ceiling for that candidate set. The retrieval and reranking diagnostics chapter uses this distinction to locate which stage lost quality.

“Full judged catalog” still means the reference set you have assessed, not an omniscient answer about every unjudged item. Keep judgment coverage and the unknown-label policy beside the score.

Handle empty ideals and tied scores explicitly

If all eligible items have zero gain, IDCG is zero and the ratio is undefined. Our series helper returns None and reports the excluded count when averaging. Some evaluators assign zero by convention. Follow the benchmark definition when reproducing it, and evaluate correct abstention separately.

If the model gives two documents the same score, their order can affect nDCG when their grades differ. Evaluate the actual displayed order, or declare another tie rule. For example, scikit-learn averages tied-score permutations by default. A deterministic item-ID tiebreak and a tie average answer different questions; do not silently interchange them.

import numpy as np
from sklearn.metrics import dcg_score, ndcg_score

grades = np.array([[3, 2, 1, 0]])  # columns A, B, C, D
scores = np.array([[2, 4, 1, 3]])  # order B, D, A, C; no ties
results = {}
for name, gains in [("exponential", 2.0 ** grades - 1), ("linear", grades)]:
    actual = dcg_score(gains, scores, k=3)
    normalized = ndcg_score(gains, scores, k=3)
    results[name] = (actual, normalized)
    print(name, round(actual, 4), round(normalized, 4))
# exponential 6.5 0.692
# linear      3.5 0.735

Relevance order is one part of a useful page

Two lists with identical grade sequences have identical nDCG, even if one shows near duplicates and the other offers distinct options. nDCG does not read brand names, novelty or redundancy unless the evaluation explicitly models them.

Add a companion measure when it represents a task requirement: a declared duplicate rate for repetitive pages, catalog coverage over a defined traffic window for exposure breadth, or task completion for user success. A precise model-number query may need one exact match; a broad discovery query may benefit from variety.

Average per-query nDCG using a declared query set and weighting, as in the previous chapter’s aggregation example. Report the cutoff, gains, tie and empty-ideal rules, judgment coverage and an uncertainty estimate. Inspect large wins and losses to connect a changed score to changed results.

References and further reading

• Manning, Raghavan and Schütze: ranked retrieval evaluation — graded gain, logarithmic discount and normalization against an ideal ranking.
• ir-measures: nDCG options — gain mappings, DCG formulations and judged-only options differ across providers.
• scikit-learn: ndcg_score — supplied relevance values act as gains; cutoffs, query averaging and the default tie-average behavior.
• NIST trec_eval: nDCG at cutoffs — an implementation that uses qrel values as gains; do not assume an implicit exponential mapping.