Clicks Are Feedback, Not Relevance Labels

September 2026 · 15 min read

Before you start: paired comparisons and the experimental unit. This chapter moves from judged relevance to behavior observed in a product.

The first result gets more clicks than the fifth. Perhaps it is better; perhaps more people examine it. A ranker chooses what users can encounter, then learns from the resulting behavior. Treating that behavior as an unbiased relevance label can reinforce the ranker’s own earlier choices.

We will answer three questions: how position changes clicks, what reweighting can recover from an old policy’s log, and why a randomized experiment still needs a carefully defined outcome. All numbers below are constructed examples, not measurements from a search system.

Separate seeing a result from clicking it

Imagine two equally attractive results, A and B. Once examined, each gets clicked with probability 0.50. But users examine position 1 with probability 0.80 and position 5 with probability 0.20. The click probabilities are therefore 0.80 × 0.50 = 0.40 and 0.20 × 0.50 = 0.10.

Now name the pieces. Let $q$ be a query, $d$ a document and $i$ its displayed position. Let $E=1$ mean the user examines it, and $C=1$ mean a click. Write $e_i$ for the examination probability at position $i$, and $c_{q,d}$ for the click probability given examination. Our model says:

This assumes clicks require examination, examination depends only on position, and attractiveness once examined does not change with position. Real behavior can violate these assumptions: layout, trust in the top result and earlier clicks can matter. Even attractiveness is not relevance; an appealing title can lead to an unhelpful page.

A non-click may mean the result was never examined, the user stopped after an earlier success, or the answer was already on the page. Record what was displayed and where, along with available visibility signals. A result entering the viewport is still only a proxy for attention.

Correcting unequal examination in this model

With a known, positive examination probability, weight a click by 1 / examination probability. At position 1, the expected weighted click is 0.40 × 1.25 = 0.50. At position 5, it is 0.10 × 5 = 0.50. Non-clicks contribute zero, and the average includes all impressions.

This recovers click attractiveness in expectation under our model. It does not automatically recover human relevance. Randomized position swaps can help estimate relative examination propensities under suitable assumptions; they do not reveal attention perfectly. Joachims, Swaminathan and Schnabel develop propensity-weighted learning and examine the effects of click noise and misspecified propensities.

A different question: how would another selection policy perform?

Now consider one recommendation slot with two possible items. The old system chooses A 90% of the time and B 10%. We want to evaluate a target policy that chooses each 50% of the time. Simply averaging old clicks evaluates the old mixture, which contains too much A for our question.

The relevant propensity is now the system’s probability of choosing an action, not the user’s probability of examining it. An action here is the item placed in the single slot. Inverse propensity scoring, or IPS, weights each logged reward by target choice probability / logging choice probability.

Before the formula, define its symbols. There are $N$ logged impressions, including non-clicks. Record $t$ contains context $x_t$, chosen action $a_t$, and reward $r_t$ (1 for a click, 0 otherwise). The logging policy $\mu(a\mid x)$ and target policy $\pi(a\mid x)$ give action probabilities in that context. The target’s expected reward is $V(\pi)$; its IPS estimate is:

For our 90/10 logger, an A click gets weight 0.50/0.90, about 0.556; a B click gets weight 0.50/0.10 = 5. This reduces A’s overrepresentation and increases B’s underrepresentation. Divide the weighted reward total by all $N$ impressions, not just the clicked ones.

Inspect the same target under four logging policies

Hold the context and slot fixed. If selected, A is clicked with probability 0.20 and B with probability 0.80. The target’s true mean in this toy world is 0.50 × 0.20 + 0.50 × 0.80 = 0.50. These conditional click rates include any examination behavior in this slot.

With the 90/10 logger, A supplies 180 clicks and B supplies 80. Weighting gives 180 × (0.5/0.9) + 80 × 5 = 100 + 400 = 500. Divide by 1,000 to recover 0.50. The raw mean, 260/1,000 = 0.26, answers the old-policy question.

When B is chosen only 1% of the time, each B click has weight 50. One such click contributes 50/1,000 = 0.05 to this estimate. Rare events can move it substantially. Capping weights at 2 limits that influence, but the illustrated estimate falls to 0.116. Clipping trades variance for bias; it changes the estimator.

When B is never chosen, its reward is absent. The target still chooses B, so the log lacks support for the target. Reweighting cannot supply the missing outcomes, and clipping cannot repair this. The figure therefore shows “Unsupported,” not a fabricated estimate. The toy value 0.50 is known only because we supplied B’s reward model separately.

For unbiased IPS with known logging probabilities, every action the target might choose needs positive logging probability in the relevant context. The reward distribution conditional on context and action, and the evaluated context population, must also match the target setting. Record the actual randomization probabilities; guessing them later can bias the result. Dudík, Langford and Li explain this correction and its variance problem.

from fractions import Fraction as F

def evaluate(b_probability, cap=None):
    mu = [1 - b_probability, b_probability]
    target = [F(1, 2), F(1, 2)]
    reward = [F(1, 5), F(4, 5)]
    impressions = [1000 * p for p in mu]
    clicks = [n * r for n, r in zip(impressions, reward)]
    raw = sum(clicks) / 1000
    if any(p > 0 and m == 0 for p, m in zip(target, mu)):
        return raw, None
    weights = [p / m for p, m in zip(target, mu)]
    if cap is not None:
        weights = [min(w, cap) for w in weights]
    ips = sum(w * c for w, c in zip(weights, clicks)) / 1000
    return raw, ips

for p in [F(1, 2), F(1, 10), F(1, 100), F(0)]:
    print(p, evaluate(p), evaluate(p, cap=F(2)))

For a whole ranking, the action may be an entire ordered list. Per-item ratios require additional assumptions to stand in for that list’s selection probability. An examination weight and a policy-choice ratio correct different mechanisms; multiplying arbitrary versions together does not establish an unbiased slate estimator.

Randomize the experience, then keep the outcome’s denominator honest

A randomized experiment directly assigns eligible users to the old or new experience. Define eligibility before assignment and keep assignment stable when effects span sessions. Log assignment, actual exposure and outcomes separately; they answer different questions. Kohavi et al. describe stable user assignment and the tradeoffs among outcome and guardrail metrics.

Consider 100 eligible users in each arm, each making one query. In this toy funnel, a task can be completed only after a click. Control gets 60 clicks and 30 completions; treatment gets 80 clicks and 32 completions. Keep those counts fixed while changing the denominator:

For all assigned users, completion increases from 30% to 32%. Among clickers, it decreases from 30/60 = 50% to 32/80 = 40%. The conditional rate may help diagnose the funnel, but it does not isolate the treatment effect on the same population. A 20-percentage-point click increase is also not a 20-point completion increase.

These small constructed counts establish no statistical significance. Use the experimental-unit discussion to plan uncertainty. Choose the primary outcome, guardrails and stopping rule in advance. Here completion per eligible user could be primary, clicks diagnostic, and latency and error rate guardrails.

Before interpreting metric movement, compare actual traffic allocation with the design and investigate missing or inconsistent events. A treatment can change what gets logged, or change a ratio’s denominator. Microsoft’s experimentation guidance gives examples of these failures. Properly designed triggered analyses need their own inclusion rules; selecting people because they clicked is not an equivalent shortcut.

Use disagreement to locate the missing mechanism

Suppose judged nDCG improves but task completion does not. Compare the evaluated queries with live traffic, inspect the affected query slices, and check whether latency or presentation changed. A gain concentrated in a rare intent may barely move the traffic average. Conversely, clearer snippets can improve clicks without changing the relevance of the underlying order.

Interleaving combines results from two rankers in one displayed list and attributes interactions under a specified protocol. It can address a ranker-preference question, while an A/B test of the deployed experience addresses broader outcomes. Neither choice makes clicks self-explanatory.

The useful chain is judged relevance → displayed experience → user outcome. Diagnose each link. The next chapter extends that separation to retrieved evidence and generated answers.

References and further reading

• Joachims, Swaminathan and Schnabel: Unbiased Learning-to-Rank with Biased Feedback — position-based examination, propensity estimation and the assumptions behind learning from clicks.
• Dudík, Langford and Li: Doubly Robust Policy Evaluation and Learning — the contextual-bandit evaluation problem, importance-weighted correction and its variance; our example isolates IPS.
• Kohavi et al.: Online Controlled Experiments at Large Scale — stable assignment, outcome metrics and practical experiment tradeoffs.
• Microsoft Experimentation Platform: interpreting an experiment — traffic allocation, missing telemetry, ratio denominators and the conditions for triggered analysis.