Evaluate Embeddings and Choose Their Dimension

September 2026 · 14 min read

Before you start: how behavioral skip-gram learns vectors. Training loss tells us whether the model solves its training comparisons. This chapter asks whether the resulting representation helps the intended task.

An embedding is useful when its neighbors support the decision you need to make. A few plausible examples are a starting point, not a model-selection procedure. We will separate relevance from nearest-neighbor accuracy, account for dimension costs, and see why deleting coordinates can change a ranking unexpectedly.

Define what a good neighbor means

For a product, “related” might mean a substitute, a complement, or another item viewed in the same session. Boots and socks are sensible neighbors for a complement task but poor substitutes. Write the criterion before looking at a model’s results.

Build held-out queries with relevance judgments that match that criterion. For a substitute task, include difficult alternatives: a visually similar boot that is not waterproof, or a waterproof shoe that lacks the required fit. If judgments simply repeat the co-occurrences used to train the model, they do not independently establish substitute quality.

Keep the prediction time, candidate catalog, eligibility filters and query exclusions fixed across models. For item-to-item retrieval, decide whether the query item itself should be excluded. Track items absent from the vocabulary; reporting only queries that happen to have vectors can hide a coverage failure. See relevance judgments and test collections for constructing the evaluation set.

Separate useful retrieval from accurate nearest-neighbor search

First evaluate the representation using exact search: score every eligible candidate and take the highest-scoring results. This removes search approximation as an explanation for a poor neighborhood. Then evaluate an approximate nearest-neighbor (ANN) index using the same vectors and score.

Consider a fully judged toy catalog for the query “substitutes for these waterproof hiking boots.” Items A, C and D are suitable boots. B is socks, E a jacket, and F a mug. The model’s exact top two are [B, A]. Its score favors socks, even though our substitute judgment does not.

Let $S_K$ be the returned top-$K$ set, $R$ the judged-relevant set, and $E_K$ the model’s exact top-$K$ set. Vertical bars mean “number of items in the set”; intersection means the items shared by both sets. Two fractions answer different questions:

Task recall asks how many relevant items were recovered. ANN overlap asks how much of the model’s exact answer was recovered; it is often called ANN recall. Here $K=2$, but there are three relevant items, so the denominators differ.

The two approximate cases have the same 50% ANN overlap but task recall of 0% and about 67%. With only two result slots, recovering two of the three relevant items is the best possible recall here. Approximation can occasionally improve a relevance score by chance; it is not a reliable relevance model. Measure both quantities so a strong index cannot conceal weak embeddings, and a strong representation is not blamed for an index miss.

For ranked usefulness, choose precision, AP or reciprocal rank, or nDCG for graded judgments, according to the user’s task. Average using a declared query weighting, and inspect head/tail items and other important slices. In a real incomplete collection, recall is relative to known judgments; unjudged items are not automatically known negatives.

Choose dimension through a controlled comparison

A dimension of 32 means each vector has 32 numbers; it does not mean it preserves 32 human-interpretable properties. Compare several dimensions while holding the evaluation protocol fixed. Record how each candidate was obtained: trained at that width, projected from a larger model, or truncated under an explicit training contract.

Use validation queries to choose the operating point and reserve a final test. Compare exact-search task quality first, then the index’s quality/latency trade-off. Use repeated training seeds when initialization matters. A smooth-looking “elbow” in a quality plot is a visual hint of diminishing returns, not an automatic choice.

Memory has a simple starting calculation. Each float32 coordinate takes 4 bytes, so N items × d coordinates × 4 bytes is the raw vector payload. With one million items and 32 coordinates, that is 128 million bytes, or 128 decimal MB.

Measure peak memory and query latency under the intended hardware, batch size, concurrency and filtering. A smaller vector does not imply a proportional latency reduction: index traversal or other work may dominate. When comparing ANN configurations, report quality alongside latency rather than quietly changing the search effort.

Before selecting a cheaper candidate, decide how much task-quality loss is acceptable. Compare models on the same queries and estimate uncertainty in their paired differences, with grouping when observations are dependent. The model-comparison chapter develops that procedure. The test set should evaluate the selected procedure, not become another tuning set.

Retraining, projection and truncation change different things

Retraining at 32 dimensions optimizes a model with that capacity. PCA projection finds directions of high variance in a fitted reference dataset. Taking the first 32 coordinates simply discards the rest. Ordinary embedding coordinates are not generally ordered by importance.

Here is a small geometric reason. Put query q at (1, 0), item A at (0.8, 0.2), and item B at (−0.5, 0). In two dimensions, A is about 0.2828 units from q; B is 1.5 units away. Rotate all three vectors together. Their full Euclidean distances stay unchanged, but the first coordinate now points along a different direction.

This does not mean dimension reduction is impossible. It means the retained directions need justification. Fit a PCA projection on the training reference data, apply its saved centering and axes to both queries and items, and re-evaluate the task. Retained variance is not itself a retrieval-quality guarantee.

Some methods explicitly train useful prefixes, such as Matryoshka Representation Learning. Follow the trained model’s supported truncation and normalization procedure. That property does not follow merely from calling a vector an embedding.

import numpy as np

relevant = {"A", "C", "D"}
exact = ["B", "A"]
cases = {"exact": exact, "lost useful item": ["B", "E"],
         "fortunate replacement": ["A", "C"]}
for name, result in cases.items():
    assert len(result) == len(set(result)) == 2
    task_recall = len(set(result) & relevant) / len(relevant)
    ann_overlap = len(set(result) & set(exact)) / len(exact)
    print(name, round(task_recall, 6), ann_overlap)
# exact                  0.333333 1.0
# lost useful item       0.000000 0.5
# fortunate replacement  0.666667 0.5

# Rotate query and both items together; then retain the first coordinate.
points = np.array([[1., 0.], [.8, .2], [-.5, 0.]])  # query, A, B
for degrees in [0, 90]:
    angle = np.deg2rad(degrees)
    c, s = np.cos(angle), np.sin(angle)
    rotation = np.array([[c, -s], [s, c]])
    rotated = points @ rotation.T
    full = np.linalg.norm(rotated[1:] - rotated[0], axis=1)
    first = np.abs(rotated[1:, 0] - rotated[0, 0])
    print(degrees, np.round(full, 6), np.round(first, 6))
# 0  [0.282843, 1.5]  [0.2, 1.5]
# 90 [0.282843, 1.5]  [0.2, 0.0]

Check stability and keep compatible versions together

Compare task scores and neighborhoods across seeds, not raw coordinates alone. A shared orthogonal rotation preserves dot products, cosine similarities and Euclidean distances. Independent training runs may also change the geometry, so the rotation example does not explain away every neighbor change.

When changing a representation, deploy its query encoder or lookup table, item vectors, preprocessing and index as a compatible set. Mixing old query vectors with a newly trained item table has no general compatibility guarantee.

A useful comparison record includes the data cutoff, vocabulary and coverage, export table, score function, dimension procedure, exact-search task score, ANN overlap, end-to-end task score, memory, latency and important slices. Add a few failure cases to explain why the chosen trade-off is acceptable.

References and further reading

• Manning, Raghavan and Schütze: evaluating retrieval sets — relevance-based precision and recall; the ANN overlap in this chapter uses a different reference set.
• Faiss: choosing an index — exact flat-search baselines and trade-offs among search quality, memory and speed.
• scikit-learn: PCA — learned projection directions and the saved training mean.
• Kusupati et al.: Matryoshka Representation Learning — training representations whose prefixes support different dimensionalities.