Dimensionality Reduction

September 2026 · 10 min read

Dimensionality reduction replaces many input features with fewer coordinates. The useful question is what those coordinates should retain. A compact reconstruction, a good predictor and a readable scatterplot are different goals; one method need not satisfy all three.

Start with the curse of dimensionality, then PCA. After that, use the paths below. The individual chapters contain the derivations and reproducible examples, so this guide focuses on choosing and evaluating a representation.

Choose the job before the algorithm

Imagine a dataset with 100 measurements per observation. Reducing it to ten coordinates might make a predictor smaller; reducing it to two might let us draw a picture. Neither choice tells us whether important information survived. That depends on the job.

Feature selection is a separate option. Selecting ten of the original measurements keeps their names and units. A ten-coordinate PCA representation usually mixes measurements into new axes. If the real objective is to reduce which measurements must be collected, an encoder that still needs all 100 inputs may not solve it.

A reading path through the seven chapters

1. The curse of dimensionality. See why covering a space becomes expensive and why irrelevant coordinates can weaken a distance calculation. This motivates reduction without assuming that every high-dimensional dataset needs it.
2. PCA. Rotate a projection axis and compare retained variation with reconstruction error. Learn the basic fit–transform–reconstruct workflow and why high variance need not mean strong predictive signal.
3. Kernel PCA. Follow a curved example through an explicit feature map before using a kernel matrix. The new-point calculation explains why training centering must be retained.
4. Linear discriminant analysis. Project two labeled classes along different directions. Compare separation of class means with scatter inside each class, then contrast the result with PCA.
5. t-SNE. Build neighbor probabilities before watching points move. The parameter comparisons explain why gaps and cluster sizes in a picture need cautious interpretation.
6. UMAP. Build directed local weights, combine them into a graph, and inspect actual layouts. Separate the neighborhood used to build the graph from the controls used to lay it out.
7. Autoencoders. Begin with a one-number code for a curve, then inspect a trained encoder and decoder. A missed held-out point makes the difference between training success and useful reconstruction concrete.

For prediction, continue from PCA to LDA if class labels are relevant, or to a nonlinear representation if the linear baseline is inadequate. For visual exploration, go from PCA to t-SNE and UMAP. For reconstruction, go from PCA to autoencoders. Kernel PCA is the bridge for understanding feature maps and kernel-based coordinates.

What each method asks the coordinates to do

“Preserve structure” is too vague to compare methods. The table names the structure each chapter actually optimizes or constructs. Its label column refers to the standard versions taught here; extensions can use additional supervision.

Judge the representation on the same task as the baseline

For a predictor, compare the original features and reduced features using the same validation design and task metric. Fit preprocessing, reduction and prediction on each fold’s training rows. Choose the number of coordinates with validation data; a two-dimensional picture does not establish that two coordinates are enough for prediction. See the cross-validation guide for the full evaluation procedure.

For reconstruction, compare errors in the same units and inspect individual inputs. The autoencoder example beats its PCA baseline on average but still misses one intervening point badly. An average can conceal the failure that matters to your application.

For a scatterplot, inspect actual neighbors and repeat plausible parameter settings and random seeds. Use the plot to form questions about the original observations. A clear-looking island is not, by itself, evidence of a real category; a wide gap is not a calibrated distance between groups.

Finally, count the costs that remain. A small output can still require storing training examples, evaluating many kernels, or running a neural encoder. Measure fit time, memory and new-row latency if those motivated the reduction.

A small first experiment

Choose one dataset and state the intended use in one sentence. Compare its original representation with a PCA baseline, using a split that matches the use case. Inspect what is lost, then choose one additional chapter based on that specific limitation. This gives each added method a question to answer.

References and further reading

• scikit-learn: PCA and decomposition — the fitted linear representation, centering and projection of new data.
• scikit-learn: dimensionality reduction with LDA — the supervised projection and its class-separation criterion.
• van der Maaten and Hinton: Visualizing Data using t-SNE — the probability-matching objective behind the visualization.
• UMAP: transforming new data — placing held-out observations with an existing fitted model.
• Goodfellow, Bengio and Courville: Autoencoders — reconstruction learning, bottlenecks and regularization.
• scikit-learn: data leakage — why fitted transformations belong inside the training split.