Scaling Laws for Neural Language Models

Kaplan et al. · 2020 · Preprint (arXiv)

Why this paper matters

Showed that test loss for autoregressive LMs follows clean power laws in model size, dataset size, and compute — over many orders of magnitude. This is the paper that made “just scale it” a defensible engineering plan rather than a hope.

The core idea

Train hundreds of small-to-medium Transformers across systematically varied N (parameters), D (tokens), and C (compute). Fit power laws of the form L = (N_c / N)^α_N + … and use them to extrapolate the loss of much larger runs.

Method (sketch)

To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).

Key results

To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.

Connections

To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.

Reference

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei. Scaling Laws for Neural Language Models. Preprint (arXiv), 2020. [link]