Training Compute-Optimal Large Language Models

Hoffmann et al. · 2022 · NeurIPS

Why this paper matters

Corrected Kaplan et al.’s scaling laws and showed that previous frontier models (GPT-3, PaLM) were dramatically under-trained. The headline finding — data should scale roughly in lockstep with parameters — reshaped how every frontier lab budgets compute.

The core idea

Run hundreds of training runs across model sizes and token counts at fixed FLOPs budgets to find the compute-optimal (N, D) pair. Conclusion: a 70B model wants roughly 1.4T tokens; D ≈ 20·N is roughly compute-optimal (ignoring inference costs).

Method (sketch)

To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).

Key results

To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.

Connections

To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.

Reference

Jordan Hoffmann et al. (DeepMind). Training Compute-Optimal Large Language Models. NeurIPS, 2022. [link]