Language Models are Unsupervised Multitask Learners

Radford et al. · 2019 · OpenAI Technical Report

Why this paper matters

The first crisp demonstration that a large enough autoregressive Transformer trained on web text can do zero-shot task transfer — no fine-tuning required. The intellectual bridge from BERT-style pretraining to GPT-3 in-context learning.

The core idea

Scale up GPT (decoder-only Transformer trained left-to-right on next-token prediction) to 1.5B parameters, train it on WebText (a curated crawl), and evaluate it on standard benchmarks by simply prompting it with task descriptions — no task-specific fine-tuning.

Method (sketch)

To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).

Key results

To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.

Connections

To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.

Reference

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Technical Report, 2019. [link]