Language Models are Few-Shot Learners

Brown et al. · 2020 · NeurIPS

Rough draft. This page is a stub from the “Recommended Papers” scaffold. Expect a proper, structured write-up — problem, method, results, what makes it special — in a future revision.

Authors: Tom B. Brown et al. (OpenAI)

Year: 2020

Venue: NeurIPS

Link: https://arxiv.org/pdf/2005.14165.pdf

Why this paper matters

Discovered in-context learning — that a 175B-parameter autoregressive LM can perform new tasks from a handful of examples in the prompt, with no parameter updates at all. The paradigm shift from fine-tuning to prompting was born here.

The core idea

Train a 175B-parameter decoder-only Transformer on a mixed corpus (Common Crawl + WebText + books + Wikipedia). Evaluate without fine-tuning, by providing zero, one, or a few exemplars in the prompt. Performance scales smoothly with model size across most tasks.

Method (sketch)

To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).

Key results

To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.

Connections

To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.

Reference

Tom B. Brown et al. (OpenAI). Language Models are Few-Shot Learners. NeurIPS, 2020. [link]