BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al. · 2019 · NAACL

Why this paper matters

The paper that defined “modern NLP” for several years. Cast every classification task as fine-tuning a single pretrained Transformer encoder, set state of the art across the GLUE board, and made pretraining + fine-tuning the default research workflow.

The core idea

Pretrain a Transformer encoder on two unsupervised objectives: masked language modeling (predict randomly masked tokens given bidirectional context) and next-sentence prediction. Then fine-tune on labelled downstream tasks with a small task head.

Method (sketch)

To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).

Key results

To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.

Connections

To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.

Reference

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019. [link]