NLP — Recommended Papers
An opinionated reading list for anyone working through the CS336 curriculum (or just trying to understand how modern language models came to be). The papers are listed chronologically; click any title for the (currently stub) per-paper page.
The list
-
Prediction and Entropy of Printed English
This is the paper that invented the language model. Shannon framed English as a stochastic source, defined its entropy, and ran human-prediction experiments to estimate it. Every probabilistic language model since — from n-grams to GPT — is doing a version of what Shannon proposed here.
-
A Neural Probabilistic Language Model
The first neural language model. Before this paper, language models were n-grams with smoothing. Bengio et al. proposed learning continuous word embeddings and feeding them through an MLP to predict the next word — the recipe every modern LM is a descendant of.
-
Sequence to Sequence Learning with Neural Networks
Established the encoder-decoder paradigm that dominates modern conditional generation: encode the input into a fixed-length vector with one RNN, decode the output with another. Even though the Transformer eventually replaced the LSTM backbone, the encoder/decoder framing survived.
-
Neural Machine Translation by Jointly Learning to Align and Translate
Introduced attention — the mechanism the entire Transformer architecture is built from. The single-vector bottleneck of vanilla seq2seq vanishes once the decoder can soft-attend to every encoder state at every step.
-
Neural Machine Translation of Rare Words with Subword Units
The paper that brought byte-pair encoding to NLP. BPE remains the tokenizer of choice for GPT-2/3/4, Llama, Mistral, and most modern frontier LMs. The CS336 lecture's tokenization section is largely a tour of this idea.
-
Attention Is All You Need
The paper of the era. Showed that you can drop the recurrence entirely and rely on self-attention + feedforward blocks stacked deep. Faster to train (parallelism over time steps), scales better, and powers essentially every frontier LM in 2026.
-
Deep Contextualized Word Representations
ELMo proved the “pretrain a deep language model, then plug it into downstream tasks” recipe before BERT made it canonical. The first crisp demonstration that contextual word embeddings dominate static ones (word2vec, GloVe).
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
The paper that defined “modern NLP” for several years. Cast every classification task as fine-tuning a single pretrained Transformer encoder, set state of the art across the GLUE board, and made pretraining + fine-tuning the default research workflow.
-
Language Models are Unsupervised Multitask Learners
The first crisp demonstration that a large enough autoregressive Transformer trained on web text can do zero-shot task transfer — no fine-tuning required. The intellectual bridge from BERT-style pretraining to GPT-3 in-context learning.
-
Language Models are Few-Shot Learners
Discovered in-context learning — that a 175B-parameter autoregressive LM can perform new tasks from a handful of examples in the prompt, with no parameter updates at all. The paradigm shift from fine-tuning to prompting was born here.
-
Scaling Laws for Neural Language Models
Showed that test loss for autoregressive LMs follows clean power laws in model size, dataset size, and compute — over many orders of magnitude. This is the paper that made “just scale it” a defensible engineering plan rather than a hope.
-
Training Compute-Optimal Large Language Models
Corrected Kaplan et al.’s scaling laws and showed that previous frontier models (GPT-3, PaLM) were dramatically under-trained. The headline finding — data should scale roughly in lockstep with parameters — reshaped how every frontier lab budgets compute.