Chapter 3
N-gram Language Models: Count, Predict, and Measure Surprise
Chapter: N-gram Language Models, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- N-grams pp. 2-6
- The Markov assumption p. 3
- Estimating conditional probabilities from counts p. 4
- Start and end markers p. 4
- Scale in large count tables p. 6
- Evaluation pp. 7-10
- Training and test splits p. 7
- Perplexity p. 9
- Perplexity as a weighted branching factor p. 10
- The relation between perplexity and entropy p. 18
- Using the model pp. 11-12
- Sampling sentences p. 11
- Generalization versus overfitting the training counts p. 12
- Repairing zero counts pp. 14-17
- Laplace smoothing p. 14
- Add-k smoothing p. 16
- Interpolation p. 17
- Stupid backoff p. 17
- Why a patched probability must still be a distribution p. 14
FAQ
Why is a conditional probability not the unigram count divided by the corpus size?
Conditioning restricts the denominator to continuations of that history. The corpus-wide count answers a different question.
How can an unseen sentence still have positive probability?
The model multiplies local n-gram probabilities. The full sentence may be new even when every counted transition occurred.
Does adding a constant only to zero cells fix the row?
Not by itself. The row must still sum to one. A smoothing rule has to define the whole distribution, then renormalize if needed.
Does lower perplexity mean the system is better?
Only as a prediction score, and only when both systems use the same test text, tokenization, vocabulary, and boundary convention. It does not measure truth or task success.