Chapter 9
Masked Language Models: Learning from Both Sides
Chapter: Masked Language Models, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Three language-model architectures p. 1
- Causal, masked, and encoder-decoder, and which positions each one may read p. 1
- Bidirectional encoders p. 2
- Masking words p. 5
- Next-sentence prediction p. 7
- Training regimes and which positions enter the loss p. 8
- Classification fine-tuning p. 9
- One sequence p. 10
- A pair of sequences p. 10
- Sequence labeling p. 12
- Named entities p. 12
- BIO tags p. 13
- Applying a label at each position p. 13
- How named-entity recognition is scored p. 15
FAQ
How is a masked model different from the causal model in Chapter 7?
A masked encoder can use tokens on both sides of a position. Its loss is on selected corrupted positions, not on every next token.
If you stop scoring an easy selected token, why can the mean loss rise?
The mean is only over selected targets. Removing an easy one can raise the average even though no remaining probability got worse.
What is a BIO tag?
A span encoding: beginning, inside, or outside an entity. It lets a per-token classifier mark multi-token names.
Is next-sentence prediction the same objective as masked token prediction?
No. It is a second signal about whether two spans followed each other. The masked-token loss still only scores the corrupted positions selected for that loss.