Chapter 8
Post-training: Teaching Models to Answer, Prefer, and Verify
Chapter: Post-training, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Instruction tuning p. 2
- Instructions as training data p. 3
- Which tokens receive the loss p. 3
- How instruction-tuned models are evaluated p. 5
- Parameter-efficient fine-tuning p. 5
- Training a small update, including LoRA, instead of every pretrained weight p. 6
- Preferences p. 7
- Chosen and rejected responses p. 8
- Reward or preference models p. 10
- Scoring a pair p. 11
- Alignment from preferences p. 11
- PPO with preference feedback p. 13
- Direct preference optimization p. 14
- Evaluation p. 15
- Limits of preference training, including that a preferred answer need not be true p. 15
FAQ
Does masking the prompt loss hide the prompt from the model?
No. The loss mask chooses which tokens are supervised. The attention mask chooses which context a prediction may read. Response-only training normally leaves the prompt visible.
Does DPO require a separately trained reward model?
Standard offline DPO needs preference pairs and a reference policy. It does not need a second model that scores rewards at training time. Someone still had to judge the pairs.
Does a lower preference loss mean the answer became true?
No. It means the policy fit that preference pair better, relative to the reference. Truth has to be checked with evidence for the task.
What does parameter-efficient fine-tuning change?
A small added set of weights, rather than every pretrained parameter. The base model still determines most of the computation.