Chapter 7
Transformers and Pretraining: From Attention Weights to the Next Token
Chapter: Transformers and Pretraining, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Attention p. 3
- Queries, keys, and values p. 4
- Scaled scores and softmax p. 5
- Causal masking p. 8
- A transformer block p. 8
- Attention plus a feedforward layer p. 10
- Residual connections and layer norm p. 10
- The same computation packed into one matrix p. 12
- Inputs and outputs pp. 15-19
- Token and position embeddings p. 15
- The language-modeling head p. 17
- Shifted next-token targets p. 19
- Decoding p. 20
- Greedy decoding p. 20
- Random sampling p. 21
- Temperature p. 22
- Top-k and top-p p. 24
- Pretraining and scale pp. 25-28
- Next-token pretraining p. 25
- Perplexity p. 26
- Model, data, and compute numbers reported in the chapter p. 28
FAQ
Why is replacing a future attention score with zero not a mask?
Softmax turns a score of zero into a positive weight. A forbidden position needs probability zero after normalization, usually from a score of negative infinity before the softmax.
Does masking only the last layer prevent future leakage?
No. An earlier layer can already have copied a future token into an allowed position. Every attention layer that could move that information has to respect the same boundary.
Do temperature, top-k, or top-p change the trained model?
No. They change which tokens the sampler can draw from the current distribution. The weights stay fixed.
What is the pretraining target in this chapter?
The next token. The current token is input. The following token is the label. Predicting the unshifted token while also attending to it would be a different, easier task.