Chapter 14
RNNs and LSTMs: How a Sequence Carries Its Past
Chapter: RNNs and LSTMs, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Recurrent networks p. 1
- Inference through a hidden state p. 2
- Training and backpropagation through time p. 3
- Why one shared weight collects gradients from several positions p. 3
- RNN language models p. 4
- Forward inference p. 5
- Training p. 6
- Weight tying p. 7
- Other sequence tasks p. 8
- Sequence labeling p. 8
- Sequence classification p. 8
- Generation p. 10
- Richer recurrence p. 11
- Stacked RNNs p. 11
- Bidirectional RNNs p. 12
- LSTM gates, the cell, and the exposed state p. 13
- Common RNN architectures p. 17
- Encoder-decoder RNNs p. 17
- Encoding a source and decoding a target p. 17
- Training that pair p. 21
- Attention over encoder states p. 21
FAQ
Why can one recurrent weight learn from several positions?
The same parameter is used at each step. Its gradient is the sum of every path by which those uses affect the loss, including paths through later states.
Does a closed output gate erase an LSTM's memory?
No. A small output gate limits the state exposed at this step. The cell's forget and input updates decide what remains for later steps.
Can a bidirectional encoder feed a left-to-right decoder?
Yes, when the whole source is available before decoding starts. The decoder still must not read future target tokens.
What is weight tying?
Using one matrix both to embed input tokens and to score output tokens, when the dimensions match. It is a parameter-sharing choice, not a change to the recurrence.