Chapter 6
Neural Networks: Follow One Prediction Forward and One Gradient Back
Chapter: Neural Networks, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Units and the XOR problem pp. 2-5
- Why one linear threshold cannot solve XOR p. 4
- A hidden layer as the fix p. 5
- Feedforward networks pp. 6-13
- Layers, activations, and vectorized inference p. 6
- Classifiers with hand-built features p. 12
- Embeddings as input instead of hand-built features p. 13
- Training pp. 18-24
- Loss p. 18
- Gradients p. 19
- Computation graphs p. 20
- Backward differentiation p. 21
- Learning-rate and optimization details in the chapter p. 24
FAQ
Why does a stack of affine layers need a nonlinearity?
Without one, the stack multiplies out to a single affine map. A hidden nonlinearity, such as ReLU, lets the response change by region of the input.
Does a zero ReLU gradient mean that unit is useless forever?
No. It means that unit was inactive for this input and these parameters. Another input can turn it on. A unit that never activates during training is a different problem.
What is the computation graph for?
It records the operations that produced the loss so each parameter can collect its local derivative. You do not differentiate the whole network as one opaque formula.
What changes when the classifier reads embeddings instead of counts?
The token's vector becomes a learned parameter. The token ID stays a lookup key. Gradient descent moves the vector, not the vocabulary index.