Chapter 16
Automatic Speech Recognition
Chapter: Automatic Speech Recognition, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- The recognition task p. 2
- Turn a waveform into a word sequence p. 2
- What the model is scored on, before the error metric p. 2
- Convolutional encoders p. 4
- Input and convolutional layers p. 9
- Inference p. 10
- Learning p. 12
- Encoder-decoder ASR p. 8
- How the acoustic encoder connects to a text decoder p. 8
- HuBERT p. 13
- The forward pass p. 13
- Learning from clustered targets p. 14
- K-means clustering of audio units p. 16
- CTC p. 17
- Inference with blanks p. 18
- Training without a frame-level transcript p. 20
- Combining CTC with an encoder-decoder p. 20
- RNN-T as a streaming alternative p. 21
- Evaluation p. 21
- Word error rate as word-level edit distance p. 21
FAQ
Is word error rate a judgment of meaning?
No. It counts word insertions, deletions, and substitutions against a reference transcript, then divides by the number of reference words. A wrong name and a wrong function word can cost the same unless you inspect the errors.
What problem does CTC solve?
A recording has many more frames than words, and the frame alignment is unknown. CTC allows blank steps and repeated labels so the model can be trained without a hand-marked alignment.
Is HuBERT ordinary next-word supervision?
No. It predicts discovered audio units, produced by clustering, from masked audio. That is separate from training a decoder to emit words.
Why does the chapter separate RNN-T from a full-utterance encoder-decoder?
A streaming model has to produce text before the utterance is finished. An encoder that waits for the whole recording can use future audio that a streaming system does not yet have.