Chapter 17
Text-to-Speech
Chapter: Text-to-Speech, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- The TTS setup p. 2
- What is being generated, and from which inputs p. 2
- Discrete audio tokens p. 3
- Codec encoder and decoder p. 4
- Vector quantization p. 5
- Residual vector quantization p. 6
- Training the EnCodec tokenizer p. 7
- Generation p. 9
- VALL-E as a two-stage language model over text tokens and codec tokens p. 9
- A voice sample as conditioning, not just the text p. 9
- Evaluation and neighboring tasks p. 11
- How synthesized speech is judged p. 11
- Other speech tasks p. 12
- Spoken language models p. 12
FAQ
Why turn audio into discrete tokens before generation?
A language model predicts items from a vocabulary. The codec learns a codebook so a later model can predict audio tokens instead of raw waveform samples.
What does residual vector quantization add?
One codebook is a coarse match. The next codebook quantizes the difference between the input and that match, and the process repeats. One frame can therefore receive several codes.
Is VALL-E only a text-to-audio map?
No. The chapter's setup also conditions on a sample of the target voice. Text is tokenized with BPE. Speech is tokenized by the codec. A language model then generates the audio tokens.
Does a fluent waveform establish that the words were right?
No. TTS evaluation has to separate whether the speech sounds like the intended voice from whether it says the intended words. The chapter treats evaluation as its own step.