Chapter 2
Words and Tokens: How Text Becomes Model Input
Chapter: Words and Tokens, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Words p. 2
- Types and instances p. 2
- Why repeating text does not double the vocabulary p. 4
- Morphemes p. 5
- Parts of words and why a whitespace split is not a linguistic analysis p. 5
- Unicode p. 7
- Code points p. 8
- UTF-8 p. 8
- Why normalization can change a later string comparison p. 8
- Byte-pair encoding pp. 10-13
- Training merge rules p. 11
- Applying those rules to new text p. 13
- Practical effects, including shorter tokens in English than in many other languages p. 13
- Corpora p. 15
- What a corpus is a sample of, and what it leaves out p. 15
- Regular expressions pp. 16-23
- Character classes, counts, anchors, and grouping p. 17
- Substitution and capture groups p. 21
- Lookahead p. 23
- Patterns used before BPE p. 23
- Tokenization tools and rules pp. 24-27
- Simple Unix tools p. 24
- Rule-based tokenization p. 26
- Sentence segmentation p. 27
- Minimum edit distance p. 28
- Insert, delete, and substitute p. 28
- The dynamic-programming algorithm p. 29
- How the string representation changes the distance p. 28
FAQ
Does duplicating a sentence double its vocabulary?
No. The instance count doubles. The type count stays the same if no new word appears. Type-to-instance ratios change with corpus length.
What is the difference between BPE training and BPE encoding?
Training learns an ordered list of merges from a corpus. Encoding applies that fixed list to new text. An unseen word can still be represented if its pieces are in the learned vocabulary.
Why can the same sentence cost more tokens in one language than another?
A tokenizer trained mostly on English often keeps English words close to one token and splits other languages into smaller pieces. Token count is not a neutral measure of length.
Can Unicode normalization change edit distance?
Yes. Two strings that look like the same word can use different code points. Normalize both before comparing if you mean the linguistic form rather than the raw encoding.