Appendix C
Kneser-Ney Smoothing
Appendix: Kneser-Ney Smoothing, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Absolute discounting p. 1
- Why a fixed discount can match held-out counts better than a proportional cut p. 1
- Saving mass for unseen n-grams p. 1
- Kneser-Ney p. 2
- Interpolated Kneser-Ney p. 2
- The continuation count that replaces a raw unigram for lower-order context p. 2
FAQ
What is being discounted?
A fixed amount is subtracted from observed n-gram counts. The saved mass is what smoothing can give to unseen n-grams. The discount is not a new corpus.
Why not use the raw unigram as the backoff probability?
A word can be common but only after a few histories. Kneser-Ney estimates the lower-order probability from the number of distinct histories a word continues, not only from how often the word occurs.
Does interpolation still use the higher-order count?
Yes. Interpolation mixes the discounted higher-order estimate with the lower-order distribution. It does not throw away an observed n-gram.
Is a lower perplexity on training data evidence the discount worked?
No. The discount is judged on held-out text. Training counts are what the discount is correcting.