Appendix J
PPMI
Appendix: PPMI, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026. The posted file is a short note and does not include the usual draft header.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Pointwise mutual information p. 1
- How often two events co-occur compared with independence p. 1
- PMI between a target word and a context word p. 1
- Positive PMI p. 1
- Why negative PMI is unstable unless the corpus is enormous p. 1
- Replacing negative values with zero p. 1
- PPMI as a weight for word-word matrices, not document counts p. 1
FAQ
What does a PMI of zero mean?
The two events co-occur exactly as often as independence would predict. Positive PMI means more than chance. Negative PMI means less.
Why drop negative PMI?
A negative value claims the words co-occur less than chance. Estimating that reliably for rare words needs a huge corpus. PPMI sets those values to zero.
Is PPMI the same as TF-IDF?
No. TF-IDF weights a term in a document. PPMI weights the association between two words in a word-word matrix. The dimensions are different.
Does a high PPMI prove a semantic relation?
No. It is a count association. Frequent named pairs, genres, and errors can score high. The appendix presents it as a weighting function, not a relation labeler.