Chapter 5
Embeddings: Learning Meaning from Neighbors
Chapter: Embeddings, in Daniel Jurafsky and James H. Martin, Speech and Language Processing, 3rd edition draft, August 19, 2026.
Topics to learn
Page numbers refer to the linked chapter PDF.
- Lexical semantics p. 1
- How neighboring contexts suggest related meanings p. 1
- Vector semantics p. 4
- The distributional intuition p. 4
- Count-based vectors p. 6
- Cosine versus dot product p. 8
- Word2vec p. 10
- Skip-gram as a classifier p. 11
- Negative sampling p. 13
- Other static embedding methods p. 16
- What the geometry does and does not show pp. 16-21
- Visualization p. 16
- Semantic regularities and historical change p. 17
- Bias absorbed from text p. 19
- Intrinsic and extrinsic evaluation p. 21
FAQ
When are dot product and cosine the same number?
When both vectors have length 1. Learned embeddings are not automatically unit length, so the two scores can rank neighbors differently.
Why does initializing every embedding to zero block learning?
The vectors stay in a symmetric state: each update is a multiple of a zero vector, so the parameters never move apart.
What does skip-gram with negative sampling train?
It raises the score of observed context words for a center word and lowers the score of sampled non-context words. The samples stand in for the rest of the vocabulary.
Does a nearby vector mean a safe or neutral meaning?
No. The geometry repeats regularities in the training text, including stereotypes. Similarity is not a fairness certificate.