Prediction and Entropy of Printed English
Why this paper matters
This is the paper that invented the language model. Shannon framed English as a stochastic source, defined its entropy, and ran human-prediction experiments to estimate it. Every probabilistic language model since — from n-grams to GPT — is doing a version of what Shannon proposed here.
The core idea
Estimate the entropy of English at the character and word level by asking humans to guess the next character given a prefix, and measuring the distribution of guesses needed. Bounds on the entropy of English fall out of the experiment.
Method (sketch)
To be expanded in the next revision. A careful walkthrough of the model architecture, training objective, and experimental setup will live here. Until then, the reader is pointed at the paper itself (linked above).
Key results
To be expanded. A concise tabulation of the main benchmarks, the headline numbers, and the ablations that matter.
Connections
To be expanded. How this paper relates to the rest of the “Recommended Papers” series and to the CS336 curriculum — especially the lecture sections where it is cited.
Reference
Claude E. Shannon. Prediction and Entropy of Printed English. Bell System Technical Journal, 1950 (published 1951). [link]