Neural Machine Translation of Rare Words with Subword Units

Sennrich, Haddow & Birch · ACL 2016 · 11 min read

A word-level vocabulary gives each known word an ID. What should happen when a new word arrives? Replacing it with one unknown ID loses its spelling. Keeping smaller reusable pieces gives a model a way to represent a word it has never seen whole.

This companion builds a new, deliberately tiny example. You will learn six merges, encode an unseen word with those saved rules, and see why “subword” does not mean “linguistically meaningful.” No neural network is needed to reproduce the tokenizer.

Separate a word’s spelling from its model IDs

Suppose a tokenizer emits b and rains</w> for brains. A model can look up two IDs even if it has never received a single brains ID. It still has to learn useful predictions from the two pieces. Representing a spelling does not establish that the model understands or translates it correctly.

Our input is already split into lowercase words. We append a reserved </w> symbol to each one: rain starts as r a i n </w>. Treat the marker as one symbol, not four punctuation characters. This follows the paper’s original convention; later subword-nmt versions attach it to the final character from the start. The authors document that version difference.

Learn from occurrences, not just distinct spellings

Use five word types with frequencies rain: 6, rains: 3, train: 4, trains: 2 and brain: 1. Together they represent 16 word occurrences. Count adjacent current symbols within each word, weighting each occurrence by its word frequency.

Initially, (a, i) appears once in every word type, so its count is 6 + 3 + 4 + 2 + 1 = 16. The pair (t, r) appears only in train and trains, for a count of 4 + 2 = 6. Counting each distinct spelling once would instead give 5 and 2, changing the learning problem.

Select the most frequent pair, replace its nonoverlapping occurrences with one joined symbol, and recount. In this example, ties go to the lexicographically smallest pair, so (a, i) wins over (i, n) and (r, a). This tie policy is ours; save whichever policy an implementation actually uses. The Hugging Face BPE walkthrough develops the distinction between training the rules and applying them.

Six decisions form an ordered tokenizer

After merging a + i, the pair ai + n has count 16. After that merge, r + ain has count 16. The pieces have changed, so the next count is over those pieces, not the original character pairs.

The corpus goes from 92 boundary-inclusive symbols to 23, a 75% reduction in this toy representation. Each selected occurrence saves one position here. At rank 4, the end marker is adjacent to rain in 11 occurrences, but not in the five plural occurrences, where s lies between them.

Our symbol inventory retains seven letters, the boundary marker and six distinct merged symbols: 14 symbols altogether. Only four different symbols appear in the final segmented training text: rain</w>, rains</w>, t and b. The retained tokenizer inventory and the symbols currently emitted by its training corpus are different sets.

We stop at six merges to keep the process visible. There are still pairs left. A merge budget is a modeling choice, not a claim that every word should become one token. Also, frequent pieces can cross apparent stems: our tokenizer shares rain inside train and brain. Pair frequency did not discover their meanings.

Apply saved rules without learning from the new word

brains did not occur in training, but all its letters did. Start from b r a i n s </w>, then replay the six rules in rank order. Each rule only acts when its two symbols are currently adjacent. Do not recount this new word and choose a fresh most-frequent pair.

The final pieces are b and rains</w>, not brain and s. The latter might look natural to a reader, but our learned rules never constructed brain. Encoding follows the saved rules, not a fresh search for meaningful or longest-looking pieces.

For this ordinary learned sequence, a simple pass through the saved rules produces the trace. An efficient encoder can instead repeatedly select the highest-priority applicable adjacent pair; the priority still comes from training. The authors’ encoder makes that rank lookup explicit.

Keep word boundaries separate from display separators

In our example, rain is a different symbol from rain</w>. The latter can only finish a word. For the two input words rain train, encoding each word separately gives rain</w> | t | rain</w>. The marker tells the decoder where to restore a word boundary; there is no merge across the original space.

The subword-nmt text format commonly prints @@ after a piece that continues a word. In that display, our unseen example is b@@ rains. The visible separator is a serialization convention, not the learned pair (@, @). Keep preprocessing, boundary rules and decoding consistent. The usage documentation shows the output convention.

Our demonstration preserves these pre-split word spellings. It does not preserve arbitrary original whitespace, capitalization removed by earlier normalization, or distinctions already discarded before tokenization. Reversibility must be defined for the complete processing chain.

An unseen word is different from an unseen character

Try rainy: the letter y is absent from our seven-letter alphabet. Joining known symbols cannot manufacture a new base character. The downloadable implementation rejects that input explicitly rather than pretending it has a model ID for it.

Character BPE therefore does not by itself guarantee coverage of every possible string. A byte-level design starts with bytes; retaining all 256 byte values supplies a different fallback guarantee. This does not turn every byte into a complete character or guarantee a model’s output byte sequence is valid text. Hugging Face’s explanation distinguishes the character and byte starting vocabularies.

There is a second coverage question: which tokenizer symbols have entries in the neural model’s vocabulary? A merged intermediate symbol can disappear from the final training segmentation, as rain does here. Keeping only final emitted symbols may leave a new segmentation without a supported ID. Save the model vocabulary policy along with the merge list. The authors’ vocabulary filtering workflow also addresses symbols seen only on the other language side when rules are learned jointly.

A wider merge inventory can shorten familiar spellings, while smaller pieces offer more sharing. Neither token count nor successful round-trip decoding proves downstream quality. Compare a tokenizer on the intended text distribution and task, including unfamiliar names, scripts and spelling variants.

Read the translation results at their actual scope

The paper compares subword systems with WDict, a word-level model using a back-off dictionary. Joint BPE learns rules from both language sides. These are reported eight-model ensemble scores on newstest2015; they are not measurements of our toy tokenizer.

Joint BPE gains 0.5 BLEU for German and 1.3 for Russian over WDict. The German 1.1-point gain belongs to C2-50k, not BPE. German OOV unigram F1 also favors WDict over joint BPE, 36.8 versus 33.6; rare-word F1 favors joint BPE, 41.8 versus 36.8. The comparison depends on the metric and subset. Read Tables 2–3 and §4.2.

Reproduce the tokenizer before connecting a model

The standard-library Python program learns every rule, generates the figure data, and checks training replay, unseen-word decoding, nonoverlapping replacement and unknown-character rejection. Run it from this repository:

python3 blog/nlp/recommended-papers/sennrich_bpe_demo.py

Its learned-rule output is sufficient to reproduce this small example together with the alphabet and boundary convention. A production tokenizer also needs its normalization and splitting settings, exact token-to-ID map, special tokens, byte or character policy, vocabulary restrictions, and implementation version. Reusing only a vocabulary-size number cannot reproduce the token sequence.

References and further reading

• Sennrich, Haddow & Birch (2016) — publication record and original paper.
• Authors’ subword-nmt repository — boundary-version differences, output conventions and vocabulary filtering.
• Hugging Face: byte-pair encoding tokenization — training, ranked application, and byte-level variants.