Deep Contextualized Word Representations

Peters et al. · NAACL 2018 · 15 min read

In “the river bank flooded” and “the bank approved a loan,” the word bank has the same spelling but different surroundings. A single lookup vector gives both occurrences the same starting representation. ELMo instead extracts several representations from a pretrained language model, then lets a downstream task learn how to combine those layers.

This companion follows one question: how can the same learned mixture produce different vectors for different uses of the same word? We will trace the layer calculation, differentiate it, take one learning step, and then separate that teaching example from the original experiments. Familiarity with vectors, softmax and a recurrent hidden state is enough; the sequence-to-sequence companion supplies recurrent-model context.

Separate the word type from this occurrence

A word type is the vocabulary item; a token occurrence is its appearance at a particular position in a particular input. A lookup vector can be a useful starting point even though it cannot distinguish the two sentences by itself. The context must enter somewhere after that lookup.

ELMo’s contribution was to expose multiple levels of a pretrained contextual model to downstream tasks. It did not introduce contextual representations from nothing: the paper discusses TagLM, context2vec and CoVe as related predecessors. Its experiments support the value of its particular pretrained features and integrations, rather than a universal claim that every contextual model beats every static representation. Paper §2.

Two directional language models supply the features

For tokens t₁ through tₙ, a forward language model predicts each token from the preceding tokens. A backward model predicts it from the following tokens. Their training loss can be written as

Boundary symbols supply the missing context at either end. The forward hidden state after consuming tₖ can predict tₖ₊₁; the backward state after consuming tₖ can predict tₖ₋₁. That one-position shift matters: using a state that already consumed the answer would make the training problem artificially easy.

The original model uses character-based token inputs and separate two-layer LSTM stacks for the two directions. The token encoder and prediction softmax are shared; the recurrent parameters are separate. A contextual layer is extracted by concatenating the two directional states at the same position. Paper §3.1–3.4.

In the released original model, each projected direction has width 512. Concatenation gives 1024 coordinates. The implementation duplicates the 512-coordinate token input to match that width; this adds no new information but makes layerwise addition defined. After removing boundary tokens, its returned array has dimensions batch × 3 layers × tokens × 1024. These are the original model’s dimensions, not requirements of every scalar mixture. Original implementation and shape conventions.

A softmax over layers, shared across tokens

At position k, let hₖ,₀ be the token layer and hₖ,₁ and hₖ,₂ the contextual layers. Each has d coordinates. For one task-specific mixture, learn three unrestricted scalar logits aⱼ and a scalar γ:

The softmax denominator runs over layers. All coordinates of a layer get the same scalar weight. The weights also stay the same for the two occurrences of bank: they belong to this task mixture, not to one token’s spelling or context. A model can use more than one mixture at different insertion points. The ScalarMix definition implements this weighted sum, with optional layer normalization.

With finite logits, every sⱼ is positive and the weights sum to one. Before γ, each output is inside the convex hull of its three layer vectors. Multiplying by γ changes that constraint. γ is a learned scalar, not a probability; it is not generally restricted to [0, 1] or even to positive values. The controls below use positive values only to make its scaling effect easy to inspect.

There is no token-to-token attention matrix in this calculation. Context has already entered through the recurrent states. A large s₂ says that this mixture gives the upper layer a large coefficient; it does not identify which neighboring word caused a prediction. Vector magnitudes, directions and the downstream model all affect the result.

Compute both meanings with the same weights

Use the invented two-coordinate features below. They isolate the mixture operation; they are not measured ELMo embeddings, and their axes have no assigned linguistic meaning. Both occurrences share h₀ = [1, 0]. Their h₁ and h₂ differ because they stand in for contextual features. Layer normalization, dropout and regularization are omitted.

Choose a = [log 0.2, log 0.3, log 0.5], so softmax(a) = [0.2, 0.3, 0.5], and start with γ = 1. For the river occurrence:

For the loan occurrence, the same three weights produce

Positive weights do not make every coordinate positive: the loan occurrence’s upper layer contributes −0.5 to coordinate 1. With γ = 2, the outputs become [3.4, 2.2] and [0.6, 2.0]. The layer weights have not changed.

Compare equal weights with lower-layer emphasis. For the latter, the river mixture becomes [0.8, 1.4] and the loan mixture [1.2, 0.4]. The feature extractor has stayed frozen; only the task’s use of its features has changed. This is a useful way to distinguish “context-dependent representation” from “context-dependent mixture weights.”

Follow the gradient into the mixture

To show what gets learned, attach a deliberately simple fixed head w = [1, −1]. It gives score zₖ = wᵀeₖ and probability σ(zₖ), where σ is the logistic sigmoid. Assign toy label 1 to the river occurrence and 0 to the loan occurrence. The mean binary log loss over these two examples is

These labels teach the derivative; they are not a word-sense benchmark. At the default weights and γ = 1, scores are 0.6 and −0.7, probabilities are 0.645656 and 0.331812, and mean loss is 0.420337. A positive score favors the assigned river label; a negative score favors the assigned loan label.

Differentiate the softmax and collect its terms. For a single occurrence, changing aⱼ changes all weights, giving

The subtraction of mₖ is essential. Increasing one logit redistributes a fixed total weight instead of adding an independent unnormalized coefficient. Because ∂ℒ/∂zₖ = (σ(zₖ) − yₖ)/2, the shared logit derivative sums contributions from both occurrences:

At the default setting, the three derivatives are approximately [0.042234, 0.272578, −0.314812], and the γ derivative is −0.222437. Their logit sum is zero, consistent with softmax being unchanged when the same constant is added to every logit. The river and loan contributions are shown separately so the global sharing is visible.

Use a′ = a − 0.1∇ₐℒ and γ′ = γ − 0.1∂ℒ/∂γ, then recompute softmax(a′). Subtracting the gradient directly from s would be a different update and would not automatically preserve valid softmax weights. The new loss is 0.398055. A small successful step here does not guarantee that every learning rate or dataset improves.

Increasing γ alone is also not universally helpful. With lower-layer emphasis, this toy head gives both examples the wrong sign. Doubling γ makes those wrong predictions more confident and raises their loss. A learned scale can respond to task supervision; simply making vectors longer has no general predictive benefit.

Freeze the feature extractor; learn how the task uses it

The original supervised setup adds ELMo features to an existing task model while holding the pretrained language model fixed. It can attach separate mixtures at input and output positions. Domain adaptation of the language model is a separate stage; “frozen for supervised task training” does not imply that its weights could never have been adapted earlier. Paper §3.3–3.4 and supplement §A.1.

The AllenNLP Elmo interface makes these choices explicit: the number of output representations determines how many mixtures are used, and requires_grad controls updates to the language-model weights. A reproducible description should state these choices rather than calling every adaptation “fine-tuning.”

For a sequence-labeling task, the task model could concatenate a conventional token embedding with eₖ and feed that combined vector into its own contextual encoder and tag predictor. The mixture coefficients, scale and task model receive the supervised gradient. In the toy calculation, we froze the head as well so the four mixture parameters could be inspected in isolation. That extra restriction belongs to the teaching example.

Three practical consequences follow. First, a cached context-independent token vector can be reused by spelling; a cached contextual vector needs the exact surrounding input and position. Second, if you cache all three frozen layer outputs, you can still learn a new mixture; caching only a final mixture discards that flexibility. Third, padding and sentence-boundary positions must be handled consistently so they are not treated as labeled tokens. The original repository describes the separate caching options and shapes.

Read the original experiments at their actual scope

These gains compare complete task systems. They do not isolate only the scalar mixture or establish that a layer has one universal linguistic role. Reported ± values describe variation across five seeds, not confidence intervals for all future datasets. Original Table 1 and §4.

Combining layers improves these reported runs over selecting only the final layer. λ regularizes the unnormalized mixture logits: stronger regularization encourages weights toward uniformity, without fixing them to exact equality. The top-layer-only condition also learns γ; it is not the same as an unscaled raw top-layer vector. Table 2 and original mixture code.

What to carry forward to later models

The reusable idea is to separate feature extraction, the choice of representation layers, and the task objective. In the worked example, changing a layer weight changed a representation even though its context and pretrained features stayed fixed. Updating the feature extractor itself would add another source of change and another set of trainable parameters.

Compared with the Transformer companion, the mechanism for communicating across positions is different: a directional recurrent state follows a sequence, while attention explicitly forms permitted query–key comparisons. ELMo’s scalar mixture acts across layers at one position; it should not be read as an attention map across token positions. The BERT paper is the next reading for a different pretraining and task-adaptation design.

Reproduce the example

The standard-library Python script exports all eight states, checks every analytic gradient with central differences, and verifies that each displayed 0.1 update reduces this example’s loss. It requires no model download. The browser implementation computes the same quantities directly.

python3 elmo_scalar_mix_demo.py
# Default outputs: river [1.7, 1.1]; loan [0.3, 1.0]
# Mean loss: 0.4203369997
# After one simultaneous step: 0.3980548517
# Maximum finite-difference error: about 2.1e-11