BERT: Learn Bidirectional Representations by Reconstructing Tokens
Why the next-token setup cannot simply lose its causal mask
In a causal language model, the representation at position $i$ predicts the token at $i+1$. If we remove the causal mask while leaving the next-token training setup unchanged, the representation can read that target token directly. A lower loss could then reflect leakage rather than learning to predict unavailable information.
BERT instead selects token positions as reconstruction targets and modifies their inputs according to a corruption policy. For “the cat sat on the mat,” hiding “cat” permits the encoder to use both “the” and “sat on the mat” to infer the missing word.
The original model and input format
The BERT paper describes a bidirectional Transformer encoder using token, learned position, and segment embeddings. The original BERT-Base configuration has 12 layers, width 768, and 12 heads. Its pretraining combines masked language modeling with next-sentence prediction. The released code is the reference for exact preprocessing and checkpoint behavior.
For paired text, special separator tokens and segment IDs indicate the input format. Segment embeddings are a learned distinction between input portions; they are not separate vocabularies or a guarantee that the model understands every semantic relation between sentences.
Selection and replacement are separate random choices
In the original policy, 15% of token positions are selected for prediction. Of those selected positions, 80% are replaced by [MASK], 10% by a random token, and 10% remain unchanged. The objective is evaluated on the selected positions, including those left unchanged.
For an illustrative group of 1000 eligible positions, the expected counts are 150 selected: 120 masked, 15 random replacements, and 15 unchanged selected targets. These are expectations under the policy, not fixed counts in each batch. The remaining 850 positions are not prediction targets for that masking draw.
Leaving some selected tokens unchanged is different from training every unchanged token as a target. Keep the selection mask separate from the replacement result; otherwise the loss denominator and task change.
Track exactly what is conditioned on
Let $x$ be the clean sequence, $\tilde x$ the corrupted input, and $\mathcal M$ the selected target positions. A simplified per-example objective is
The prediction head receives contextual encoder states and produces vocabulary logits. If several tokens are masked, they are predicted from the same corrupted input. A single forward pass does not condition the prediction of one mask on the newly sampled value of another mask unless an additional iterative procedure is defined.
Suppose the sentence has two masked positions and the model outputs marginals that favor individually plausible but mutually inconsistent choices. Independent sampling from those marginals can produce an awkward combination. Contextual encodings can encourage compatibility through observed context, but multiplying token marginals does not automatically represent every dependency among jointly missing tokens.
Fine-tuning changes what the representations are asked to do
A classifier can read the designated sequence representation. Token tagging applies a head at each position. Extractive question answering can predict start and end positions in the provided passage. These heads use a shared encoder backbone but define different output spaces and losses.
Fine-tuning generally updates the chosen parameters to the downstream objective. Freezing the encoder and training only a head is another possible procedure, with different capacity to adapt. State which one is used when comparing results.
Masked prediction is not yet a complete diffusion sampler
BERT supplies conditional reconstruction predictions under its corruption training. It does not by itself specify the normalized left-to-right joint distribution used by GPT, nor a reverse diffusion process from an all-noise prior. One can construct additional generation procedures using a masked model, but their transitions and consistency need separate justification.
This is exactly where the language diffusion series begins: a corruption schedule, training weighting, and sampling rule turn denoising predictions into a specified generative construction. Reusing a bidirectional transformer backbone does not make the two training setups identical.
Three implementation checks
Confirm that changing right context can change a target prediction. Confirm that loss is computed on selected targets rather than all visible positions. Confirm that special-token handling and segment IDs match the checkpoint's expected format. These tests target the defining information flow before comparing downstream accuracy.
Try it: If a selected position happens to keep its original token, is it excluded from the original MLM loss?
No. Selection determines the prediction target set. The corruption branch determines what the input looks like. The unchanged subset still contributes to the loss.
Compare objectives without conflating backbones
Read GPT-2 for causal likelihood and the encoder guide for pooling and masking details.