Transformer Architectures: Choose the Information Flow

Updated September 2026 · 6 min read

A transformer block updates one vector per token. Stacking the blocks gives a model depth. To understand the architecture, keep that block in mind and change one thing at a time: which tokens can it read?

We will first use a stack to represent a sentence, then use one to generate a continuation, and finally connect the two. The attention and MLP boxes keep the same roles throughout.

Use one task for all three diagrams

Suppose the source is “the cat sleeps” and the target translation is “le chat dort.” An encoder can build source representations using the whole source. A causal decoder can predict target tokens one at a time. An encoder-decoder connects those operations through cross-attention.

A decoder-only model can also perform translation by receiving a prompt containing the source and then generating the target. Thus a task does not uniquely determine a model family. The families describe computation graphs, not mutually exclusive lists of applications.

First, let every token read the sentence

An encoder applies bidirectional self-attention over valid input positions, interleaved with token-wise MLPs. The representation of the middle word can depend on words to both its left and right. A task head can classify the sequence, label each position, or predict a missing token.

Bidirectional context is appropriate when those surrounding tokens are available at inference time. If the task is next-token prediction from a prefix, allowing the representation to see its target would leak the answer. The encoder chapter follows padding, pooling, and the output head in detail.

Then, hide the future to generate a continuation

Keep the same attention-and-MLP pattern, but restrict the attention connections. A position can now use only the prefix it would have seen when generating text. A vocabulary head turns its final vector into a next-token prediction.

A causal decoder allows position $i$ to read positions $j\le i$. Its output at $i$ predicts the next token in a shifted language-model setup. During training, all positions can be evaluated in parallel because their input tokens are known; the mask preserves the prefix dependency.

At generation time, the next input token depends on the previous prediction, so a new token normally requires another forward step. KV caching avoids recomputing earlier layers' K/V entries but does not remove the sequential choice of future tokens. See the decoder chapter.

Finally, connect the two with cross-attention

The target still needs causal self-attention to follow what it has generated. Add a second read, this time from the encoded source. Its queries come from the decoder’s current representations; its keys and values are projections of the encoder’s output. The MLP then transforms the result.

Let the source length be $n$ and target length be $m$. Encoder self-attention is $n\times n$. Decoder causal self-attention is $m\times m$. Decoder cross-attention uses target queries and source keys/values, making an $m\times n$ score matrix. The source can be encoded once and reused for every target generation step.

For target position 2, self-attention reads only the allowed target prefix, while cross-attention may read all valid encoded source positions in ordinary offline translation. Source padding remains masked. Streaming translation needs additional constraints on which source tokens have arrived.

The original Transformer paper uses this encoder-decoder arrangement. Many later language models keep only one side or alter the residual and normalization structure; “Transformer” does not require every model to reproduce that original graph.

Compare the graph before the model name

Architecture and objective are separate choices

BERT combines an encoder with a corruption-and-reconstruction pretraining objective. GPT-2 combines a causal decoder with next-token likelihood. A bidirectional transformer can also parameterize a language diffusion denoiser; that requires an explicit corruption process and denoising/generative construction, beyond the attention mask.

The language diffusion series develops that additional machinery. The transformer chapters here explain the reusable network components. This distinction prevents treating “bidirectional” as synonymous with “diffusion” or “decoder-only” as a statement about every possible training loss.

Inspect complete examples

Continue with BERT, GPT-2, and Vision Transformers, then compare their independent design choices on the architecture map.