Deep Learning · Transformers· Position and architecture

Cross-Attention: Queries Read a Different Sequence

Updated September 2026 · 4 min read

Cross-attention changes the source of the keys and values. Queries come from the sequence being updated. Keys and values come from a separate source representation. The weighted-read operation is otherwise the same.

Name the two sequences

Let decoder states be $D\in\mathbb R^{m\times d_D}$ and encoder states be $E\in\mathbb R^{n\times d_E}$. Define $Q=DW_Q$, $K=EW_K$, and $V=EW_V$. The query/key projection output widths must agree so their dot products are defined; the original encoder and decoder widths need not agree.

$$Q\in\mathbb R^{m\times d_k},\quad K\in\mathbb R^{n\times d_k},\quad V\in\mathbb R^{n\times d_v},\quad O=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V.$$

The score matrix is rectangular, $m\times n$. Each output row belongs to a decoder position; each column indexes an encoder position. A common confusion is to expect the output length to equal source length. The query sequence determines the number of output rows.

A source with three facts and two queries

Imagine encoder positions represent three source fragments, and two decoder positions have different needs. If their attention rows are $(0.7,0.2,0.1)$ and $(0.1,0.2,0.7)$, the two outputs mix the same value vectors in different proportions. For scalar values $(10,20,30)$, the outputs are 14 and 26.

This is not a hard alignment that assigns exactly one source word to one target word. Attention can distribute mass across many source positions, and its projections can encode relationships that do not map neatly to word alignments. Multiple heads provide several such reads.

Where it appears in an encoder–decoder block

A standard sequence-to-sequence decoder first uses causal self-attention over target-side inputs, then cross-attends to the encoder output, then applies an MLP, with the chosen normalization and residual arrangement around each sublayer. The first attention collects target-prefix context. The second supplies source information relevant to that context.

For translation, the source sentence is observed completely before generating the target. Cross-attention may therefore read all valid source positions; there is no target-to-source triangular mask. Target self-attention remains causal. The source padding mask still matters.

A different task may require a different source mask

In streaming speech recognition, future audio frames may not yet be available. A cross-attention mask must then reflect which source frames the output is allowed to depend on. Cross-attention is not intrinsically noncausal; its information boundary is determined by the task and source availability.

For multimodal models, visual features can serve as source keys and values while text states provide queries. Other multimodal architectures instead insert projected visual tokens into a common self-attention sequence. Both are valid conditioning designs, and the presence of images does not imply cross-attention must be used.

Why source K/V can often be precomputed

If encoder states remain fixed during target generation, a decoder layer's projected source keys and values remain fixed too. Cache them once per decoder layer. New target queries can read that source cache at each step. This is separate from the growing self-attention cache over previously generated target tokens.

Different decoder layers typically have different source K/V projection weights, so they cannot generally share one projected source cache. If the source changes—for example, new audio frames arrive—the relevant source states and caches must be updated according to the encoder's dependency structure.

A shape-focused reference

def cross_attention(decoder, encoder, Wq, Wk, Wv, allowed):
    q = decoder @ Wq  # [target_length, dk]
    k = encoder @ Wk  # [source_length, dk]
    v = encoder @ Wv  # [source_length, dv]
    return attention(q, k, v, allowed)[0]

This reuses the single-head reference. Its mask shape is $[m,n]$. In a multi-head implementation, project and reshape the two sequences independently because their lengths differ.

Count source reads, not only target tokens

The attention products cost $O(mnd_k+mnd_v)$ before head aggregation. At one-token decoding, $m=1$, so each step still reads across the source length. Precomputing source K/V saves their projections, not the new query's score and weighted-value computations. A long source can therefore remain expensive even when no target-side history is retained.

Try it: Can encoder width be 1024 while decoder width is 768?

Yes. Choose $W_Q$ from 768 to the chosen query/key width and $W_K$ from 1024 to that same width. The value width and final output projection then determine the decoder update width.

Assemble the two streams

Read the encoder–decoder decoder for the complete dataflow. The primary architecture is documented in Attention Is All You Need.