2. The Vision Transformer (ViT) Used in CLIP: Pre-Encoder LayerNorm (ln_pre), Bias-Free Patch Conv, QuickGELU & Multimodal Projection

October 2026 · 16 min read

When Radford et al. (2021) adopted Dosovitskiy et al.'s Vision Transformer as the primary image encoder for CLIP (Contrastive Language–Image Pre-training), they did not simply drop a `timm` or `torchvision` supervised ViT into a dual-tower contrastive loop. In Section 2.4 of the CLIP paper ("We follow their implementation closely with only the minor modification of adding an additional layer normalization to the combined patch and position embeddings before the transformer and use a slightly different initialization scheme"), OpenAI introduced six subtle architectural changes that govern how `openai/clip-vit-base-patch32`, `clip-vit-base-patch16`, and `clip-vit-large-patch14` operate today—and why loading a standard supervised ViT checkpoint into a CLIP ViT class (or vice versa) fails unless you account for them.

1. Modification #1 & #2: The Extra Pre-Encoder LayerNorm (ln_pre) and Bias-Free Patch Stem

In the original Dosovitskiy et al. (2020) ViT (covered in Chapter 1), the initial token sequence $\mathbf{z}_0$ is formed by concatenating the learnable [CLS] token with the projected patch embeddings and adding $\mathbf{E}_{\text{pos}}$, then passing $\mathbf{z}_0$ directly into the residual stream of Transformer Block 1:

Notice a subtle asymmetry in Standard ViT: while the branch input to $\operatorname{MSA}_1$ is normalized by $\operatorname{LN}_1(\mathbf{z}_0)$, the identity residual skip path carries the unnormalized sum $\mathbf{z}_0 = [\mathbf{x}_{\text{cls}}; \mathbf{x}_p E + \mathbf{b}] + \mathbf{E}_{\text{pos}}$ all the way through the network! During multimodal contrastive pretraining with a learnable inverse temperature $\tau^{-1} = \exp(t)$ (which reaches $100.0$ in CLIP), drift in the scale of $\operatorname{Conv2d}(\mathbf{I})$ relative to $\mathbf{E}_{\text{pos}}$ destabilizes the residual stream norm in early blocks.

OpenAI fixed this by inserting an explicit Pre-Encoder LayerNorm (`ln_pre`) immediately after adding $\mathbf{E}_{\text{pos}}$ and before entering Block 1:

Why Does CLIP's Patch `conv1` Set `bias=False`?

In `openai/CLIP`'s `VisionTransformer.__init__`, the patch extraction convolution is defined as:

self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)

Why omit the bias vector $\mathbf{b} \in \mathbb{R}^{d_i}$? Look at what happens to each patch token $\tilde{\mathbf{z}}_{0, i} = \mathbf{x}_p^{(i)} E + \mathbf{b} + \mathbf{E}_{\text{pos}, i}$ before `ln_pre`:

1. First, $\mathbf{E}_{\text{pos}, i} \in \mathbb{R}^{d_i}$ is already an unconstrained learnable additive parameter per token position, making a shared additive bias $\mathbf{b} \in \mathbb{R}^{d_i}$ across all patches strictly redundant (a linear reparameterization).
2. Second, `ln_pre` immediately subtracts the per-token channel mean $\mu_i = \frac{1}{d_i}\sum_{c=1}^{d_i} \tilde{z}_{0, i, c}$ and applies its own learnable affine shift $\boldsymbol{\beta}_{\text{pre}} \in \mathbb{R}^{d_i}$.

Importantly, notice that inside Transformer Block 1, $\mathbf{z}_0 = \operatorname{LN}_{\text{pre}}(\tilde{\mathbf{z}}_0)$ is normalized again by Block 1's pre-attention LayerNorm $\operatorname{LN}_1^{(1)}(\mathbf{z}_0)$:

Even though applying $\operatorname{LN}_1^{(1)}$ right after $\operatorname{LN}_{\text{pre}}$ looks like a double LayerNorm on the attention branch, they have separate learnable affine parameters $(\boldsymbol{\gamma}_{\text{pre}}, \boldsymbol{\beta}_{\text{pre}})$ and $(\boldsymbol{\gamma}_1, \boldsymbol{\beta}_1)$—and more importantly, $\operatorname{LN}_{\text{pre}}$ guarantees that the residual stream highway itself starts at unit variance $\mathcal{O}(1)$ before the first residual addition!

2. Modification #3: QuickGELU vs. Exact Gaussian Error Linear Unit

Every MLP block in Standard ViT expands the token width from $d_i \to 4 d_i \to d_i$ using the exact Gaussian Error Linear Unit:

Because CLIP's Transformer implementation was adapted directly from OpenAI's internal GPT codebase (which predated fast fused `erf` kernels on early GPU drivers), CLIP replaces exact $\operatorname{GELU}$ in both the Vision Transformer and the Text Transformer with Hendrycks & Gimpel's (2016) sigmoid approximation, termed QuickGELU:

While the maximum absolute difference $|\operatorname{QuickGELU}(x) - \operatorname{GELU}(x)|$ is only $\approx 0.005$ around $|x| \approx 1.3$, compounding that difference across $L = 12$ or $L = 24$ layers shifts the final unit-norm embedding enough to degrade zero-shot cosine similarity if you accidentally load official OpenAI CLIP weights into a ViT configured with `nn.GELU()` instead of `QuickGELU()`.

3. Modification #4: Zero Dropout & GPT-2 Residual Depth Scaling $(2L)^{-1/2}$

When training a supervised ViT on ImageNet-1k ($1.28\text{M}$ images) or ImageNet-21k ($14\text{M}$ images), the model sees each image dozens of times and requires `dropout = 0.1` (plus stochastic depth / DropPath in DeiT) to prevent overfitting.

In contrast, CLIP trains on WIT-400M ($400\text{ million}$ unique web image–text pairs) for $32$ epochs at minibatch size $N = 32{,}768$. In this massive web-scale regime, underfitting—not overfitting—is the bottleneck, so OpenAI sets all dropout probabilities in CLIP ViT to $0.0$.

Furthermore, to keep the variance of the residual stream bounded across $L$ layers without relying on Post-Norm rescaling, CLIP initializes the output projection matrices of every $\operatorname{MSA}$ block ($W_{\text{out}}^{(\ell)}$) and every $\operatorname{MLP}$ block ($W_2^{(\ell)}$) with GPT-2's residual depth scaling factor $\frac{1}{\sqrt{2L}}$ (since each of the $L$ blocks performs $2$ residual additions):

Because each of the $2L$ sub-layers adds a residual update of variance $\propto \frac{1}{2L}$, the cumulative sum of $2L$ residual branches has total variance $2L \cdot \mathcal{O}\!\left(\frac{1}{2L}\right) = \mathcal{O}(1)$ at initialization regardless of whether $L = 12$ (`ViT-B`) or $L = 24$ (`ViT-L`).

4. Modification #5: ln_post, Bias-Free Linear Projection $W_i \in \mathbb{R}^{d_i \times d_e}$ & $\ell_2$ Normalization

After the $L$-th Pre-Norm Transformer block, both Standard ViT and CLIP ViT slice out token index $0$ (the evolved [CLS] token $\mathbf{z}_L^0 \in \mathbb{R}^{d_i}$, where `x[:, 0, :]` holds the global visual representation) and discard the $N$ spatial patch tokens $\mathbf{z}_L^{1:N}$:

• Standard Supervised ViT Head: Applies LayerNorm and maps $\mathbb{R}^{d_i} \to \mathbb{R}^K$ class logits via a linear classifier with bias ($\mathbf{y} = \operatorname{LN}(\mathbf{z}_L^0) W_{\text{cls}} + \mathbf{b}_{\text{cls}}$), trained with softmax cross-entropy over a fixed vocabulary of $K$ classes.
• CLIP ViT Multimodal Head: Applies post-encoder LayerNorm (`ln_post`), multiplies by a bias-free learnable projection matrix $W_i \in \mathbb{R}^{d_i \times d_e}$ (`self.proj = nn.Parameter(scale * torch.randn(width, output_dim))`), and $\ell_2$-normalizes onto the unit hypersphere $\mathbb{S}^{d_e - 1}$:
  $$\mathbf{h}_I = \operatorname{LN}_{\text{post}}\!\left(\mathbf{z}_L^0\right) \in \mathbb{R}^{d_i}, \qquad \mathbf{v}_I = \mathbf{h}_I W_i \in \mathbb{R}^{d_e}, \qquad \hat{\mathbf{i}} = \frac{\mathbf{v}_I}{\|\mathbf{v}_I\|_2} \in \mathbb{S}^{d_e - 1}$$

Why is $W_i \in \mathbb{R}^{d_i \times d_e}$ a bare parameter matrix without bias? Because $\mathbf{v}_I = \mathbf{h}_I W_i$ is immediately divided by $\|\mathbf{v}_I\|_2$ for cosine similarity with the text embedding $\hat{\mathbf{t}}$. Adding a constant bias vector $\mathbf{b}_i$ before $\ell_2$ normalization would pull all image embeddings toward a fixed point $\mathbf{b}_i / \|\mathbf{b}_i\|_2$ on the hypersphere, worsening representation collapse and exacerbating the modality gap.

5. Modification #6: High-Resolution Adaptation in CLIP ViT-L/14@336px

Following Dosovitskiy et al.'s high-resolution fine-tuning recipe, OpenAI took their largest pretrained Vision Transformer—CLIP ViT-L/14 trained for $32$ epochs at $224 \times 224$—and trained it for one additional contrastive epoch at $336 \times 336$ pixel resolution to produce CLIP ViT-L/14@336px:

That single high-resolution epoch ($576 + 1 = 577$ tokens per image) boosted zero-shot ImageNet top-1 accuracy from $75.5\%$ to $76.2\%$ and improved fine-grained OCR, counting, and small-object tasks across 27 benchmarks—making `CLIP ViT-L/14@336px` the default vision encoder chosen by LLaVA-1.5 and many open-source multimodal LLMs. (Note: when LLaVA feeds visual tokens into an LLM, it intercepts the $N = 576$ spatial patch tokens $\mathbf{z}_{L-1}^{1:576} \in \mathbb{R}^{576 \times 1024}$ from the penultimate layer before [CLS] pooling!)

6. Complete Tensor-Shape Table Across All Official CLIP ViT Models

7. Worked Numerical Trace: Why ln_pre Stabilizes Early Attention Logits

Suppose a patch token after `conv1` and position addition has unnormalized coordinates $\tilde{\mathbf{z}}_{0, 1} = [12.0, -4.0, 8.0, -16.0]^\top \in \mathbb{R}^4$ because a high-contrast image patch summed constructively with $\mathbf{E}_{\text{pos}, 1}$.

• Without `ln_pre` (Standard ViT): Even though $\operatorname{MSA}_1$ normalizes its input to compute bounded attention output $\Delta \mathbf{z}_1 \approx \mathcal{O}(1)$, the identity skip connection adds $\tilde{\mathbf{z}}_{0, 1} = [12, -4, 8, -16]^\top$ directly to $\Delta \mathbf{z}_1$. Thus the output of Block 1 is still dominated by $\tilde{\mathbf{z}}_{0, 1}$ ($\|\tilde{\mathbf{z}}_{0, 1}\|_2 = \sqrt{144 + 16 + 64 + 256} = \sqrt{480} \approx 21.91$), drowning out the learned attention update of Layer 1.
• With `ln_pre` (CLIP ViT): Before Block 1, `ln_pre` computes:
  $$\mu = \frac{12 - 4 + 8 - 16}{4} = 0.0, \qquad \sigma^2 = \frac{144 + 16 + 64 + 256}{4} = 120.0, \qquad \sigma \approx 10.954$$
  $$\mathbf{z}_{0, 1} = \operatorname{LN}_{\text{pre}}(\tilde{\mathbf{z}}_{0, 1}) = \left[\frac{12}{10.954}, \frac{-4}{10.954}, \frac{8}{10.954}, \frac{-16}{10.954}\right]^\top \approx [1.095, -0.365, 0.730, -1.461]^\top$$
  Now $\|\mathbf{z}_{0, 1}\|_2 = \sqrt{4} = 2.0$, matching the scale of the residual updates $\Delta \mathbf{z}_\ell$ scaled by $(2L)^{-1/2}$!

8. Exact PyTorch Implementation of OpenAI's CLIP Vision Transformer

Below is the faithful PyTorch implementation matching `openai/CLIP`'s `VisionTransformer` module line for line, including `conv1(bias=False)`, `ln_pre`, `QuickGELU`, `(2L)**-0.5` residual initialization, `ln_post`, and the bias-free multimodal projection parameter `proj`:

from collections import OrderedDict
import torch
import torch.nn as nn
import torch.nn.functional as F

class QuickGELU(nn.Module):
    """OpenAI's fast sigmoid GELU approximation used across CLIP."""
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return x * torch.sigmoid(1.702 * x)

class ResidualAttentionBlock(nn.Module):
    def __init__(self, d_model: int, n_head: int):
        super().__init__()
        self.attn = nn.MultiheadAttention(d_model, n_head, batch_first=True)
        self.ln_1 = nn.LayerNorm(d_model)
        self.mlp = nn.Sequential(OrderedDict([
            ("c_fc", nn.Linear(d_model, d_model * 4)),
            ("gelu", QuickGELU()),
            ("c_proj", nn.Linear(d_model * 4, d_model)),
        ]))
        self.ln_2 = nn.LayerNorm(d_model)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Bidirectional Pre-Norm self-attention (no causal mask in Vision tower)
        ln1_x = self.ln_1(x)
        x = x + self.attn(ln1_x, ln1_x, ln1_x, need_weights=False)[0]
        x = x + self.mlp(self.ln_2(x))
        return x

class CLIPVisionTransformer(nn.Module):
    """
    Exact OpenAI CLIP Vision Transformer (Radford et al., ICML 2021).
    Highlights all 6 modifications relative to Dosovitskiy et al. (2020).
    """
    def __init__(
        self,
        input_resolution: int = 224,
        patch_size: int = 16,
        width: int = 768,
        layers: int = 12,
        heads: int = 12,
        output_dim: int = 512,
    ):
        super().__init__()
        self.input_resolution = input_resolution
        self.output_dim = output_dim
        num_patches = (input_resolution // patch_size) ** 2

        # Modification 1: Bias-free stride-P patch extraction stem
        self.conv1 = nn.Conv2d(
            in_channels=3, out_channels=width,
            kernel_size=patch_size, stride=patch_size, bias=False
        )

        scale = width ** -0.5
        self.class_embedding = nn.Parameter(scale * torch.randn(width))
        self.positional_embedding = nn.Parameter(scale * torch.randn(num_patches + 1, width))

        # Modification 2: Extra Pre-Encoder LayerNorm (ln_pre) before Block 1
        self.ln_pre = nn.LayerNorm(width)

        # Modification 3 & 4: Pre-Norm blocks with QuickGELU and 0.0 dropout
        self.blocks = nn.Sequential(*[
            ResidualAttentionBlock(width, heads) for _ in range(layers)
        ])

        # Modification 5: Post-Encoder LayerNorm + bias-free multimodal projection W_i
        self.ln_post = nn.LayerNorm(width)
        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))

        self._init_parameters(width, layers)

    def _init_parameters(self, width: int, layers: int) -> None:
        # GPT-2 style residual depth scaling: (2 * L)^(-0.5) on output projections
        proj_std = (width ** -0.5) * ((2 * layers) ** -0.5)
        attn_std = width ** -0.5
        fc_std = (2 * width) ** -0.5
        for block in self.blocks:
            nn.init.normal_(block.attn.in_proj_weight, std=attn_std)
            nn.init.normal_(block.attn.out_proj.weight, std=proj_std)
            nn.init.normal_(block.mlp.c_fc.weight, std=fc_std)
            nn.init.normal_(block.mlp.c_proj.weight, std=proj_std)

    def forward(self, x: torch.Tensor, normalize: bool = True) -> torch.Tensor:
        # [B, 3, H, W] -> [B, width, grid, grid] -> [B, width, N] -> [B, N, width]
        x = self.conv1(x)
        x = x.reshape(x.shape[0], x.shape[1], -1).permute(0, 2, 1)

        # Prepend learnable [CLS] token: [B, 1, width] + [B, N, width] -> [B, N+1, width]
        cls_token = self.class_embedding.to(x.dtype) + torch.zeros(
            x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device
        )
        x = torch.cat([cls_token, x], dim=1)

        # Add learnable positional embeddings and apply Pre-Encoder LayerNorm (ln_pre)
        x = x + self.positional_embedding.to(x.dtype)
        x = self.ln_pre(x)

        # Pass through L Pre-Norm Transformer blocks
        x = self.blocks(x)

        # Slice [CLS] token at index 0 and apply ln_post
        x = self.ln_post(x[:, 0, :])

        # Project from d_i (width) to shared multimodal embedding space d_e (output_dim)
        if self.proj is not None:
            x = x @ self.proj

        return F.normalize(x, dim=-1) if normalize else x