Deep Learning · Popular Videos · Umar Jamil· Part 1 · Chapter 2

Model Parameters & Training FLOPs

September 2026 · 24 min read · Distributed Training from First Principles

Before you parallelize anything, learn to count. Parameter count sets your memory bill, FLOP count sets your time bill, and the ratio between them decides which hardware limit you hit first. Almost every design decision later in the series is a trade between these two numbers.

Anatomy of a decoder-only block

Fix the architecture so the counting is unambiguous. A modern pre-norm decoder block, of the kind used by the Llama and Qwen families, contains:

A normalization layer (RMSNorm), then self-attention with four projections $W_Q,W_K,W_V,W_O$.

A residual add, a second normalization, then a gated feed-forward network with three matrices $W_{\text{gate}},W_{\text{up}},W_{\text{down}}$, and another residual add.

No biases, since gated FFNs and RMSNorm make them redundant in practice. With hidden size $d$, $h$ query heads of size $d_h=d/h$, and feed-forward width $d_{\text{ff}}$:

Component	Shape	Parameters
$W_Q$	$d\times h\,d_h$	$d^2$
$W_K$, $W_V$	$d\times h_{kv}\,d_h$ each	$2d^2\cdot\dfrac{h_{kv}}{h}$
$W_O$	$h\,d_h\times d$	$d^2$
$W_{\text{gate}}$, $W_{\text{up}}$	$d\times d_{\text{ff}}$ each	$2\,d\,d_{\text{ff}}$
$W_{\text{down}}$	$d_{\text{ff}}\times d$	$d\,d_{\text{ff}}$
2 RMSNorm gains	$d$ each	$2d$

With plain multi-head attention ($h_{kv}=h$) the block total is

$$N_{\text{block}}=4d^2+3\,d\,d_{\text{ff}}+2d.$$

Outside the stack there is a token embedding table $V\times d$, a final RMSNorm of size $d$, and an output projection $V\times d$ which is often tied to the embedding.

$$N_{\text{total}}=L\big(4d^2+3\,d\,d_{\text{ff}}+2d\big)+d+\underbrace{V d}_{\text{embedding}}+\underbrace{V d}_{\text{head, if untied}}.$$

Why the 12Ld² rule keeps showing up

Two conventions happen to agree. The classic transformer uses an ungated FFN with $d_{\text{ff}}=4d$ and two matrices, giving $2\cdot d\cdot 4d=8d^2$. A gated FFN uses three matrices but shrinks the width to $d_{\text{ff}}=\tfrac83 d$ to keep the parameter count identical: $3\cdot d\cdot\tfrac83 d=8d^2$. Either way,

$$N_{\text{block}}\approx 4d^2+8d^2=12d^2,\qquad\boxed{N\approx 12Ld^2}$$

for the non-embedding parameters. This is the number to keep in your head. It also explains why the "$\tfrac83 d$ rounded to a multiple of 256" widths in open-weight models look so arbitrary: they are chosen to hit a parameter budget, not a mathematical constant.

Embeddings are not part of $N$. When people quote a "7B model" they usually mean total parameters, but when they use $N$ in a scaling law or a FLOP estimate they usually mean non-embedding parameters. For a small model with a 128k vocabulary the embedding table can be a third of the total, so mixing the two conventions produces badly wrong estimates.

Forward FLOPs: two per parameter per token

Every weight matrix is used in a matrix multiplication. Multiplying an $m\times k$ activation by a $k\times n$ weight costs $m\,k\,n$ multiply-accumulate operations, and the convention is to count a multiply-accumulate as two floating-point operations.

For a single token, $m=1$, so the cost is $k\,n$ MACs, which is exactly the number of entries in that weight matrix. Therefore:

Each weight is touched once per token, and costs 2 FLOPs. The forward pass over one token costs $2N$ FLOPs, where $N$ counts the parameters that participate in matmuls.

The embedding lookup is a gather, not a matmul, so it contributes essentially nothing. The output head is a matmul, so it contributes $2Vd$ per token and should be counted separately when the vocabulary is large.

Backward FLOPs: twice the forward

This is the step people memorize rather than derive, so it is worth doing properly. Consider one linear layer $Y=XW$ with $X\in\mathbb R^{m\times k}$ and $W\in\mathbb R^{k\times n}$. Backpropagation receives $\bar Y=\partial\mathcal L/\partial Y$ and must produce two things:

The gradient with respect to the input, needed to keep propagating backwards:

$$\bar X=\bar Y\,W^\top\in\mathbb R^{m\times k},\qquad m\,n\,k\ \text{MACs}.$$

The gradient with respect to the weights, needed by the optimizer:

$$\bar W=X^\top\bar Y\in\mathbb R^{k\times n},\qquad m\,k\,n\ \text{MACs}.$$

Both have exactly the same cost as the forward matmul $XW$. So the backward pass costs two forward passes, and

$$\boxed{C_{\text{fwd}}=2N,\qquad C_{\text{bwd}}=4N,\qquad C_{\text{train}}\approx 6N\ \text{FLOPs per token}.}$$

Over a dataset of $D$ tokens, the total training cost is the famous

$$C\approx 6ND.$$

The first layer is cheaper

The very first layer does not need $\bar X$, because nothing precedes it. Saving one third of one layer out of $L$ is a rounding error, which is why the rule is stated as $6N$ flat.

Recomputation costs more

Activation checkpointing re-runs the forward pass inside the backward pass. That turns $6N$ into roughly $8N$ per token: a 33% compute surcharge in exchange for a large memory saving. Keep this in mind when a throughput number looks disappointing.

The term that 6ND leaves out

Two operations inside attention involve no parameters at all, so $6N$ misses them entirely: the score matrix $QK^\top$ and its application to $V$. For one sequence of length $S$ in one layer,

$$QK^\top:\ S\cdot S\cdot d\ \text{MACs},\qquad \text{softmax}(\cdot)V:\ S\cdot S\cdot d\ \text{MACs},$$

$$\Rightarrow\ 4S^2d\ \text{FLOPs per layer per sequence}\ =\ 4Sd\ \text{FLOPs per layer per token}.$$

Including the backward factor of three, the full training cost per token becomes

$$\boxed{C_{\text{train}}\approx \underbrace{6N}_{\text{weight matmuls}}+\underbrace{12\,L\,S\,d}_{\text{attention scores}}.}$$

Substituting $N\approx 12Ld^2$ gives a strikingly clean expression for how much of the compute attention accounts for:

$$\frac{12LSd}{72Ld^2+12LSd}=\frac{S}{6d+S}.$$

Attention becomes significant once $S\gtrsim 6d$. At $d=4096$ and $S=8192$ it is already a quarter of all training FLOPs. This single ratio is the reason context parallelism exists as a separate strategy, and the reason long-context training is qualitatively harder than large-model training.

hidden size $d$ layers $L$ context $S$

non-embedding $N$

total parameters

training FLOPs / token

attention share

Assumes a gated FFN with $d_{\text{ff}}=\tfrac83 d$ rounded to a multiple of 256, multi-head attention, a 128k vocabulary, and an untied output head. The wall-clock estimate assumes 1024 accelerators at 989 TFLOP/s of dense BF16 peak and 40% model FLOP utilization.

From FLOPs to wall-clock time

The point of counting FLOPs is to predict time. Two definitions make that precise.

Model FLOP utilization (MFU) is the ratio of useful model FLOPs actually performed to the hardware's theoretical peak over the same wall-clock window:

$$\text{MFU}=\frac{6N\cdot(\text{tokens per second})}{P\cdot F_{\text{peak}}}.$$

Crucially the numerator uses $6N$ — the FLOPs the model logically needs — so recomputation counts against you rather than for you. The variant that counts recomputation as useful work is called hardware FLOP utilization, and it always looks better.

Inverting gives the estimate you actually want:

$$T\approx\frac{6ND}{P\cdot F_{\text{peak}}\cdot\text{MFU}}.$$

Work one example by hand. A 70B-parameter model trained on 15T tokens, on 1024 H100s at 989 TFLOP/s dense BF16 and 40% MFU:

$$T=\frac{6\cdot 7\times10^{10}\cdot 1.5\times10^{13}}{1024\cdot 9.89\times10^{14}\cdot 0.4}\ \text{s}\approx 1.56\times10^{7}\,\text{s}\approx 180\ \text{days}.$$

Six months on a thousand accelerators, which is the right order of magnitude for publicly reported runs of that size. Halving the time means doubling the fleet or doubling MFU, and the second option is usually cheaper — which is the entire motivation for the rest of this series.

Good large-scale runs land somewhere between 35% and 50% MFU. If you measure 15%, something structural is wrong — a communication phase that is not overlapping, a pipeline bubble, or a data loader that cannot keep up — and much of this series is about finding which.

The other ledger: memory

FLOPs tell you whether a run is fast. Parameters tell you whether it fits. Mixed-precision training with Adam has a well-known per-parameter footprint:

Item	Precision	Bytes per parameter
Parameters used in the forward pass	BF16	2
Gradients	BF16 (often FP32-accumulated)	2
Optimizer: FP32 master copy	FP32	4
Optimizer: Adam first moment	FP32	4
Optimizer: Adam second moment	FP32	4
Total		16

Sixteen bytes per parameter, before a single activation is stored. A 70B model therefore needs about 1.1 TB of state, which is roughly fourteen 80 GB accelerators just to hold the model — with nothing left for activations. That number, and not FLOPs, is what forces sharding.

Takeaway

$N\approx12Ld^2$, training costs $6N$ FLOPs per token plus $12LSd$ for attention, and Adam mixed precision costs 16 bytes per parameter. Those three numbers determine, before you write any distributed code, how many devices you need and what your ceiling is.

Check yourself

Derive $N_{\text{block}}$ for grouped-query attention with $h_{kv}$ key-value heads, and evaluate it for $h=64$, $h_{kv}=8$. counting
Show that an ungated FFN with $d_{\text{ff}}=4d$ and a gated FFN with $d_{\text{ff}}=\tfrac83 d$ have the same parameter count. algebra
Explain why the backward pass costs exactly twice the forward pass, naming both matmuls. derivation
At what context length does attention reach half of all training FLOPs for $d=8192$? calculation
A run reports 52% utilization but uses full activation checkpointing. Is that MFU or HFU, and what is the other number? interpretation