Flow Matching, Score Matching & Diffusion Models

September 2026 · 25 min read · MIT 6.S184, Lecture 3

What this lecture adds

Lecture 2 chose a sequence of distributions from noise to data and derived the velocity that follows it. Here we learn that velocity from examples. We also learn an alternative, the score, and show how it gives the velocity needed for generation.

The route is: make training examples → learn velocity or score → generate with an ODE or SDE. Each step below explains what we know, what the network predicts, and why the formula works.

The training example and its symbols

Time runs from noise at $t=0$ to data at $t=1$. Pick a data example $Z$ and independent Gaussian noise $\varepsilon$, then mix them:

$\alpha_t$ controls how much data we include; $\beta_t$ controls the noise scale. They start at $(0,1)$ and end at $(1,0)$, giving $X_0=\varepsilon$ and $X_1=Z$. Here $d$ is the number of coordinates; $I_d$ means the noise has independent coordinates of variance one. Capital letters denote random samples; lowercase $x,z$ denote particular values.

Throughout the losses, $\mathbb E$ means average over the sampled training examples, and $\|\text{prediction}-\text{label}\|^2$ means sum the squared errors across coordinates. Training estimates this average with a batch. Unless stated otherwise, sample $t$ uniformly; score formulas need $\beta_t>0$ and the time weighting discussed below.

1. Flow matching: predict a velocity

What should the model learn? Given only $(t,x)$, output the velocity $u_t(x)$ that moves the full distribution along our chosen path. Ideally we would minimize:

The problem is the label $u_t(X_t)$: evaluating it requires averaging over all possible hidden data examples. We can instead compute the velocity for the particular pair $(Z,\varepsilon)$ we sampled. Hold that pair fixed and differentiate its position:

This gives conditional flow matching (CFM): use the same network, with a label we can calculate.

Input: time and the noisy point. Label: the conditional velocity. We use $Z$ and $\varepsilon$ to build the example, but do not give them to the network. Here “conditional” refers to the known training endpoint, not a text prompt.

The simplest choice: a straight line

Choose $\alpha_t=t$ and $\beta_t=1-t$, so $\dot\alpha_t=1$ and $\dot\beta_t=-1$:

The input changes with time, but the velocity label stays constant for a fixed pair. One training step is:

1. Sample a batch of independent $(t,Z,\varepsilon)$ and construct each $X_t$.
2. Predict $u_t^\theta(X_t)$ and compare it with the label $Z-\varepsilon$.
3. Average $\|u_t^\theta(X_t)-(Z-\varepsilon)\|^2$ over the batch, then update $\theta$ by gradient descent.

Training does not require solving an ODE.

Each training pair follows a straight line. Generated paths can still curve: the learned velocity averages suggestions from many possible pairs, and those averages change as the sample moves. The lecture calls this a conditional optimal-transport (CondOT) path; that name does not mean the independently sampled noise–data pairs give a globally optimal matching between distributions.

2. Why those labels teach the right velocity

Different hidden pairs can produce the same observed $(t,x)$ and suggest different velocities. Squared-error training learns their average, weighted by how likely each pair is after observing $(t,x)$. That is exactly the marginal velocity from Lecture 2:

The part after the bar means “given this time and this observed point.” The expectation averages the remaining uncertainty about $Z$. It uses the posterior weights from Lecture 2, so equally frequent data examples need not get equal weight at a particular $x$.

More precisely, the two losses differ only by a nonnegative constant that the network cannot change:

Consequently they have the same expected parameter gradients and the same best-fitting network. A perfect marginal prediction can still have positive conditional loss, because the individual labels disagree.

3. Score matching: predict how density changes with position

Why learn a score? It supplies the correction needed when generation adds random motion. For our Gaussian path, it also lets us recover the velocity, so we can train one network and use either sampler.

This differentiates log density with respect to position, holding time fixed. In one dimension, a positive score means log density increases to the right; a negative score means it increases to the left. It is a direction of increasing density, not itself a velocity. A zero score can occur at a peak or a valley.

As before, the ideal label is hard to compute for the whole data mixture. The ideal score-matching (SM) loss is:

Derive a label we can calculate

For one fixed endpoint, $p_t(x\mid z)=\mathcal N(x;\alpha_tz,\beta_t^2I_d)$. Its density contains $\exp(-\|x-\alpha_tz\|^2/(2\beta_t^2))$. Taking the log removes the exponential; the normalization becomes a term $C$ independent of $x$:

Differentiate with respect to $x$. The derivative of $C$ is zero. The squared term gives $2(x-\alpha_tz)$, just as the derivative of $(x-c)^2$ is $2(x-c)$:

For our sampled point, $X_t=\alpha_tZ+\beta_t\varepsilon$, subtracting the mean leaves $\beta_t\varepsilon$. Therefore:

The conditional score points back toward its Gaussian mean. Its one-dimensional slope is $-1/\beta_t^2$: smaller noise means a narrower cloud and a stronger score for the same displacement from the mean. At zero noise the conditional law is a point mass, so these density derivatives no longer apply.

4. Denoising score matching: use the known noise as a label

We know the noise used to construct each training example, so the score label above is available. Replace the unavailable marginal score in the loss with this conditional label:

The plus sign comes from subtracting the negative label: prediction minus $(-\varepsilon/\beta_t)$. This is denoising score matching (DSM): learn from noisy examples how to point toward higher density.

Why averaging the training labels gives the mixture's score

The network sees only $(t,x)$. Different hidden pairs $(Z,\varepsilon)$ can produce that same input, with different labels $-\varepsilon/\beta_t$. As in flow matching, squared-error training learns their average, weighting each possible pair by how likely it is given this input. We must show that this average is the score we want.

Start with the mixture density: average the conditional densities over data examples.

Its score is the derivative of its log. Use $\nabla_x\log p_t=(\nabla_xp_t)/p_t$, then differentiate the mixture inside the integral:

To expose the conditional score, multiply and divide the integrand by $p_t(x\mid z)$. Nothing changes in value; the two resulting fractions have useful meanings:

The second fraction is Bayes' rule: it weights each endpoint $z$ by how likely it is to have produced the observed $x$. Thus the mixture's score is a weighted average of conditional scores. An endpoint whose cloud is denser at $x$ gets more weight when the prior weights are equal.

Finally, we already derived $s_t(X_t\mid Z)=-\varepsilon/\beta_t$. Replace each conditional score in that average by its noise-based label:

The bar means keep time and the observed point fixed; average over the hidden data–noise pairs that could have produced that point. This is exactly the average learned by squared-error training, so the learned target is the marginal score.

The same loss proof therefore gives $\mathcal L_{\mathrm{DSM}}=\mathcal L_{\mathrm{SM}}+C_{\mathrm{score}}$, when the losses are finite. Here $C_{\mathrm{score}}$ is the average squared difference between an individual conditional label and the marginal score:

Different hidden pairs can give different labels for the same $(t,x)$; this constant measures their disagreement around their average. It contains no network prediction or parameters $\theta$, so training cannot change it. Even a network that predicts the exact marginal score has DSM loss $C_{\mathrm{score}}$. The density calculation above assumes we can differentiate inside the integral.

Predict noise to avoid a label that grows as noise shrinks

Dividing by small $\beta_t$ makes the score label large. Instead, let a network $\varepsilon_t^\theta(x)$ predict the noise, then convert its prediction to a score with $s_t^\theta(x)=-\varepsilon_t^\theta(x)/\beta_t$. Substitution into the squared error gives:

Multiplying each example's score loss by $\beta_t^2$ cancels that factor. We get ordinary noise-prediction mean squared error:

The training procedure is the flow-matching procedure with a different label: predict $\varepsilon$ instead of $Z-\varepsilon$. The best noise prediction is $\mathbb E[\varepsilon\mid t,X_t]$. This is why “predict the noise” does not mean the model can identify the exact noise in every example.

5. Turn a score prediction into a velocity

Purpose: sampling needs a velocity, but we may have trained a score or noise network. Both labels come from the same Gaussian construction, so we can express one using the other.

Work at a time with $\alpha_t\ne0$ and $\beta_t>0$, since the derivation divides by these quantities. For one training pair, abbreviate its conditional score as $s=s_t(x\mid z)$. First solve for the hidden noise, then for the endpoint:

Substitute these into the conditional velocity $\dot\alpha_tz+\dot\beta_t\varepsilon$:

The equation above still assumes we know the endpoint $z$. During generation we only have the current time and position, so we average over the endpoints that could explain them. Use the posterior weights $p_t(z\mid x)$: how likely each endpoint is after observing $x$ at time $t$. These weights integrate to one.

In this average, only $z$ varies. We hold $t$ and $x$ fixed, so $\alpha_t$, $\beta_t$, and their time derivatives are fixed too. The two multipliers in the conversion therefore come outside the integral:

The first line is the marginal velocity's averaging rule from Section 2. In the second line, averaging the same value $(\dot\alpha_t/\alpha_t)x$ over weights that sum to one leaves that value unchanged. In the last line, the weighted average of conditional scores is exactly the marginal score, as proved in Section 4. This gives:

For the straight-line schedules

Substitute $\alpha_t=t$, $\dot\alpha_t=1$, $\beta_t=1-t$, and $\dot\beta_t=-1$. The score coefficient simplifies to $(1-t)^2/t+(1-t)=(1-t)/t$:

If the network predicts noise, substitute $s_t^\theta=-\varepsilon_t^\theta/(1-t)$; the factors $1-t$ cancel:

Going the other way, multiply the score–velocity equation by $t$, subtract $x$, then divide by $1-t$:

These are conversions between network outputs; they do not require knowing the endpoint at generation time. They also involve divisions: score-to-velocity needs $t>0$, while velocity-to-score needs $t<1$.

6. Use the learned quantities in a diffusion model

In this lecture, a denoising diffusion model uses a Gaussian probability path. The conversion above means its velocity and score can come from one trained network.

The ODE moves samples with velocity $u_t$. To allow random motion while keeping the same distributions at each time, Lecture 2 adds both fresh noise and a score correction:

$W_t$ is Brownian motion. The chosen coefficient $\sigma_t$ controls noise added during sampling; $\beta_t$ describes the noise already present in the chosen probability path. They have different jobs.

Why add the score term? Random movement spreads probability out. The score adds movement toward increasing density, counteracting that extra spreading. With the exact fields and correct starting distribution, the resulting SDE follows the same densities $p_t$ as the ODE. Setting $\sigma_t=0$ recovers the ODE.

Generate a new sample, one step at a time

Start with fresh $x_0\sim\mathcal N(0,I_d)$. For a velocity network, split time into $N$ steps of width $h=1/N$, with $t_k=kh$. Euler sampling uses “new position = old position + velocity × elapsed time”:

Return $x_N$ as the generated example. We choose no data endpoint $Z$ during this process; the learned field guides the sample. More, smaller steps reduce solver error under suitable smoothness, but cannot fix an inaccurate network.

For stochastic sampling, add the score correction and draw fresh independent $\xi_k\sim\mathcal N(0,I_d)$ each step:

The $\sqrt h$ comes from Brownian increments: their variance over a time interval $h$ is $h$, so their standard deviation is $\sqrt h$. This update is called Euler–Maruyama.

Keep these training labels together

All three networks receive the same $(t,X_t)$, with $X_t=\alpha_tZ+\beta_t\varepsilon$. The table separates the label for one pair from what squared-error training learns after averaging possible pairs.

Where this sits in the literature

The lecture uses MovieGen and Stable Diffusion 3 as flow-matching applications. The training target is one part of a generator; architecture, representation, and inputs such as text prompts are additional choices. See the lecture's application examples.

• Discrete diffusion: add noise through a sequence of steps and learn a reverse denoising chain; DDPM is an example.
• Score-based SDE models: start with a data-to-noise SDE and reverse it using the score. Song et al. also give an ODE with the same distributions.
• Flow matching and stochastic interpolants: choose an interpolating path, learn its velocity, and optionally add the score-corrected random motion above.

Check the time direction when comparing formulas: these notes go from noise to data; many diffusion papers start their clock at data. Reversing an ODE changes its velocity's sign. Reversing an SDE also requires a score term, so flipping the drift's sign alone is insufficient.

Beyond Gaussian noise

Flow matching needs a conditional path we can sample and a velocity label we can calculate. Its source need not be Gaussian: the lecture discusses bridges such as low-resolution to high-resolution images, silent video to video with audio, and unperturbed to perturbed cell states. See the bridge examples.

A bridge must specify how source and target examples are paired, called their coupling. Producing plausible high-resolution images alone does not ensure they match a particular low-resolution input; pairing or conditioning must encode that relationship. Different modalities also need a compatible representation. The Gaussian score–velocity conversion above applies to our specific construction, not automatically to every bridge.

Optional checks with short answers

Sources and reading route

Based on my six-page handwritten notes, Lecture 3 floor matching and diffusion models (14 December 2025): flow training and its proof on pages 1–3; score matching and DSM on pages 3–4; conversion and bridges on pages 5–6. The intermediate algebra, loss weighting, and sampling explanations expand those notes.

Companion material: MIT 6.S184 Lecture 3 slides and the 2025 course page. For the construction of the targets themselves, return to Lecture 2.

Check yourself

Continue into language: the Large Language Diffusion Models series builds on these probability paths and conditional targets to explain token corruption, masked training, discrete flow matching, and practical text generation.