Linear Regression: From Gaussian Noise to Ridge and Lasso

October 2026 · 10 min read

Expanded from Linear Regression.pdf · View a handwritten page

The MLE notes explain how to choose a probability model that makes observed data plausible. Linear regression applies that idea to numerical outputs. We will fit three rows by hand, connect Gaussian noise to squared error, then use priors to obtain ridge and lasso regularization.

A small table before the equation

Centering subtracts the average number of hours, which is 2. It makes the arithmetic easier without changing which straight lines we can fit. Our prediction rule is $\hat y=b+wu$: $b$ predicts points at two hours, and $w$ is the predicted change per additional hour.

Find the line with the smallest squared errors

A residual is observed output minus predicted output: $r_i=y_i-\hat y_i$. Ordinary least squares chooses $b$ and $w$ to minimize the sum of their squares.

Why centering makes the intercept the mean output

Centering means $u_i=x_i-\bar x$, where $\bar x$ is the mean input. Subtracting that mean makes the centered inputs sum to zero. In our table, $-1+0+1=0$. Their mean $\bar u$ is therefore zero too.

Average the predictions $\hat y_i=b+wu_i$. The intercept contributes $b$ to every row, while the slope contributions cancel when averaged:

So $b$ is the average prediction. We still need to show why least squares makes it equal the average observed output.

Hold the slope $w$ fixed for a moment. Changing $b$ shifts every prediction up or down by the same amount. To find the best shift, differentiate the sum of squared errors with respect to $b$:

Each residual has derivative $-1$ with respect to $b$, which gives the factor $-2$ by the chain rule. At the minimum, this derivative is zero, so the residuals sum to zero. Expanding that sum gives:

The last term vanishes because the inputs are centered. Divide by $n$ to obtain:

This holds for any fixed slope, so we can determine the intercept before calculating the slope. The fitted line predicts the mean output when the input is at its mean.

Now calculate the slope

With $b=\bar y$ fixed, choose $w$ to minimize the squared residuals $(y_i-\bar y-wu_i)^2$. Set the derivative with respect to $w$ to zero and collect the terms containing $w$:

Dividing by the sum of squared inputs gives the slope:

The numerator measures how inputs and outputs vary together around their means. The denominator accounts for the input's spread. Here $\bar y=2$, so the centered outputs are $-1,-1,2$.

The numerator terms are $(-1)(1-2)=1$, $0(1-2)=0$, and $1(4-2)=2$. The squared inputs in the denominator are 1, 0, and 1. Therefore:

The sum of squared errors, abbreviated SSE, is 1.5. On the original hours scale the same line is $2+1.5(x-2)=-1+1.5x$. Its intercept changes because zero now refers to zero hours instead of average hours.

Why would likelihood prefer this line?

Suppose a line describes the average output, with independent Gaussian noise of the same variance for every row:

From noise to observed outputs

Output = predicted value + noise. For a fixed input and a fixed line, the prediction is a fixed number. Adding it to zero-mean Gaussian noise shifts the bell curve's center to the prediction without changing its spread. So our assumption about $\epsilon_i$ also determines the distribution of $y_i$:

A Gaussian with mean $\mu$ and variance $\sigma^2$ has this density:

For row $i$, substitute the observed output $y_i$ and predicted mean $b+wu_i$. Their difference is the residual $r_i=y_i-(b+wu_i)$. Our model attributes this difference to noise. Evaluating the Gaussian noise density at this residual gives the output's density:

From the row densities to a loss

Hold $\sigma^2>0$ fixed while comparing lines. Independence lets us multiply the observed outputs' densities to get the likelihood for all $n$ rows:

The symbol $\prod$ means “multiply over the rows.” Multiplying $n$ copies of the factor in front gives $(2\pi\sigma^2)^{-n/2}$; multiplying the exponentials adds their exponents:

Now take the natural log: products become sums, powers become multipliers, and $\log(e^a)=a$. Since $\mathrm{SSE}=\sum_i r_i^2$, this gives:

Because log is increasing, maximizing $\log L$ chooses the same line as maximizing $L$. Negate it to turn maximization into loss minimization:

What does the first term do?

The first term comes from taking the negative log of the Gaussian normalization factor, $(2\pi\sigma^2)^{-n/2}$. It is not a regularizer: with $\sigma^2$ fixed, it is constant for every choice of $b,w$. A penalty such as $\lambda w^2$ instead favors smaller slopes, as we will see in ridge regression.

The remaining term is SSE times a fixed positive number. Maximizing Gaussian likelihood therefore gives the least-squares fit. Least squares itself does not require Gaussian noise; that assumption gives it this probabilistic interpretation.

If we also estimate $\sigma^2$, we must keep both terms. Increasing variance reduces the SSE term but broadens the Gaussian curve and increases the log term. This prevents choosing arbitrarily large noise variance just to make the error term small.

Here the MLE variance estimate is SSE/$n=1.5/3=0.5$. The usual unbiased estimate divides by $n-2$ to account for the fitted intercept and slope, giving 1.5 instead.

Write the same calculation as a matrix

Place one row per student in a design matrix $X$. Its first column is all ones for the intercept; its second column contains centered hours. Put the two coefficients in a vector $\theta$.

The objective $J(\theta)=\frac12\lVert X\theta-y\rVert^2$ has gradient $X^\top(X\theta-y)$. Setting it to zero gives the normal equations:

This gives $b=2$ and $w=1.5$ again. The inverse formula $(X^\top X)^{-1}X^\top y$ requires independent columns. In code, use a least-squares solver for better numerical stability:

import numpy as np

hours = np.array([1., 2., 3.])
y = np.array([1., 1., 4.])
u = hours - hours.mean()
X = np.column_stack([np.ones(3), u])
theta = np.linalg.lstsq(X, y, rcond=None)[0]
print(theta)                   # [2.  1.5]
print(np.sum((y - X @ theta)**2))  # 1.5

With $n$ rows and $p$ columns, forming these equations costs roughly $O(np^2)$ and solving the dense system costs $O(p^3)$. Other solvers have different costs.

MAP estimation gives ridge regression

Keep the Gaussian likelihood and give the slope a prior $w\sim\mathcal N(0,\tau^2)$, favoring smaller slopes. Hold both variances fixed and leave the intercept unpenalized with a flat prior on $b$. For our observed dataset $D$, MAP maximizes the posterior:

The likelihood and prior have these Gaussian densities:

Taking negative logs turns the product into a sum and maximization into minimization. The two terms become:

With both variances fixed, the log terms are constant in $b,w$. Dropping them gives the MAP objective:

Multiplying by $2\sigma^2$ gives $\mathrm{SSE}+\lambda w^2$, with $\lambda=\sigma^2/\tau^2$. A smaller prior variance means a stronger penalty. This is the MAP interpretation of ridge under this specific summed-loss convention.

Ridge trades larger training errors for a smaller slope; check on held-out data whether predictions improve. Feature units matter because the penalty acts on coefficient sizes.

The matrix equations become $(X^\top X+\lambda P)\theta=X^\top y$, where $P=\operatorname{diag}(0,1,\ldots,1)$ leaves the intercept unpenalized.

MAP estimation gives lasso regression

Keep the Gaussian noise model and change the slope's prior to $w\sim\operatorname{Laplace}(0,s)$, where $s>0$ is the prior's scale. Hold $\sigma^2$ and $s$ fixed and keep the intercept unpenalized. MAP again maximizes likelihood times prior:

The likelihood is Gaussian; the prior now has a Laplace density:

Taking negative logs gives:

Drop terms constant in $b,w$ to obtain the MAP objective:

Multiplying by $2\sigma^2$ gives the lasso objective:

A smaller prior scale gives a stronger penalty. For several coefficients with independent Laplace priors of the same scale, the penalty becomes $\lambda\sum_j|w_j|$. This is L1 regularization, which can set some coefficients exactly to zero.

From the handwritten notes

This guide expands Linear Regression.pdf with worked tables and intermediate steps. The PDF contains all the original pages.

View original page 1

Further reading:

• Stanford CS229: linear regression and Bayesian regularization