Machine Learning · Regression · Linear Regression · Part 1 of 12

What Linear Regression Estimates

July 2026 · 13 min read

Before computing $\hat\beta$, it helps to know what the coefficient is supposed to estimate. Linear regression does not usually recover an exact rule for every observation. It estimates an average relationship, or the best straight-line summary of that relationship. This distinction prevents three common mistakes: treating association as causation, expecting a line to reproduce a curve, and confusing statistical significance with practical importance.

The Main Target: the Conditional Mean

For one fixed input value $x$, the outcome may still vary because of noise and other influences. The conditional mean is the average outcome among all population rows with that input:

$$\mu(x) = E[Y \mid X = x].$$

The function $\mu(x)$ belongs to the population, not to one dataset. A sample contains only a limited number of noisy observations, so a fitted model is an estimate of this unseen population function.

The Best Line May Still Miss a Curve

With one predictor, ordinary linear regression searches only among lines of the form $a+bx$. If the true conditional mean is a line, OLS can recover that shape. If the true mean is curved, OLS chooses the line with the smallest average squared gap:

$$(a^*,b^*) = \operatorname*{arg\,min}_{a,b}\; E\big[(\mu(X)-(a+bX))^2\big],$$

For one predictor, $b^*=\operatorname{Cov}(X,\mu(X))/\operatorname{Var}(X)$ and $a^*=E[\mu(X)]-b^*E[X]$. This is the best linear predictor. As the sample grows, OLS moves toward this line. It does not move toward the curved function unless the design matrix includes features that can represent the curve.

▶ More Data Improves the Best Line, Not the Missing Curve

The dashed curve is the true mean $\mu(x)=2+3x-0.3x^2$. Each step adds more noisy observations. The fitted orange line becomes more stable and approaches the best population line, but it still cannot reproduce the curve at the ends.

The line misses the curve by $-1.5$ at $x=1$ and $x=7$, by $0$ at $x=2$ and $x=6$, and by $1.2$ at $x=4$. Least squares balances these gaps over the full distribution of $X$. It does not force the gap to zero at every input.

Reading a Coefficient Correctly

In $E[Y\mid X]\approx\beta_0+\beta_1X_1+\cdots+\beta_pX_p$, the coefficient $\beta_j$ describes how the fitted value changes when $X_j$ increases by one unit while the other included predictors stay fixed. More precisely, it uses the part of $X_j$ not already explained by the other columns. This is a conditional association. It is not automatically a causal effect.

Two extra conditions give the coefficient a stronger interpretation:

Correct model shape. If the true mean is represented correctly by the chosen columns of $X$, the coefficient describes that mean rather than only its best linear summary.
No remaining confounding pattern. The condition $E[\varepsilon\mid X]=0$ rules out an average error pattern tied to the included predictors. Regression output alone cannot prove this condition.

Interview trap. A positive, statistically significant coefficient does not prove that $X$ causes $Y$. It shows a conditional association in the observed data. A causal interpretation also needs a credible reason to rule out omitted confounding, reverse causation, and important model misspecification.

Population Function Versus Sample OLS Projection

Three objects must remain separate:

The true conditional mean $\mu(x)$: the population average outcome at each input.
The population best line $\beta^*$: the line that best approximates that mean over the population distribution of $X$.
The sample estimate $\hat\beta$: the line fitted from the available rows.

The first two are fixed population quantities. The sample estimate changes from sample to sample. With representative data and a full-rank design, $\hat\beta$ approaches $\beta^*$. It approaches the true model coefficient only when the linear specification and identification assumptions are correct.

How an Omitted Variable Changes a Coefficient

A two-predictor population model is

$$Y=\beta_0+\beta_1X_1+\beta_2X_2+\varepsilon.$$

If $X_2$ is removed and $X_2$ is related to $X_1$ through $X_2=\delta_0+\delta_1X_1+v$, the short-regression slope approaches

$$\operatorname{plim}\hat\beta_1^{\text{short}}=\beta_1+\beta_2\delta_1.$$

For $\beta_1=2$, $\beta_2=3$, and $\delta_1=0.5$, the fitted slope approaches $3.5$. The extra $1.5$ comes from the omitted predictor. More data and robust standard errors do not remove it. If $\delta_1=0$, omitting $X_2$ does not bias the slope on $X_1$ because the two predictors are unrelated in the population projection.

Prediction, Interpolation, and Extrapolation

A point prediction $\hat y_0=x_0^\top\hat\beta$ estimates the average outcome at $x_0$. A future observation also contains fresh noise, so its prediction interval is wider than an interval for the mean. Interpolation predicts inside the observed feature region. Extrapolation predicts outside it.

For the line $5.6+0.6x$, $x=4$ gives $8.0$ and lies inside the displayed range $1$ through $7$. The value $x=10$ gives $11.6$ by algebra, but no data in the example supports that extension. With several predictors, each value may be individually familiar while their combination is new.

Edge Cases Worth Naming Directly

Situation	What actually happens
Constant predictor column	The column contains no information about a slope and makes the design matrix rank deficient.
Prediction far outside the observed range	The formula still returns a number, but the data provides no evidence that the fitted shape remains valid there.
Correct mean but changing error variance	The OLS slope can remain unbiased, while the usual standard-error formula becomes wrong.
Predictor measured with noise	The fitted slope usually moves toward zero relative to the slope for the true error-free predictor.

What the Interpretation Requires

The best-line interpretation needs finite variances and enough independent predictor variation. A causal interpretation needs a separate study-design argument. Simpson's paradox shows why: an overall positive slope can become negative inside every important group when group membership is omitted. Measurement error, outcome-based selection, and a restricted data range can also change what the coefficient estimates.

The C++ linear-regression overview turns these ideas into a fit/predict contract. The next theory page derives $\hat\beta$ through the normal equations and projection geometry.