Gauss-Markov, Standard Errors, Tests, and Intervals
The fitted coefficient $\hat\beta$ depends on the sample. A different sample from the same population gives a different estimate. This page explains when OLS is unbiased and efficient, how its sampling variation is measured, and how that variation produces standard errors, tests, and intervals.
The Classical Linear Model Assumptions
| Assumption | Statement |
|---|---|
| Linear in coefficients | The model is $y=X\beta+\varepsilon$. Columns of $X$ may still contain transformations such as $x^2$. |
| Full rank | No column of $X$ is an exact combination of the others, so the coefficients are unique. |
| Zero conditional mean | $E[\varepsilon\mid X]=0$: after accounting for $X$, no average error pattern remains. |
| Constant, independent error variance | $\operatorname{Var}(\varepsilon\mid X)=\sigma^2I$: every row has the same error variance, and different rows' errors are uncorrelated. |
Normal errors are not required for the Gauss-Markov theorem. Normality is needed later for exact small-sample $t$ and $F$ distributions.
The Gauss-Markov Theorem
Under these assumptions, OLS is BLUE: the best linear unbiased estimator. Here, best means the smallest variance among estimators that are linear functions of $y$ and unbiased for $\beta$. It does not mean best among every possible nonlinear or biased estimator.
The proof compares OLS with any other linear unbiased estimator $\tilde\beta=Ay$. Unbiasedness requires $AX=I$. The matrix $A$ can be written as the OLS matrix plus another term $D$ satisfying $DX=0$. Its variance becomes
$$\text{Var}(\tilde\beta\mid X) = \sigma^2 AA^\top = \sigma^2\Big[(X^\top X)^{-1} + DD^\top\Big],$$The extra matrix $DD^\top$ cannot reduce variance. Setting $D=0$ gives OLS, so no other linear unbiased estimator has lower variance under the stated assumptions.
Variance of $\hat\beta$ and Standard Errors
The sampling covariance of OLS is
$$\text{Var}(\hat\beta\mid X) = \sigma^2 (X^\top X)^{-1}.$$The unknown error variance $\sigma^2$ is estimated from the residuals:
$$\hat\sigma^2 = \frac{\text{RSS}}{n-p-1} = \frac{\sum_i e_i^2}{n-p-1},$$The denominator is $n-p-1$ because the model estimated $p$ slopes and one intercept. The standard error of coefficient $j$ is
$$\text{se}(\hat\beta_j) = \hat\sigma\sqrt{\big[(X^\top X)^{-1}\big]_{jj}}.$$The animation below recalculates the slope, residual variance, standard error, and confidence interval for each displayed sample.
Homoskedastic Versus Sandwich Covariance
If error variance changes across rows, the simple covariance formula is wrong. The general formula is
$$\operatorname{Var}(\hat\beta\mid X)=(X^\top X)^{-1}X^\top\Omega X(X^\top X)^{-1}.$$The usual formula assumes $\Omega=\sigma^2I$. Sandwich estimators use residual-based estimates of $\Omega$ instead. HC1 applies a degrees-of-freedom correction, while HC2 and HC3 also adjust for leverage. These methods change the estimated uncertainty, not the OLS coefficients.
▶ Different Samples, Different Slopes and Intervals
Six samples come from the same line $y=1+2x+\varepsilon$. Each sample produces a different slope and confidence interval. The true slope remains fixed at $2$.
The six estimated slopes average $2.0433$, close to the true value $2$. Five intervals contain the true slope and one misses it. A 95% procedure is not expected to cover in every sample.
Hypothesis Tests and Confidence Intervals
A two-sided test of $H_0:\beta_j=\beta_{j0}$ uses
$$t = \frac{\hat\beta_j-\beta_{j0}}{\text{se}(\hat\beta_j)},$$With normal errors, this follows a $t$ distribution with $n-p-1$ degrees of freedom under the null. The common test $\beta_{j0}=0$ asks whether the coefficient differs from zero after accounting for the other included predictors. The matching confidence interval is
$$\hat\beta_j \pm t_{\alpha/2,\,n-p-1}\cdot\text{se}(\hat\beta_j).$$Several coefficients can be tested together with a Wald or nested-model $F$ test. For $q$ restrictions $R\beta=r$,
$$F=\frac{(R\hat\beta-r)^\top[R\widehat{\operatorname{Var}}(\hat\beta)R^\top]^{-1}(R\hat\beta-r)}{q},$$Under homoskedastic normal errors, this has an exact $F$ reference distribution. Robust-covariance versions use large-sample approximations.
Mean Interval Versus Prediction Interval
A confidence interval for the average response at $x_0$ is
$$x_0^\top\hat\beta\ \pm\ t_{\alpha/2,\nu}\hat\sigma\sqrt{x_0^\top(X^\top X)^{-1}x_0},$$A prediction interval for one new observation must also include that observation's fresh noise:
$$x_0^\top\hat\beta\ \pm\ t_{\alpha/2,\nu}\hat\sigma\sqrt{1+x_0^\top(X^\top X)^{-1}x_0}.$$In the six-point example, the fitted mean at $x_0=3.5$ is $5.400$. The 95% interval for that mean is $(5.051,5.749)$, while the prediction interval for one new observation is $(4.476,6.324)$. The prediction interval is wider because it includes both coefficient uncertainty and individual noise.
When Normal Errors Matter
Normal errors are not required for unbiased OLS coefficients or for Gauss-Markov. They make the stated $t$ and $F$ distributions exact in small samples. With a large sample, approximate normality of $\hat\beta$ often follows from the central limit theorem even when the errors themselves are not normal.
When the Assumptions Break
- Changing error variance: OLS coefficients may remain unbiased, but the usual standard errors and BLUE claim fail. Robust standard errors fix the uncertainty estimate, not the coefficients.
- Correlated errors: common in time-ordered or grouped data, this also makes the usual standard errors wrong.
- Endogeneity: when $E[\varepsilon\mid X]\ne0$, the coefficients themselves become biased. A different study design or identification method is needed.
The C++ diagnostics implementation computes standard errors, leverage, influence, and a Breusch-Pagan style check. The next page explains why correlated predictors can make individual coefficients unstable.