Machine Learning · Regression· Assumptions
Assumptions of Linear and Logistic Regression
An assumption is a condition needed for a particular claim. Some assumptions affect the coefficient itself. Others affect only standard errors, confidence intervals, or probability calibration. A violation does not always make the entire model useless; it changes which conclusions remain valid.
Assumptions Shared by Both Models
| Assumption | Plain meaning | Common failure |
|---|---|---|
| Representative data | Training rows resemble the population where the model will be used. | Selection bias or deployment drift. |
| No leakage | Every feature is available at prediction time. | Future values or post-outcome fields enter training. |
| Correct feature shape | The chosen columns can represent the important relationship. | Missing curvature, interaction, or subgroup effect. |
| Independent information in columns | No predictor is an exact copy or combination of others. | Duplicate columns or the dummy-variable trap. |
| Appropriate row dependence | Rows treated as independent are genuinely independent, or dependence is modeled. | Repeated patients, users, locations, or time points. |
| Stable preprocessing | Training and serving use the same units, categories, scaling, and missing-value rules. | Train-serve skew. |
Linear Regression Assumptions
The population model is
$$Y=X\beta+\varepsilon.$$| Assumption | Why it matters | If it fails |
|---|---|---|
| Linear conditional mean in the chosen features | $X\beta$ correctly describes $E[Y\mid X]$. | Residual curvature calls for justified transformations or another model. |
| $E[\varepsilon\mid X]=0$ | Included predictors have no remaining average error pattern. | Coefficients may be biased because of omitted variables, reverse causation, or selection. |
| Full column rank | Coefficients are unique. | Redundancy must be removed or resolved with a constraint or pseudoinverse. |
| Constant error variance | Needed for the textbook OLS standard-error formula and BLUE claim. | Coefficients may remain unbiased, but robust or variance-aware uncertainty is needed. |
| Uncorrelated errors | Needed for ordinary standard errors. | Cluster, time-series, or another dependence-aware covariance is needed. |
| Normal errors | Needed for exact small-sample $t$ and $F$ inference. | Large-sample inference may still work; tail behavior determines a suitable alternative. |
Key distinction. Heteroscedasticity does not automatically bias OLS coefficients. It mainly breaks the usual variance formula. By contrast, $E[\varepsilon\mid X]\ne0$ can bias the coefficients themselves.
Logistic Regression Assumptions
Logistic regression models
$$\operatorname{logit}\big(P(Y=1\mid X)\big)=X\beta.$$| Assumption | Why it matters | If it fails |
|---|---|---|
| Binary outcome | The Bernoulli likelihood expects labels $0$ and $1$. | A multinomial, ordinal, count, or other outcome family is required. |
| Linear log-odds in the chosen features | The score $X\beta$ must represent the logit shape. | Justified basis terms, splines, or interactions can represent the missing shape. |
| No exact multicollinearity | Coefficients must be identifiable. | Redundant columns need removal or regularization. |
| No complete separation for unregularized MLE | Both classes need overlap for a finite coefficient solution. | Coefficients diverge; regularization or a bias-reduced method restores a finite estimate. |
| Enough positive and negative events | Rare classes provide little information for many coefficients. | Lower complexity, more data, regularization, or a suitable prior may be needed. |
| Stable class process | Probabilities are calibrated only for the population represented in training. | Prior or concept shift changes calibration and threshold performance. |
Class imbalance is not itself a broken assumption. It changes uncertainty, metric choice, and threshold selection. A well-specified model can still estimate probabilities under imbalance when enough information exists.
What Different Violations Look Like
▶ Six Formula-Derived Assumption Cases
Each step shows a small dataset, the model shape, and the specific conclusion that fails.
What Each Failure Changes
| Failure | Coefficient | Uncertainty | Prediction |
|---|---|---|---|
| Wrong mean or logit shape | Misspecified target | Misleading | Systematic errors |
| Heteroscedastic linear errors | Can remain unbiased | Classical SE is wrong | Mean prediction may remain useful |
| Omitted confounder | Can be biased | Precise but wrong is possible | May fail under intervention or shift |
| Multicollinearity | Unstable or nonunique | Large SE | Can remain stable on familiar support |
| Logistic separation | No finite unregularized MLE | Unstable | Ranking may look perfect in sample |
| Population shift | Old relationship may no longer apply | Training intervals irrelevant | Calibration and error degrade |
A Short Practical Checklist
- Outcome: continuous for linear regression, binary for logistic regression.
- Feature timing: no value from after the prediction point.
- Shape: residual or logit diagnostics show no important missing pattern.
- Rank: no exact duplicate columns; condition number is monitored.
- Dependence: split and covariance methods match grouping or time order.
- Support: deployment rows resemble the training feature region.
- Evaluation: metrics, calibration, and thresholds match the real decision.
Common Misconceptions
- Linear regression does not require every predictor to look normally distributed.
- Normal residuals are not required for unbiased OLS coefficients.
- Logistic regression does not require balanced classes.
- A high $R^2$ or accuracy does not prove the assumptions are correct.
- Robust standard errors do not fix omitted-variable bias.
- Regularization can control separation or instability, but it does not remove leakage.