Regression: Overview and Model Guide
Regression uses input variables $X$ to estimate something about an outcome $Y$. The right model depends first on the type of outcome. A continuous outcome such as price or demand usually starts with linear regression. A binary outcome such as pass or fail usually starts with logistic regression. Counts, positive skewed values, censored outcomes, and time-to-event data need other regression families.
This page is the map for the whole regression series. It explains what linear and logistic regression share, where they differ, and when a different family is a better match.
Average Value Versus Class Probability
A regression model learns a population pattern rather than memorizing each row. For a continuous outcome, squared-error regression targets the average outcome at a given input:
$$\mu(x)=E[Y\mid X=x].$$For a binary outcome, the average of zeros and ones is the probability of a one:
$$E[Y\mid X=x]=0\cdot P(Y=0\mid x)+1\cdot P(Y=1\mid x)=P(Y=1\mid x).$$Logistic regression models this probability while keeping it between $0$ and $1$. The two models therefore begin with the same broad question, but the outcome type changes the link function, loss, and evaluation metrics.
One Score, Different Outputs
Both linear and logistic regression start with the same kind of feature row:
$$\phi(x)=(1,x_1,\ldots,x_p,\text{derived terms}),\qquad s=\phi(x)^\top\beta.$$The leading $1$ creates the intercept. Linear regression returns the score $s$ directly as a continuous prediction. Logistic regression passes $s$ through the sigmoid, $\sigma(s)=1/(1+e^{-s})$, to produce a probability. A class label is created later by applying a decision threshold to that probability.
- Continuous outcome: linear regression is the usual starting point.
- Binary outcome: logistic regression is the usual starting point.
- Count, positive, censored, or time-to-event outcome: a specialized family may match the data better.
▶ Choosing a Regression Family
Each step starts with the outcome type and a visible data problem, then shows the target, model, and correction that match the case.
The first two steps choose the basic linear and logistic models. Later steps show why a model may need to change: a fan-shaped residual plot calls for different uncertainty estimates, curvature calls for richer features, leakage invalidates evaluation, and count data calls for a count model. Least squares can be fitted to binary labels, but it may predict values below $0$ or above $1$, so it is usually a poor probability model.
Why the Link Function Changes
A link function connects the linear score $X\beta$ to the mean of the outcome.
- Linear regression: the score itself is the estimated mean, $E[Y\mid X]=X\beta$.
- Logistic regression: the score is the log-odds, $\log\dfrac{p(X)}{1-p(X)}=X\beta$, and the sigmoid converts it to a probability.
Linear regression usually minimizes squared error. Logistic regression minimizes Bernoulli log-loss. Squared error is minimized by the conditional mean. Bernoulli log-loss is minimized by the true class probability. The loss therefore matches the quantity the model is meant to estimate.
| Property | Linear Regression | Logistic Regression |
|---|---|---|
| Target | Continuous $Y \in \mathbb{R}$ | Binary $Y \in \{0,1\}$ |
| Quantity estimated | $E[Y\mid X]$ | $P(Y=1\mid X)$ |
| Link function | Identity | Logit (log-odds) |
| Training objective | Least squares | Maximum likelihood / log-loss |
| Closed-form solution | Yes, normal equations | No, solved iteratively (Newton or IRLS) |
| Important property | Gauss-Markov under its assumptions | Convex log-loss |
| Typical evaluation | $R^2$, RMSE, residual diagnostics | Log-loss, ROC-AUC, calibration |
Problems Shared by Both Models
Linear and logistic regression can fail for many of the same reasons.
- Correlated predictors: coefficients become unstable and harder to interpret.
- Extrapolation: predictions outside the observed feature range rely on an untested shape.
- Missing structure: omitted variables, curvature, or interactions can shift coefficients and leave patterns in the errors.
- Leakage: information unavailable at prediction time creates unrealistically good evaluation results.
A Small Model-Choice Example
Both models need representative data, stable feature definitions, no leakage, and enough observations in the region where predictions will be made. Linear-regression inference also needs independent design columns and assumptions about the errors. Unregularized logistic regression needs overlapping classes; complete separation can send its coefficients toward infinity.
For $(\text{hours},Y)=(1,0),(2,0),(3,1),(4,1)$, the outcome is binary, so the target is $P(Y=1\mid\text{hours})$. With $\beta=(-3,1)$, three hours gives score $0$ and probability $\sigma(0)=0.5$. Treating the score itself as a probability fails at six hours because the score becomes $3$. The sigmoid avoids this range problem.
Three Regression Roadmaps
Linear Regression: Theory Roadmap
Continuous outcomes, ordinary least squares, assumptions, inference, correlated predictors, model design, numerical solvers, regularization, robust losses, diagnostics, and production failures.
Logistic Regression: Theory Roadmap
Binary and multiclass outcomes, probabilities and odds, log-loss, optimization, regularization, thresholds, metrics, calibration, and production failures.
Regression Families and Interview Guide
Polynomial regression, WLS and GLS, generalized linear models, splines and GAMs, Bayesian regression, PCR and PLS, and censored or survival outcomes.
Focused Families Series
The Regression Types for Interviews overview organizes the extra families by the type of outcome, shape of the relationship, error structure, and training method.
From Theory to the C++ Contract
The standard-library C++ companion starts at From Scratch · C++ · Regression and follows the same order from data validation and fitting to prediction, metrics, and diagnostics.