Regression Types for Interviews
Regression is a family of models, not only ordinary least squares. Model choice starts with the quantity being estimated and the kind of outcome being observed. A real-valued amount, binary event, count, positive duration, and partly observed event time need different assumptions.
The exact target is called the estimand. It might be a conditional mean, a median, another quantile, an event rate, or a survival probability. The possible values of the outcome are its support. Naming these two items prevents many model-choice mistakes.
For models of a conditional mean, a common template is
$$g\big(E[Y\mid X]\big)=\eta(X).$$The link $g$ connects the mean to a predictor $\eta(X)$. A polynomial or spline changes the shape of $\eta$. A generalized linear model can change the outcome distribution and link. WLS and GLS change how errors are weighted or correlated. Quantile and survival models use different targets, so they do not fit this mean template directly.
Five Parts of a Regression Choice
| Choice | Plain-language question | Examples |
|---|---|---|
| Outcome and target | What can the outcome be, and which part of it matters? | Mean amount, binary probability, count rate, quantile, survival probability |
| Relationship shape | Is the relationship straight, curved, grouped, or interactive? | Linear terms, interactions, polynomials, splines, additive smooths |
| Fitting rule | What rule chooses the coefficients? | Least squares, weighted least squares, likelihood, quantile loss, posterior updating |
| Error pattern | Do errors have unequal spread or depend on one another? | Constant variance, inverse-variance weights, GLS, cluster-robust covariance |
| Complexity control | How are unstable or overly flexible estimates restrained? | Ridge, lasso, smoothing penalties, Gaussian and hierarchical priors |
These choices are separate. The quadratic model $\beta_0+\beta_1x+\beta_2x^2$ changes the relationship shape, but it is still linear in the coefficients. OLS can estimate those coefficients. Polynomial regression is therefore not an alternative fitting rule to OLS; it is a different set of columns for OLS to fit.
Which Part of the Model Must Change?
Each state shows evidence for a different change: relationship shape, error pattern, outcome family, pooling, representation, or censoring-aware likelihood.
A Selection Framework for Interview Cases
| Observed clue | Likely starting point | Main condition to check |
|---|---|---|
| A continuous outcome follows a smooth curve | Polynomial basis, spline, or GAM | Held-out error improves without unstable extrapolation |
| Residual spread changes or nearby rows share errors | WLS or GLS | The proposed weights or dependence pattern are credible |
| The outcome is binary, a count, or strictly positive | GLM | The family, link, and mean-variance relationship fit the outcome |
| Related groups have little data individually | Hierarchical Bayesian regression | Partial pooling across groups is scientifically reasonable |
| Many predictors carry overlapping information | PCR or PLS | Component count is chosen inside validation |
| Some event times are not fully observed | Cox, parametric survival, or AFT | The censoring and model-specific time assumptions are credible |
A Complete Answer Structure
A clear model recommendation covers six parts:
- Target: the mean, quantile, event rate, hazard, survival probability, or full predictive distribution.
- Outcome values: real-valued, binary, count, positive, ordinal, or censored.
- Relationship: straight, curved, additive, interactive, grouped, or low-dimensional.
- Fitting method: least squares, weighted least squares, likelihood, partial likelihood, or posterior computation.
- Assumptions: independent errors, correct family, credible covariance, non-informative censoring, or exchangeable groups.
- Validation: split design, tuning boundaries, diagnostics, uncertainty, and deployment range.
Interview Priority Map
| Priority | Expected coverage |
|---|---|
| Must know | Simple and multiple OLS, assumptions, logistic regression, regularization, residual diagnostics, feature leakage, and train-validation-test separation |
| Common extensions | Polynomial terms, interactions, WLS, count GLMs, splines, robust and quantile losses, and cross-validated tuning |
| Specialized | GLS covariance models, Bayesian hierarchy, PCR and PLS, zero-inflated counts, GAM interactions, and censored survival likelihoods |
Recurring Interview Traps
- Outcome type narrows the choices but does not determine relationship shape, error dependence, or validation.
- Unequal error variance does not by itself bias OLS coefficients when $E[\varepsilon\mid X]=0$, but ordinary standard errors become wrong.
- Many zeros do not by themselves prove that a zero-inflated model is needed.
- Polynomial degree, spline flexibility, and component count are tuning choices and belong inside validation.
- Hazard ratios, odds ratios, rate ratios, and time ratios have different meanings.
Focused Series Roadmap
Polynomial and Basis Regression
Linear models in transformed features, hierarchy, conditioning, orthogonal bases, regularization, and extrapolation.
Weighted and Generalized Least Squares
Heteroskedasticity, correlated errors, whitening, feasible GLS, sandwich inference, and efficiency.
Generalized Linear Models
Exponential-family likelihoods, links, variance functions, IRLS, deviance, and family selection.
Splines and GAMs
Local bases, knots, natural constraints, smoothing penalties, effective degrees of freedom, and additive effects.
Bayesian Regression
Prior-likelihood updating, conjugacy, shrinkage, hierarchical pooling, posterior prediction, and diagnostics.
PCR and PLS
SVD scores, supervised latent components, standardization, component selection, interpretation, and leakage.
Censored and Survival Regression
Survival and hazard functions, censoring likelihoods, Kaplan-Meier, Cox regression, AFT models, and evaluation.
Foundation and Implementation Links
The linear regression foundation explains conditional and partial effects. The OLS projection chapter supplies the geometry behind basis regression, WLS, PCR, and smoothers. Separate chapters cover coefficient penalties and alternative losses and targets.
The logistic regression series provides the binary-outcome and IRLS foundation. Implementation companions begin at C++ Regression from Scratch.