← All Posts
Machine Learning · Regression · Interview Series · Part 7 of 7

Censored and Survival Regression

Suppose you track machines from installation until their first failure. One fails after two months; another is still working when monitoring stops after three months. The second machine's lifetime is longer than three months, not equal to three months. Survival analysis keeps that distinction while estimating how event risk changes over time.

“Survival” means remaining free of the event you defined. The event can be failure, churn, recovery, or another first occurrence. Start by specifying the event, the time origin, and when observation ends.

A censored row is partly observed, not useless. It tells us that no event occurred during its observed follow-up. Treating its follow-up time as a failure invents an event; dropping the row discards valid information.

Five Timelines, One Dataset

All five machines below start observation at month zero. Record $t_i=\min(T_i,C_i)$ and $\delta_i=\mathbf1(T_i\le C_i)$, where $T_i$ is the event time and $C_i$ is the censoring time. The baseline score $x_i$ will be used later in the Cox example.

The complete toy dataset; times are months since installation
MachineObserved time $t_i$Event $\delta_i$What is knownScore $x_i$
A21Failed at month 20
B30Still working when monitoring stopped1
C41Failed at month 41
D61Failed at month 60
E70Still working at the end of the study$-1$
Five machine timelines end at months 2, 3, 4, 6, and 7. Failures are circles at 2, 4, and 6; censoring is marked by crosses at 3 and 7. The Kaplan–Meier curve drops only at failures: from 1 to 0.8, then 0.533, then 0.267. It stays level at censoring times.
Figure 1. Follow-up ends for both events and censoring, but only an observed event lowers the estimated survival curve. Immediately before month 4, the risk set is C, D, and E. The curve ends at the last observed follow-up; its positive final height does not imply that any machine lasts forever.

Build the Kaplan–Meier Curve from Risk Sets

The risk set contains machines that are still observed and have not yet failed immediately before a given time. At event time $t_j$, let $n_j$ be the number at risk and $d_j$ the number of failures. The estimated conditional probability of getting through that time is $1-d_j/n_j$.

To survive to a later time, a machine must get through every earlier event time. The Kaplan–Meier estimator multiplies these conditional survival factors:

$$\hat S(t)=\prod_{t_j\le t}\left(1-\frac{d_j}{n_j}\right).$$

A censoring time adds no failure factor. It removes that machine from later risk sets. This is why censoring changes later drops without causing an immediate drop.

Compute Every Step

“At risk” is counted just before the listed time
MonthAt riskObserved changeSurvival factor$\hat S(t)$ after change
25: A–EA fails$4/5$$4/5=0.800$
34: B–EB censored1$4/5=0.800$
43: C, D, EC fails$2/3$$8/15\approx0.533$
62: D, ED fails$1/2$$4/15\approx0.267$
71: EE censored1$4/15\approx0.267$

At month 4, the calculation is $(4/5)(2/3)=8/15$. Counting only the two machines still observed after month 4 would instead give 40%. That incorrectly treats B’s unknown status at month 4 as a failure. Kaplan–Meier uses what B contributed before monitoring stopped.

The estimated median is the first time $\hat S(t)\le0.5$: month 6 here. If a curve never reaches $0.5$ during follow-up, its median is not estimable from that curve. With only five machines, the steps are highly uncertain; a real report should include confidence intervals and numbers at risk, especially near the tail.

Reproduce the curve and table values
from scipy import stats

data = stats.CensoredData(uncensored=[2, 4, 6], right=[3, 7])
survival = stats.ecdf(data).sf
print(survival.quantiles)      # [2., 3., 4., 6., 7.]
print(survival.probabilities)  # [0.8, 0.8, 0.53333333, 0.26666667, 0.26666667]

SciPy's censored-data estimator uses Kaplan–Meier for right-censored observations. The figure source computes the risk sets with exact fractions and checks them against that implementation. The original method is described by Kaplan and Meier (1958).

What Censoring Allows Us to Learn

For a continuous event time, an observed failure at $t_i$ contributes a density $f(t_i\mid x_i)$. A right-censored observation contributes the probability of surviving beyond its follow-up, $S(t_i\mid x_i)$:

$$L_i\ \propto\ f(t_i\mid x_i)^{\delta_i}S(t_i\mid x_i)^{1-\delta_i}.$$

This event-time likelihood omits factors for the censoring process. Doing so requires independent censoring conditional on the information modeled, with separate event and censoring parameters. Informally, among otherwise comparable machines still being observed, losing monitoring must not reveal extra information about when an unseen failure occurs.

A fixed study end can make that assumption plausible. A sensor that stops transmitting because a machine is about to fail can violate it. Ordinary survival software cannot repair such missing information by itself. Also, conditional independence given $x$ does not guarantee that a pooled, unadjusted Kaplan–Meier curve is valid: censoring must be independent within the population or strata used for that curve.

Other incomplete-observation patterns require different handling:

The R survival-response documentation shows how these observation types are encoded. Software event codes differ, so verify which value means an event before fitting.

Survival Probability and Hazard Answer Different Questions

The survival function is $S(t)=P(T>t)$; event probability by $t$ is $1-S(t)$. For continuous event times, the hazard is an instantaneous rate among those still event-free:

$$h(t)=\frac{f(t)}{S(t)},\qquad H(t)=\int_0^t h(u)\,du,\qquad S(t)=e^{-H(t)}.$$

Over a small interval of length $\Delta t$, the conditional event probability is approximately $h(t)\Delta t$. Hazard has units of inverse time and can exceed one; a probability cannot. The continuous-time identity above does not mean that exponentiating an estimated Nelson–Aalen cumulative hazard exactly reproduces a finite-sample Kaplan–Meier curve.

Cox Regression: Compare Subjects Still at Risk

$$h(t\mid x)=h_0(t)e^{x^\top\beta}.$$

The Cox model lets all subjects share an unspecified baseline hazard $h_0(t)$ and scales it by their covariates. Holding other covariates fixed, a one-unit increase in $x_j$ multiplies the hazard by $e^{\beta_j}$. This ratio stays constant over time under the proportional-hazards assumption.

For distinct event times, the partial likelihood compares each failing subject with the risk set at that moment:

$$L_p(\beta)=\prod_{i:\delta_i=1}\frac{e^{x_i^\top\beta}}{\sum_{j\in R(t_i)}e^{x_j^\top\beta}}.$$

At month 4 in our dataset, C fails while C, D, and E are at risk. Their scores are $(1,0,-1)$. At the illustrative candidate $\beta=\log2$, their relative hazards are $(2,1,1/2)$, giving one contribution:

$$\frac{2}{2+1+1/2}=\frac47\approx0.571.$$

This is C's share of the risk-set hazard conditional on one event occurring then, not C's probability of failing by month 4. Machine B is absent at month 4 but contributed to earlier risk sets before being censored. The value $\log2$ is chosen to make the arithmetic clear; it is not a fitted coefficient from these five rows.

The baseline hazard cancels from each comparison. For time-fixed covariates, absolute survival prediction still needs an estimated baseline survival: $S(t\mid x)=S_0(t)^{e^{x^\top\beta}}$. If $S_0(t)=0.8$ and the hazard ratio is 2, survival is $0.8^2=0.64$. The corresponding event probabilities are $0.20$ and $0.36$, not a doubling to $0.40$.

Inspect scaled Schoenfeld residuals or time interactions for proportional-hazards violations. Check nonlinear covariate effects separately: proportional hazards does not establish a linear log-hazard relationship. For tied event times, use a stated method such as Efron or Breslow rather than applying the no-ties formula unchanged. The Cox fitting documentation covers ties, delayed entry, and time-dependent covariates. Only covariate values already available at each risk interval belong in the model.

AFT Models: Interpret Changes in Time

An accelerated failure time model instead writes

$$\log T=x^\top\beta+\varepsilon.$$

With a shared error distribution, changing $x_j$ by one unit multiplies conditional event-time quantiles by $e^{\beta_j}$. For example, a time ratio of $1.5$ turns a model median of 8 months into 12 months, holding other covariates fixed. A Cox hazard ratio generally has no such direct median-time interpretation.

Weibull, log-normal, and log-logistic AFT models imply different tail behavior. The Weibull family also has a proportional-hazards representation, with a different coefficient parameterization. Check the chosen software convention; R's parametric survival documentation explicitly discusses this distinction.

Tobit: Censoring of a Measurement

Tobit addresses a different observation process: a continuous value clipped at a known limit. A left-censored Gaussian Tobit model uses a latent outcome $y_i^*=x_i^\top\beta+\varepsilon_i$, with $\varepsilon_i\sim N(0,\sigma^2)$, but records $y_i=\max(c,y_i^*)$.

An uncensored measurement contributes a Gaussian density. A measurement at the lower limit contributes the probability that the latent value is at or below it:

$$P(y_i^*\le c\mid x_i)=\Phi\left(\frac{c-x_i^\top\beta}{\sigma}\right).$$

This requires a credible latent Gaussian measurement process. A pile of zeros caused by nonparticipation, such as choosing not to spend, may need a two-part model instead. Tobit is not a generic replacement for time-to-event modeling.

Evaluate the Prediction You Will Use

The scikit-survival evaluation guide explains the estimators and follow-up restrictions. Good ranking alone does not establish calibrated survival probabilities.

Competing events change the target. If retirement permanently prevents an in-service failure, retirement is a competing event. Censoring retirements and reporting $1-\hat S(t)$ as the real-world failure probability generally overestimates that probability. Use a cumulative-incidence estimator such as Aalen–Johansen. Censoring other event types can be appropriate for fitting a cause-specific hazard, but that hazard alone is not cumulative incidence; all competing hazards matter. See the survival package's competing-risk documentation.

Choose the Model by the Question

QuestionStarting pointKey check
What fraction remains event-free?Kaplan–MeierIndependent censoring in the population or strata analyzed
How do covariates change relative event rates?Cox regressionProportional hazards and covariate specification
How do covariates stretch or shorten event times?AFT regressionDistribution and time-ratio assumptions
What is one event's probability when another prevents it?Cumulative incidenceCorrect competing-event definitions
What latent continuous measurement was clipped?TobitActual measurement censoring and latent model

Quick Checks