The Kaplan-Meier estimator, and why censoring needs it

CS2 · Risk Modelling & Survival Analysis · 11 min read

The problem, in one sentence

You watched 10 lives for ten years. Six of them died, at known times. The other four were still alive when you stopped watching — one moved away after 2.8 years, one surrendered the policy at 4.1, one was lost at 6.3, and one was simply still there when the study closed at 9.0. What is the probability of surviving five years?

Two obvious answers are both wrong, and seeing why is most of the topic.

Drop the four. Then you have six lives, all of whom died, and your estimated survival probability eventually reaches zero. That cannot be right: you have thrown away the lives that did best, and the four who vanished did so for reasons that had nothing to do with dying. Dropping them biases every estimate downwards.

Count them as survivors to the end. Then the life lost at 2.8 years is credited with surviving nine years, which you do not know and have no right to assume. This biases every estimate upwards.

The right answer uses each life for exactly as long as it was watched and not one day longer. That is what the Kaplan-Meier estimator does.

What censoring is, and the assumption it carries

An observation is censored when you know the event had not happened by a certain time but you do not know when it did happen. The common case is right censoring, which is the one above: the life left the study alive. It arises every time real data is collected — studies end, policies lapse, patients move, machines are retired before they fail.

Everything that follows depends on one assumption, and a question will often test whether you have noticed it. The censoring must be non-informative: the reason a life left must carry no information about when it would have died. A study that ends on a fixed date satisfies this easily. A policy surrendered because the holder has just been diagnosed with something serious does not, and no amount of correct arithmetic afterwards repairs it. If the people who leave are systematically healthier or sicker than those who stay, the estimate is biased and the method cannot tell you so.

The idea: break survival into a chain of one-step survivals

Surviving to time t means surviving every stretch between here and there. So write the survival probability as a product:

Now choose the stretches so that each one contains exactly one death time. Between death times nothing happens, so the probability of surviving such a gap is 1 and it drops out of the product. All that is left is one factor per observed death time, and each factor is a conditional probability estimated from the lives who were actually there to be at risk.

That gives the Kaplan-Meier or product-limit estimator. Let the distinct death times be t1 < t2 < …, let dj be the number who died at tj, and let nj be the number at risk immediately before tj:

In words: at each moment somebody died, work out the fraction of those present who got through it, and multiply all those fractions together.

The risk set is where the method lives

nj is the number still being watched and still event-free just before tj. A life that was censored earlier is not in it. A life censored at exactly tj is, by the standard convention, counted as at risk at tj and removed afterwards — it survived up to that instant, which is all the information it had.

So a censored life never appears in any numerator and it does appear in the denominator of every death time before it left. That is precisely the "used for exactly as long as it was watched" rule, and it is the whole mechanism. The estimator does not model the censoring, allow for it, or adjust for it. It simply never asks a life a question it cannot answer.

The curve's shape

The estimate is a step function: flat between death times, dropping at each one. Censoring times produce no step, but they make the next step bigger, because the risk set they left is smaller. That is worth knowing because it is visible on a plotted curve: a long flat stretch followed by a large drop usually means the risk set had thinned out, not that something dramatic happened.

The curve reaches zero only if the largest observed time is a death. If the largest observation is censored, the estimate stops above zero and is simply undefined beyond that point — the data has run out, and saying so is the honest answer rather than extrapolating.

The Nelson-Aalen estimator, and how it differs

The same risk-set reasoning applied to the integrated hazard instead of the survival function gives the Nelson-Aalen estimator:

In words: at each death time, the fraction of the risk set that died is an estimate of the hazard concentrated there; add those fractions up.

Since S = exp(−H) for the true functions, it is tempting to treat exp(−Ĥ(t)) as another estimate of survival. It is one, and it is a legitimate one — but it is not equal to the Kaplan-Meier figure, and a page that implied otherwise would be teaching you something false. Because 1 − x ≤ e−x for every x, every Kaplan-Meier factor is at or below the matching Nelson-Aalen factor, so

The two are close while every dj/nj is small, and they separate once risk sets get thin — which on a small data set happens at the right-hand end. The worked example shows the gap, and it is large. Use Kaplan-Meier when the question asks for a survival probability and Nelson-Aalen when it asks for an integrated hazard or for something to graduate; do not use one as a shortcut for the other.

Attaching a standard error

An estimate from 10 lives deserves an error bar, and Greenwood's formula supplies it:

The term for each death time grows as nj falls, so the late steps dominate the variance, exactly as intuition says they should. An approximate 95% interval is the estimate plus and minus 1.96 standard errors, and on small data sets it comes out embarrassingly wide. That is not a defect of the formula; it is the honest width, and the worked example reports it rather than hiding it.

Where marks are lost

Getting the risk set wrong after a censoring. This is the single most expensive error in the topic. Write a column for nj and reduce it by every departure, deaths and censorings alike, before moving to the next row. Checking that the departures add to the number who started is a five-second audit that catches it.

Putting a step in at a censoring time. A censored observation produces no factor at all. If your product has as many factors as you have observations, you have treated censorings as deaths.

Writing a tie as two rows. Two deaths at the same recorded time are one row with dj = 2, not two rows with dj = 1. Write it as one row, because that is what the estimator says and because the table is then readable.

It is worth knowing, though, that this particular slip costs nothing numerically, and knowing why. With five at risk and two deaths, one row gives a factor of 3/5; two rows give (4/5)(3/4), which telescopes to 3/5. Greenwood's terms do the same: 2/(5×3) = 0.133333 and 1/(5×4) + 1/(4×3) = 0.05 + 0.083333 = 0.133333. Both identities hold for any n and d, so a split tie is harmless here. Ties are not harmless everywhere — in Cox regression they genuinely need an approximation, which is a topic of its own — so do not carry "ties don't matter" forward from this page.

Reading the estimate past the last observation. If the final observation is censored, the estimator says nothing beyond it. "The data does not reach ten years" is a complete answer.

Using exp(−Ĥ) when the question said Kaplan-Meier. They differ, sometimes a lot. Answer the question that was asked.

Questions students ask

Why is it called the product-limit estimator?

Because it is the limit of the product you get by chopping time into ever finer intervals and estimating the conditional survival probability in each. In the limit, every interval that contains no death contributes a factor of 1 and vanishes, leaving one factor per death time. The name describes the derivation rather than the formula.

Is Kaplan-Meier a non-parametric method?

Yes, and that is the point of it. It assumes no shape for the hazard at all — no Weibull, no Gompertz, nothing. It is therefore the natural first thing to compute on a new data set, and the natural thing to plot a fitted parametric model against to see whether the shape you assumed was defensible.

What if two lives are censored at exactly a death time?

The convention is that they were at risk at that instant, so they count in nj and leave immediately after. State the convention you used; an examiner cares that you knew a choice existed.

Does it handle left truncation?

Yes, with the same risk-set machinery, and this is what makes it useful for actuarial data. A life that enters observation at age 55 is simply not in the risk set at any age below 55. The method only ever asks who was being watched at each death time, so lives joining late are no harder than lives leaving early.

A worked example

Ten lives are observed from entry. The times below are in years and the figures are invented for this example — they are not from any study, portfolio or published data set.

Six deaths, four censorings, ten lives. The audit adds up before any arithmetic starts.

Warm up: what happens with no censoring at all

Suppose for a moment that all ten of those times had been deaths. Then the product up to and including t = 5.0 is

and every numerator cancels the next denominator except the ends, leaving 3/10 = 0.3000000. Count directly instead: of the ten lives, seven have died by t = 5.0, so three are left, so the proportion surviving is 3/10 = 0.3000000. The same number by two routes, and the cancellation is the reason: with no censoring Kaplan-Meier reduces to the obvious head count. It is only when the risk set falls for a reason other than death that the product stops telescoping and starts earning its keep.

The real table, with the censoring put back

Five rows for six deaths, because 5.0 is a tie and takes one row. Follow the risk set down and check every step:

Zero at the bottom, and 6 + 4 = 10. Had that not closed, the table would be wrong and nothing after it would be worth computing.

The answers

Note what the censorings have done. Three lives left before t = 5.0, and the estimate at 5.0 is 0.411 rather than the 0.3 the no-censoring version gave — higher, because the deaths that did occur were spread over a thinner population than a raw head count assumes. Beyond t = 9.0 the estimate is undefined: the last observation is a censoring, so the curve stops at 0.2057 and the honest answer to "what is the ten-year survival probability?" is that this data cannot say.

Nelson-Aalen on the same table

Against the Kaplan-Meier figure of 0.2057143. The two are a long way apart, and that is the honest result, not an arithmetic slip. The inequality holds as it must — 0.2057 ≤ 0.2854 — and the reason is visible in the terms: the last two rows contribute d/n of 0.4 and 0.5, and the approximation 1 − x ≈ e−x is poor at x = 0.5. Had every risk set been large, the two would have agreed to three decimal places.

How much to trust 0.4114

Greenwood's formula, summed to t = 5.0:

The interval runs from 7% to 76%. That is the correct answer and it is the useful one: ten lives cannot pin down a five-year survival probability, and the single number 0.4114 on its own would have implied a precision the data does not have. Notice also that the tied row supplies 0.1333333 of the total 0.1821428 — nearly three quarters of the variance comes from the one step where two of the five remaining lives died at once.

Where this fits in CS2

This page explains one idea from CS2 Risk Modelling & Survival Analysis. The full CS2 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

Other CS2 topics explained

• The survival function and the hazard rate, explained
• The Cox proportional hazards model, and what a hazard ratio means
• Markov chains: transition matrices, Chapman-Kolmogorov and the stationary distribution
• Central exposed to risk, and the two mortality rates it produces
• Bühlmann credibility: how much to trust one risk's own experience
• Extreme value theory: block maxima, thresholds and the generalised Pareto distribution
• Copulas and Sklar's theorem: modelling dependence without touching the margins

Stuck on a term rather than the idea? The actuarial glossary defines the notation used above. If something here is wrong or unclear, tell us — these pages get corrected.