Likelihood ratio test explained

CS1 · Actuarial Statistics · 12 min read

The problem it solves

Testing a normal mean is easy because the statistic is obvious: look at the sample mean, standardise it, done. Now test whether a claim rate has changed, whether a shape parameter equals 2, or whether three regression coefficients are all zero at once. What do you look at?

The likelihood ratio answers that question once and for all. It takes any hypothesis you can write as a restriction on parameters and manufactures a test statistic from the likelihood itself. You never have to invent a statistic again, and — in the cleanest case — the statistic it gives you is provably the best one available.

Neyman-Pearson: for one value against one value, the ratio is optimal

Start with the simplest possible question. The parameter is either θ₀ or θ₁, nothing else is on the table, and you want the test with the highest possible power for a given size. The Neyman-Pearson lemma says the answer is to reject when

L(θ₁) / L(θ₀) > k

with k chosen so the size comes out at the α you wanted.

In words: reject the null when the data is more than k times as likely under the alternative as under it. No other test of the same size has more power. That is a strong claim and it is proved, not asserted.

The intuition behind the proof is worth a sentence, because it demystifies the result. You have a fixed budget of type-one error to spend — five per cent of the probability under the null. You want to buy as much probability under the alternative as you can with it. So you buy the sample outcomes that give the best ratio of alternative-probability to null-probability first, then the next best, and stop when the budget is gone. That greedy rule is the likelihood ratio rule.

In practice the ratio nearly always simplifies into something recognisable. For a normal mean it reduces to a condition on x̄ alone, which is why the obvious test was the right one all along: the lemma does not replace the usual z test, it explains it.

The generalised likelihood ratio, for hypotheses that are not single values

Real hypotheses are rarely one value against one value. "The rate is 0.10" against "the rate is something else" leaves a whole range unspecified, so there is no single L(θ₁) to put on top. The fix is to use the best each hypothesis can do:

Λ = (largest likelihood allowed under H₀) / (largest likelihood allowed overall)

In words: let the null do its best, let the unrestricted model do its best, and compare. The ratio is at most 1, because the unrestricted maximum cannot be beaten by a restricted one. Values near 1 mean the restriction costs almost nothing, so the data has no complaint about it; small values mean the restriction is expensive and the null looks wrong.

The test statistic is

−2 ln Λ which under H₀ is approximately χ²_q

where q is the number of free parameters the null fixes — the difference in dimension between the two models. One restriction, one degree of freedom. Three coefficients set to zero at once, three degrees of freedom. Large values of the statistic are the evidence against the null, so it is always a one-tailed test on the chi-square, however two-sided the hypothesis was.

Why minus two times the log, and why chi-square

The log is there because likelihoods multiply, and nobody wants to handle products of n small numbers. Taking logs turns Λ into a difference of log-likelihoods, which is also exactly what the computer prints:

−2 ln Λ = 2[ ℓ(θ̂) − ℓ(θ₀) ]

The factor of two and the chi-square distribution arrive together. Expand the log-likelihood as a Taylor series about its maximum. The linear term vanishes, because the first derivative at a maximum is zero, so the leading behaviour is quadratic:

ℓ(θ₀) ≈ ℓ(θ̂) − ½ I(θ̂) (θ̂ − θ₀)²

Rearranged, 2[ℓ(θ̂) − ℓ(θ₀)] ≈ I(θ̂)(θ̂ − θ₀)², and since the maximum likelihood estimator is asymptotically normal with variance 1/I(θ), that right-hand side is the square of a standard normal — a chi-square with one degree of freedom. The factor of two is what the ½ in the Taylor expansion leaves behind. It is not a fudge; it is the only constant that makes the statistic come out as a chi-square.

The word approximately is load-bearing. The expansion is a local quadratic approximation to a curve, so the chi-square distribution is an asymptotic result. For small samples the approximation can be poor, and the worked example shows just how poor.

Wald and score: the same curve, measured in different places

Three tests of the same hypothesis are in the syllabus, and the way to keep them apart is to picture the log-likelihood as a hill with its summit at θ̂.

The likelihood ratio test measures the drop in height from the summit to the point θ₀. A big drop means the null is in a bad place.

The Wald test measures the horizontal distance from the summit to θ₀, scaled by how sharply the hill falls away at the summit:

W = (θ̂ − θ₀)² I(θ̂)

The score test stands at θ₀ and measures the slope there. If the null were right you would be standing near the summit and the ground would be flat; a steep slope says the summit is somewhere else.

S = [ℓ′(θ₀)]² / I(θ₀)

All three have the same chi-square distribution asymptotically, and all three agree in the limit, because a quadratic hill has only one shape. They differ in finite samples because a real log-likelihood is not exactly quadratic: the Wald test uses the curvature at the summit, the score test the curvature at the null, and the likelihood ratio uses the actual heights and so is usually the most trustworthy of the three.

Each has its practical niche. The score test needs only the null fit, which makes it cheap when the unrestricted model is expensive to fit. The Wald test needs only the unrestricted fit, which is why software prints a Wald test beside every coefficient it estimates. The likelihood ratio needs both.

Where it is used after this topic

Deviance. The deviance of a generalised linear model is a likelihood ratio statistic: twice the log-likelihood gap between your model and the saturated one. Comparing two nested models by the difference in their deviances is a likelihood ratio test, and that is the single most examined piece of the GLM topic.

Every coefficient table. The z or t value printed beside a fitted coefficient is a Wald test of that coefficient being zero.

Model selection. The Akaike information criterion is −2ℓ + 2p: the same log-likelihood scale, with a penalty for parameters instead of a chi-square cutoff.

Where marks are lost

Getting q wrong. The degrees of freedom are the number of restrictions, not the number of parameters, not the sample size. Count how many free parameters the null removes.

Putting the likelihoods the wrong way up. Λ has the restricted maximum on top, so Λ ≤ 1 and −2 ln Λ ≥ 0. A negative statistic means the ratio was inverted.

Making the chi-square test two-tailed. Evidence against the null lives in the upper tail only. A two-sided hypothesis still gives a one-sided chi-square test.

Trusting the chi-square in a small sample. It is asymptotic. With a handful of events the three statistics can differ by fifty per cent, and quoting one of them to three decimal places as though it were exact is the real error.

Forgetting to maximise under the null. If the null leaves some parameters free — testing one coefficient in a model that has five others — the numerator is the likelihood maximised over those remaining free parameters, not the likelihood at some convenient guess.

Questions students ask

Is the likelihood ratio test always the best one?

For one simple hypothesis against another, yes, and the Neyman-Pearson lemma proves it. Beyond that case, no such guarantee exists; it is usually the best behaved of the three in finite samples, which is a different and weaker claim. Where a uniformly most powerful test exists, the likelihood ratio construction will find it.

Why do three tests of the same thing give three different answers?

Because they are three different approximations to the same underlying quantity, and approximations only have to agree in the limit. If they disagree materially on your data, that is a signal: the sample is too small for the asymptotics, and the right response is an exact test if one exists, or honesty about the uncertainty if one does not. The worked example below is a case where all three are inconclusive and the sensible answer is to say so.

Can I use it to compare two models that are not nested?

No, and this is a real constraint. The construction needs one model to be a restriction of the other, because otherwise "the number of parameters fixed by the null" is meaningless. Comparing a gamma fit with a lognormal fit is not a likelihood ratio test; that is what information criteria are for.

A worked example

Two parts. The first shows the Neyman-Pearson lemma turning into an ordinary test you already know. The second tests a claim rate four different ways and lets them disagree. Both data sets are invented for this example.

Part (a): the lemma, made concrete

Claim amounts are normal with a known standard deviation of 20 thousand rupees. A sample of 16 is drawn. The two hypotheses are simple: the mean is 100 thousand, or the mean is 110 thousand. Find the most powerful test of size 5 per cent, and its power.

For a normal sample the likelihood ratio is

L(110)/L(100) = exp[ (110 − 100)(Σx)/σ² − 16(110² − 100²)/(2σ²) ]

Everything in the exponent is fixed except Σx, and it enters with a positive coefficient. So "reject when the ratio exceeds k" is identical to "reject when x̄ exceeds some c", and the lemma has handed back the obvious test. Finding c needs only the null distribution:

standard error = 20/√16 = 5.0
c = 100 + 1.6448536(5.0) = 108.2242681

The power is the probability of landing above c when the mean really is 110:

z = (108.2242681 − 110)/5.0 = −0.3551464
power = 1 − Φ(−0.3551464) = Φ(0.3551464) = 0.6387600

The check: do it in likelihood-ratio units instead. Substitute Σx = 16(108.2242681) = 1731.5882896 into the exponent. The first term is 10(1731.5882896)/400 = 43.2897072 and the second is 16(12100 − 10000)/800 = 42, so the exponent is 1.2897072 and

k = e^1.2897072 = 3.6317

So "reject if the data is more than 3.6317 times as likely under 110 as under 100" and "reject if the sample mean exceeds 108.2242681" are the same test written in two languages. Inverting the first expression — solving for the sample mean that produces a ratio of exactly 3.6317232 — returns 108.2242681, and the probability of exceeding it under the null is 0.0500000. Both the size and the rejection region are confirmed from the other direction.

One honest observation. The best possible test of this hypothesis at this sample size still misses a real ten-thousand-rupee shift 36 per cent of the time. No cleverness improves on that; only more data does. With 36 observations the same test has power 0.9123145.

Part (b): a claim rate, tested four ways

A book of business is priced assuming 0.10 claims per policy-year. Over 250 policy-years it produces 18 claims. Is the pricing assumption wrong?

The count is Poisson with mean λ × 250. The estimate and the null expectation are

λ̂ = 18/250 = 0.072 expected under H₀ = 0.10 × 250 = 25

So the book produced 18 claims where 25 were priced for. That looks like a large shortfall. Four tests disagree about whether it is.

The likelihood ratio. For a Poisson count x with exposure E the log-likelihood is −λE + x ln(λE) − ln x!, and the factorial term is the same under both hypotheses, so it cancels:

−2 ln Λ = 2[ x ln(x/λ₀E) − (x − λ₀E) ]
= 2[ 18 ln(18/25) + 7 ] = 2[ 18(−0.328504067) + 7 ]
= 2[ −5.9130732 + 7 ] = 2(1.0869268) = 2.1738536

Check that against the full log-likelihoods, factorial term included and nothing cancelled: ℓ(λ̂) = −2.3687536 and ℓ(λ₀) = −3.4556804, so twice the gap is 2(−2.3687536 + 3.4556804) = 2(1.0869268) = 2.1738536. The two routes agree, which is the reassurance that the cancellation was done correctly.

One restriction, so one degree of freedom. The 5 per cent point of a chi-square with 1 degree of freedom is 3.8414588, and 2.1738536 does not reach it. The p-value is 0.1403741.

The Wald test. The variance of λ̂ is λ/E, estimated at the estimate:

se = √(0.072/250) = √0.000288 = 0.0169706
z = (0.072 − 0.10)/0.0169706 = −1.6499 → W = 2.7222222

That figure is exactly 49/18, since 0.028²/0.000288 = 0.000784/0.000288. Its p-value is 0.0989602.

The score test. Identical arithmetic, except that the standard error is computed under the null:

se₀ = √(0.10/250) = √0.0004 = 0.0200000
S = (0.028/0.02)² = 1.4² = 1.9600000

exactly 1.96, by coincidence of the data rather than for any deep reason. Its p-value is 0.1615133.

The exact test. The count is Poisson and 250 policy-years is a known exposure, so there is no need to approximate anything. Summing the Poisson probabilities for 0 to 18 claims when the mean is 25:

P(X ≤ 18 | mean 25) = 0.0920409

computed two ways, by summing the nineteen terms directly and through the incomplete gamma function, agreeing to twelve decimal places. Doubling for a two-sided test gives 0.1840817.

What to do with four different answers

Test	Statistic	p-value
Likelihood ratio	2.1738536	0.1403741
Wald	2.7222222	0.0989602
Score	1.9600000	0.1615133
Exact Poisson	—	0.1840817

The statistics range from 1.96 to 2.72 and the p-values from 0.099 to 0.184 — nearly a factor of two. They are not supposed to agree at 18 claims. The chi-square distribution for all three is asymptotic, and eighteen events is not asymptotia. Writing "they are asymptotically equivalent" in a solution is correct and is also the explanation of the table, not an excuse for it.

The conclusion is the same whichever row you read: do not reject. Eighteen claims against an expected twenty-five is not evidence that the pricing rate is wrong, which is a genuinely useful thing to know, because it is the opposite of what the raw numbers feel like. A 28 per cent shortfall looks alarming and is well inside Poisson noise at this exposure.

A fifth consistency check, free of charge. The Wald interval for the rate is 0.072 ± 1.9599640(0.0169706), which is 0.0387383 to 0.1052617. It contains 0.10, and a test that fails to reject must produce an interval that contains the null value. If your interval and your test ever disagree, one of them has an arithmetic error.

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

The likelihood ratio test, and why the Wald and score tests exist too