Chi-square goodness of fit test

CS1 · Actuarial Statistics · 10 min read

The problem it solves

Fitting a distribution is the easy half. A moment estimator or a likelihood estimator will hand you a gamma or a Poisson for any data you point it at, including data that has no business being gamma or Poisson. Nothing in the fitting process asks whether the family was right.

The chi-square goodness of fit test asks. It compares the counts you observed with the counts the fitted distribution predicts, reduces the comparison to one number, and tells you whether that number is larger than chance alone would produce. It works on any distribution, discrete or continuous, as long as you can group the data into categories — which is both its strength and the source of every subtlety in it.

The statistic

Sort the data into k categories. Let O_i be the number observed in category i and E_i the number the fitted model expects there. Then

χ² = Σ (O_i − E_i)² / E_i

In words: for every category, take how far off you were, square it so that overshoots and undershoots both count, and divide by what you expected. Add up.

Both operations in that sentence are doing something.

Squaring is needed because the discrepancies must sum to zero — the fitted counts add to n and so do the observed ones — so without squaring the total would always be zero and carry no information.

Dividing by E_i is the part worth understanding properly. A discrepancy of 10 means something quite different in a cell expecting 12 than in a cell expecting 1,200. A count in a cell behaves roughly like a Poisson variable, whose variance equals its mean, so E_i is the natural yardstick for that cell's noise. Dividing by it converts each discrepancy into units of its own standard deviation, squared — which is exactly what makes the total a chi-square.

That reading also tells you what the statistic is made of: it is a sum of squared standardised residuals, and inspecting the individual terms is usually more informative than the total. A single cell contributing most of the statistic is a specific, reportable finding.

A warm-up with round numbers

A coin is tossed 100 times and lands heads 60 times. Is it fair? Two categories, each expecting 50:

χ² = (60−50)²/50 + (40−50)²/50 = 2 + 2 = 4

With one degree of freedom the 5 per cent point is 3.8414588, so 4 just crosses it. And the same question by the proportion test: z = (0.6 − 0.5)/√(0.25/100) = 0.1/0.05 = 2, and z² = 4. The two tests are not merely similar, they are algebraically the same test, and seeing that once removes the worry about which to use.

Degrees of freedom, which is where the marks are

df = (number of categories) − 1 − (number of parameters estimated from the data)

Two separate subtractions, for two separate reasons.

The 1 is because the counts must add to n. Once you know k − 1 of them, the last is determined; there were never k free discrepancies.

The parameters are subtracted because each one was tuned to make the fit better. If you estimated the Poisson mean from the same data you are now testing, the expected counts have already been bent towards the observations, so the statistic comes out smaller than it should and the test becomes too generous. Removing a degree of freedom for each estimated parameter corrects for that.

The practical consequence is worth stating plainly: the degrees of freedom depend on where the parameters came from. Testing against a Poisson with a mean specified in advance by a pricing basis is a different test, with different degrees of freedom, from testing against a Poisson whose mean you estimated from the data in front of you — even though every observed count, every expected count and the statistic itself are identical.

The small expected count rule

The chi-square distribution of the statistic is an approximation, and it comes from treating each cell's count as roughly normal. A cell expecting 0.7 observations is not remotely normal, and a discrepancy there, divided by 0.7, can inflate the statistic out of all proportion.

The working rule is that every expected count should be at least 5 — some texts allow a single cell slightly below — and the fix when one is not is to pool adjacent categories until it is. For a claim count distribution this almost always means combining the tail into a single "3 or more" category.

Two things to remember when pooling. Pool the observed and the expected counts together, so the cell stays consistent. And recount the categories afterwards, because pooling reduces k and therefore reduces the degrees of freedom. Forgetting the second is a standard loss of two marks: the arithmetic is right and the critical value is wrong.

Where it is used after this topic

Testing a claim count model. Poisson against the observed distribution of claims per policy is the canonical application, and the one in the worked example.

Contingency tables. The same statistic with expected counts built from row and column totals tests whether two classifications are related.

Graduation. The chi-square test of a graduation compares actual deaths with expected deaths age group by age group, and is the first of the standard battery of tests in the later syllabus.

Deviance. The alternative statistic G² = 2ΣO ln(O/E) is the likelihood ratio version of the same comparison and has the same asymptotic distribution. In generalised linear model output it is the deviance.

Where marks are lost

Subtracting for parameters you did not estimate, or failing to subtract for ones you did. The single most common error in this topic, and it changes the critical value rather than the statistic, so nothing in your arithmetic warns you.

Using proportions instead of counts. The statistic is built from frequencies. Feed it percentages and the answer scales by whatever n happens to be, which is meaningless.

Not pooling, or pooling and forgetting to recount. Both change the conclusion.

Treating a large p-value as proof the model is right. It is not evidence of fit, only an absence of evidence of misfit — and with a small sample the test can barely detect anything at all. A high p-value from 40 observations and a high p-value from 40,000 are very different statements.

Making it two-tailed. Only large values are evidence against the model. A suspiciously small chi-square is not evidence of a wonderful fit; if anything it suggests the data has been smoothed or fitted twice.

Questions students ask

How should I choose the categories for continuous data?

By a rule fixed before you look at the discrepancies, not by one chosen to flatter or damn the fit. Equal-probability intervals under the fitted model are the usual choice, since they make every expected count equal and the pooling question disappears. Choosing boundaries after seeing where the data clusters invalidates the test, and it is not a subtle invalidation — it can manufacture any conclusion you like.

What is the difference between this and the deviance?

They are two statistics for the same comparison, from two different starting points. The Pearson chi-square squares the discrepancies; the deviance 2ΣO ln(O/E) compares log-likelihoods. They are asymptotically equivalent and in practice usually agree closely — in the worked example, 0.0628 against 0.0625. When they disagree materially it is a sign of small expected counts, which is useful information in itself.

The test says my model fits. Can I now use it?

With the caution above, and with one addition: this test only looks at the categories you built. A distribution can match the grouped counts well and still be wrong about the thing you care about, which in insurance is usually the tail. A model that passes a goodness of fit test on five cells of claim counts has said nothing whatever about the size of the largest claim.

A worked example

A portfolio of 200 policies produced the following numbers of claims in a year. The data is invented for this example.

Claims in the year	0	1	2	3	4
Policies	109	65	22	3	1

Is a Poisson distribution a reasonable model?

Step one: fit it

The total number of claims is 0(109) + 1(65) + 2(22) + 3(3) + 4(1) = 65 + 44 + 9 + 4 = 122, so

λ̂ = 122/200 = 0.61 claims per policy

Note what has just happened, because it governs the degrees of freedom later: the mean was estimated from the data being tested.

Step two: the expected counts

Poisson probabilities with λ̂ = 0.61, multiplied by 200. The first is e^−0.61 = 0.5433509, and each one after that is the previous one times 0.61/k:

Claims	Probability	Expected	Observed
0	0.5433509	108.6702	109
1	0.3314440	66.2888	65
2	0.1010904	20.2181	22
3	0.0205551	4.1110	3
4 or more	0.0035596	0.7119	1

The expected counts add to 200.0000 exactly, which is the arithmetic check on the whole column: they must, because the probabilities add to 1 by construction of the last cell.

Step three: pool

Two cells expect fewer than 5 — 4.1110 and 0.7119. Pooling just those two gives 4.8229, still under 5, so the "2" cell has to come in as well. That leaves three categories:

Category	Observed	Expected
0 claims	109	108.670174
1 claim	65	66.288806
2 or more	26	25.041020

Both columns still add to 200, which is the check that the pooling was done to both sides. Six decimal places are carried in that last column on purpose: the contributions below are quoted to seven, and four-decimal expected counts cannot reproduce them.

Step four: the statistic, two ways

(109 − 108.670174)²/108.670174 = 0.0010011
(65 − 66.288806)²/66.288806 = 0.0250573
(26 − 25.041020)²/25.041020 = 0.0367254
χ² = 0.0627838

Now the same number by the computational identity ΣO²/E − n, which is the form worth knowing because it needs no subtractions until the end:

109²/108.670174 + 65²/66.288806 + 26²/25.041020 = 200.0627838
− 200 = 0.0627838

The two agree to seven decimal places, which is the arithmetic confirmed.

Step five: the degrees of freedom, and only now the conclusion

Three categories, less one for the total, less one for the estimated mean:

df = 3 − 1 − 1 = 1

The 5 per cent point of a chi-square with 1 degree of freedom is 3.8414588. The statistic is 0.0627838, nowhere near it, with a p-value of 0.8021489. There is no evidence against the Poisson model.

And if you had forgotten to subtract for the estimated mean, you would have used 2 degrees of freedom and reported a p-value of 0.9690957. The same conclusion here, by luck, because the statistic is so small — but the error is live, and on a borderline statistic it flips the answer. The habit to build is to write the two subtractions down separately, every time.

The cross-check, and an honest reading

The deviance form of the same comparison gives G² = 2ΣO ln(O/E) = 0.0624868, with a p-value of 0.8026078. Note that this is not an identity — 0.0625 is not 0.0628 — and it is not meant to be. These are two different statistics for the same comparison, agreeing closely because the expected counts are comfortable, which is the circumstance in which they are supposed to agree.

A second piece of evidence, from outside the test: a Poisson distribution has its variance equal to its mean. The sample variance of these claim counts is 0.6109548 against a mean of 0.61, a ratio of 1.0015652. For real claim count data that is remarkably close, and it is the single most informative line in this example — two independent angles both saying the Poisson is doing its job.

One caveat to finish, because it is the part a careful answer earns credit for. The test passed on three categories with 200 policies, and that is weak evidence: this test could not have detected a mild departure in the tail, because the tail had to be pooled away before the test would run. The honest conclusion is "no evidence against a Poisson at this sample size", and not "the claim counts are Poisson".

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

The chi-square goodness of fit test, and how to get its degrees of freedom right