Contingency tables: testing whether two classifications are related

CS1 · Actuarial Statistics · 10 min read

The problem it solves

Two classifications of the same set of things. Policies by region and by whether they claimed. Claims by branch and by whether they settled on time. Students by study method and by outcome. The question is always the same: are the two classifications related, or is the apparent pattern just the shuffle?

It is a question with real consequences in actuarial work, because a relationship found here is what justifies a rating factor. If claim frequency genuinely varies by region, region belongs in the pricing model. If the variation is noise, putting it in the model prices noise.

The test is the chi-square goodness of fit statistic again. What changes is where the expected counts come from, and that is the only new idea on this page.

Expected counts under independence

The null hypothesis is that the two classifications are independent. Independence has a definition — P(A and B) = P(A)P(B) — and the expected counts are that definition, applied cell by cell.

Estimate each marginal probability from its own total: the probability of row i is Ri/n and of column j is Cj/n. If they are independent, the probability of the cell is the product, and the expected count is n times that:

In words: row total times column total, divided by the grand total. The famous recipe is not a recipe at all; it is the definition of independence with the marginals estimated from the data.

And the expected counts have a property worth checking every time: each row of expected counts adds to that row's observed total, and each column to that column's total. If your table of expected counts does not reproduce the margins, you have made an arithmetic error, and it is quicker to find it there than at the end.

The statistic itself is unchanged:

Degrees of freedom

This is worth deriving once rather than memorising, because the derivation is the same accounting as the previous page. There are rc cells. The total costs one. The row probabilities cost r − 1 estimates — not r, because they sum to 1 — and the column probabilities cost c − 1:

So the neat product form is the goodness of fit rule with the estimated marginals counted properly. A 3 by 2 table has 2 degrees of freedom; a 2 by 2 table has 1.

There is also a direct way to see it: once you have fixed the margins, how many cells can you still choose freely? In a 2 by 2 table, exactly one — after that the margins force the rest. In an r by c table, a block of (r−1)(c−1) cells.

What the test will not tell you

A significant chi-square says the classifications are related. It does not say how, how much, or which way round, and a solution that stops at "reject the null" has answered only the first third of the question.

Where the relationship is comes from the individual contributions, or better from the standardised residuals (O − E)/√E, which are roughly standard normal and so can be read directly: anything beyond about 2 in magnitude is a cell worth naming.

How large the effect is comes from comparing the proportions, not from the statistic. The statistic grows with the sample size for a fixed pattern — double every count and the chi-square doubles while the percentages do not move at all. A huge chi-square from a huge table can represent a difference too small to act on.

Which way the causation runs is not a statistical question. A table cannot distinguish a study method that works from a study method chosen by the students who were going to do well anyway.

The two-by-two table, where several tests turn out to be one

For a 2 by 2 table the chi-square statistic is exactly the square of the usual two-proportion z statistic, with the pooled proportion as the standard error. Not approximately: identically, as algebra, and the worked example demonstrates it to seven decimal places. If you have ever wondered which of the two tests to use on a 2 by 2 table, the answer is that it does not matter.

Two refinements belong here too.

Yates's continuity correction subtracts ½ from each |O − E| before squaring. The motivation is the same as the continuity correction in a normal approximation to a discrete count: the statistic jumps in steps and the chi-square distribution is smooth. It always reduces the statistic, making the test more conservative, and it is applied only to 2 by 2 tables. Modern practice is divided on whether to use it; what is examinable is knowing what it does and why.

Fisher's exact test abandons the approximation and computes the probability of the observed table, and every more extreme one, directly from the hypergeometric distribution with the margins held fixed. It is the right tool when expected counts are small, and it is what statistical software reaches for on a sparse 2 by 2.

Where it is used after this topic

Rating factors. Deciding whether a classification belongs in a pricing structure at all, before any model is fitted.

Generalised linear models. A contingency table is a saturated Poisson model on the cell counts, and the chi-square test of independence is a test of the no-interaction model against the saturated one. That is the same deviance comparison as on the previous page, and it generalises to three-way and higher tables, which this test cannot handle.

Mortality and morbidity investigations, where a classification by two factors at once is the usual shape of the data.

Where marks are lost

Expected counts from the wrong totals. Row total times column total over grand total. Dividing by a row total instead is the classic slip and it produces counts that do not reproduce the margins — which is why the margin check is worth thirty seconds.

Degrees of freedom as rc − 1. It is the product of the reduced dimensions.

Running the test on percentages. Counts only. Always.

Stopping at the verdict. Name the cells that drove it and quote the proportions. That is where the remaining marks are, and it is also the only part a practitioner cares about.

Treating a significant result as a causal claim. Independence is what is tested. Causation is not.

Questions students ask

Can I use this on a table with small counts?

The same rule as for goodness of fit: expected counts of at least 5. Rows or columns can be combined if the combination is meaningful, and Fisher's exact test is available for a sparse 2 by 2. Pooling categories that have nothing in common to satisfy an arithmetic rule is worse than the problem it solves.

What exactly is being held fixed when I compute the expected counts?

Both sets of margins. That is why the degrees of freedom come out as they do, and it is also the assumption behind Fisher's exact test, which enumerates all the tables with those same margins. It is worth noticing that the test treats the marginal totals as given information and asks only about the pattern inside them.

Is a bigger chi-square a stronger relationship?

No — it is stronger evidence of a relationship, which is a different thing. The statistic scales with n. For the strength of the relationship, look at the proportions themselves, or at a measure built to be independent of sample size such as Cramér's V. The worked example has a chi-square of 13.73 from 400 claims; the same percentages from 4,000 claims would give 137.3, and the practical conclusion would be identical.

A worked example

An insurer reviews 400 claims handled by three branches and records whether each was settled within 30 days. The table is invented for this example; it is not an insurer's data.

Is settlement speed related to branch?

The expected counts

Row total times column total over 400. The column totals happen to be equal here, so every expected count is half its row total:

Check the margins: the rows give 120, 160 and 120, and the columns give 200 and 200. Both reproduce the observed totals, so the expected table is right.

The statistic, two ways

The last two terms are 256/60, so the 6 recurs and the total is 13.73333…; seven decimal places is as far as it is worth carrying.

By the computational identity instead: ΣO²/E is 86.4 + 38.4 + 88.2 + 72.2 + 32.26667 + 96.26667 = 413.73333, and subtracting n = 400 gives 13.73333. Agreed to every digit carried.

The conclusion, and then the part that matters

Degrees of freedom: (3−1)(2−1) = 2. The 5 per cent point of a chi-square with 2 degrees of freedom is 5.9914645, so 13.73 is well past it. The p-value is 0.0010419.

That p-value can be checked without any table at all, because a chi-square with exactly 2 degrees of freedom has a closed-form tail: P(χ²2 > x) = e−x/2. Here e−6.8666667 = 0.0010419, matching the computed value to ten decimal places. It also puts the result in context: the 0.1 per cent point for 2 degrees of freedom is 13.8155106, so this statistic falls just short of significance at the 0.1 per cent level.

So: settlement speed is related to branch. Now say how.

South contributes 62 per cent of the statistic on its own, and West is almost exactly where independence predicts. The finding is not "branches differ" but "South is slow and North is quick, by about 23 percentage points, while West is at the average" — which is a sentence someone can act on.

The two-by-two check, where the test turns out to be one you already knew

Take just North and South, which is a 2 by 2 table on 240 claims: 72 and 48 against 44 and 76. The column totals are now 116 and 124, so the expected counts are 120(116)/240 = 58.0 and 120(124)/240 = 62.0 in both rows.

Now the same question as a test of two proportions. North settles 0.6000000, South settles 0.3666667, and the pooled proportion is 116/240 = 0.4833333:

Identical to the chi-square, which the other route gave as 13.0812013 — and taking that figure the other way round, √13.0812013 = 3.6167943, is the z to its full precision. Four decimals is as far as the z can be carried from a standard error printed to seven, which is why the square is quoted to four as well. The two-sided p-value is 0.0002983 by either route.

With Yates's correction — subtracting ½ from each discrepancy of 14 before squaring — the statistic falls to 12.1635150 and the p-value rises to 0.0004873. The correction has made the test more conservative, which is what it is for, and here it changes nothing about the conclusion. On a borderline table it would.

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

Other CS1 topics explained

• Moment generating functions, and how to get moments out of them
• The central limit theorem, and when the normal approximation is safe
• Maximum likelihood estimation, step by step
• Confidence intervals: what the 95% actually means
• Hypothesis testing and p-values, without the folklore
• Simple linear regression: least squares, fitted by hand
• Generalised linear models: the three parts, and the link function
• Bayesian estimation, conjugate priors and the credibility formula inside them
• Conditional expectation, and the variance decomposition that comes with it
• Sampling distributions: what the sample mean and sample variance actually do
• The method of moments, and when maximum likelihood is worth the extra work
• The likelihood ratio test, and why the Wald and score tests exist too
• The chi-square goodness of fit test, and how to get its degrees of freedom right
• Analysis of variance: comparing three or more means at once
• Multiple linear regression, and what controlling for another variable really does

Stuck on a term rather than the idea? The actuarial glossary defines the notation used above. If something here is wrong or unclear, tell us — these pages get corrected.