Analysis of variance: comparing three or more means at once

CS1 · Actuarial Statistics · 10 min read

The problem it solves

Three claim-handling teams, three study methods, three regions. Do their averages differ?

The obvious plan is three t tests: one against two, one against three, two against three. The obvious plan is wrong, and the reason is arithmetic rather than opinion. Each test is run at 5 per cent, so each has a 95 per cent chance of behaving. If the three were independent, the chance that all three behave is

Fourteen per cent, not five. With six groups there are fifteen pairs and the figure climbs past half. Somewhere in that pile of tests you will find a "significant difference" that is nothing but the number of tests you ran.

Analysis of variance replaces the pile with one test of one hypothesis — all the group means are equal — carried out at exactly the size you chose.

The idea: two estimates of the same variance

Suppose the null hypothesis is true and every group is drawn from the same distribution. Then there are two quite different ways to estimate that distribution's variance.

From inside the groups. Pool the variation of the observations around their own group means. This estimate is valid whether or not the null is true, because group means cancel out of it.

From between the groups. The group means should themselves be scattered around the grand mean with variance σ²/n per group, so their observed scatter is also an estimate of σ² — but only if the null is true. If the true means differ, this estimate is inflated by the real differences on top of the sampling noise.

So compare them. If the between-groups estimate is much larger than the within-groups estimate, something other than noise is separating the means. That ratio is the F statistic, and the whole method is that one sentence.

The sums of squares identity

With k groups, group means x̄j, and grand mean x̄, the total variation splits exactly in two:

In words: how far each observation is from the overall average, squared and added, equals how far the group averages are from the overall average, plus how far the observations are from their own group average.

It is an identity, not an approximation: write each deviation as (xij − x̄j) + (x̄j − x̄), square, and the cross-term vanishes because deviations from a group mean sum to zero within that group. That cancellation is the single piece of algebra in the topic, and it is also why the identity fails the moment you use anything other than the group means as centres.

This is the same split as the population variance decomposition — within plus between — carried out on a data set instead of on a distribution. Anyone who has met conditional expectation will recognise it, and the recognition is worth having: analysis of variance is that theorem, done with arithmetic.

A warm-up with round numbers

Two groups of two: 1 and 3 in the first, 5 and 7 in the second. The group means are 2 and 6 and the grand mean is 4.

And 16 + 4 = 20. The identity holds, and with 1 and 2 degrees of freedom the mean squares are 16 and 2, giving F = 8. Everything on this page is that calculation with more data.

The F statistic and its degrees of freedom

Divide each sum of squares by its degrees of freedom to get a mean square, which is a variance estimate:

The degrees of freedom follow the same accounting as everywhere else. There are k group means, costing one for the grand mean, so k − 1 between. There are N observations, costing one per group mean estimated, so N − k within. They add to N − 1, matching the total, which is a useful check on an ANOVA table: both columns must add up.

Under the null the statistic follows an F distribution with those two degrees of freedom, and the test is one-tailed. Only a large F is evidence of differing means; a small F means the group means are closer together than noise alone would place them, which is not evidence of anything the test is looking for.

MSwithin is worth a name of its own. It is the pooled estimate of the error variance, its square root estimates the standard deviation of an individual observation, and it is the denominator of every comparison made after the F test.

What the assumptions are, and which one matters

The test assumes independent observations, normal errors, and equal variances across the groups. The normality assumption is fairly forgiving at reasonable group sizes, for the usual central limit reason. The equal-variance assumption is the one to take seriously, because the method pools the within-group variation into one number, and that pooling is meaningless if one group is three times as variable as another. Look at the group standard deviations before trusting an F.

What to do after a significant F

A significant F says the means are not all equal. It does not say which differ, and the question is not answered until you have said so. The standard moves:

Compare the group means against a least significant difference built from the pooled error variance. Any two means further apart than that are different at the stated level.

Adjust for the number of comparisons. Bonferroni is the crudest and the easiest to defend: divide the level by the number of comparisons you intend to make. It is conservative, and the honest way to use it is to decide on the comparisons before looking at the data.

Quote the sizes, not just the verdicts. A difference of 12 days in settlement time is a management fact; "p < 0.05" is not.

Where it is used after this topic

Regression. The ANOVA table printed under a fitted regression is the same decomposition, with "between" replaced by "explained by the predictors". One-way analysis of variance is a regression on dummy variables for the groups, and the two F statistics are identical.

Generalised linear models. The analysis of deviance table is the same idea for non-normal responses, with deviances in place of sums of squares.

Experimental design. Two-way and higher analyses add rows for each factor and for interactions, all built on the same identity.

Where marks are lost

Degrees of freedom that do not add up. (k−1) + (N−k) = N−1. Check it before using the table.

Dividing the between sum of squares by k. It is k − 1.

Forgetting the group size in the between sum of squares. Each squared deviation of a group mean is weighted by how many observations that group has. Omitting nj is the most common arithmetic error in the topic.

Making the F test two-tailed. It is not.

Stopping at the F. Name the groups that differ.

Reporting a significant F as proof of a cause. The groups differ. Why they differ is not something the arithmetic knows.

Questions students ask

Why is it called analysis of variance when the hypothesis is about means?

Because the method gets at the means through variances. The hypothesis is about means; the evidence is a comparison of two variance estimates. The name describes the machinery, not the question.

What if the group sizes are unequal?

Everything above holds as written — the nj weight in the between sum of squares is there precisely for that case. Equal sizes make the test somewhat more robust to unequal variances, which is a reason to design for them when you can, not a requirement for the method.

With only two groups, is this different from a t test?

No, and this is the best check available on an ANOVA calculation. With two groups F = t² exactly, where t is the two-sample pooled t statistic, and the two p-values are identical. The worked example below does it on real numbers: a t of 3 and an F of 9, both giving 0.0170717.

A worked example

An insurer measures how long three claim-handling teams take to settle a claim, in days, with five claims each. The data is invented for this example.

All fifteen values total 990, so the grand mean is 990/15 = 66 days.

The sums of squares

Between the groups, each squared deviation weighted by the five observations behind it:

Within each group the five values are 8, 4, 0, 4 and 8 away from their own mean, so each group contributes 64+16+0+16+64 = 160:

Now compute the total directly, as the check. Deviations from 66: team A gives −14, −10, −6, −2, 2 and squares summing to 340; team B gives 160, as its mean is the grand mean; team C gives 2, 2, 6, 10, 14 in magnitude and sums to 340 as well.

The identity holds exactly, which it must, and a mismatch here would mean an arithmetic error somewhere above rather than anything interesting.

The table

Both columns add up, which is the second check. The 5 per cent point of an F with 2 and 12 degrees of freedom is 3.8852938, and 4.5 is past it; the p-value is 0.0348154. At the 1 per cent level the critical value is 6.9266081 and 4.5 is nowhere near, so the evidence is real but modest.

Two more readings before moving on. The pooled estimate of the standard deviation of a single settlement time is √40 = 6.3245553 days. And the proportion of the total variation attributable to the team is 360/840 = 0.4285714 — 43 per cent, with the remaining 57 per cent being claim-to-claim variation inside the teams.

The check: with two groups, F is a t test squared

Drop team B and rerun on A and C alone. Their grand mean is (300+360)/10 = 66, unchanged, so

Now the ordinary two-sample t test on the same two groups. The pooled variance is 320/8 = 40, so

And 3² = 9. The two tests agree exactly, as algebra rather than as a coincidence, and so do their p-values: the F with 1 and 8 degrees of freedom gives 0.0170717 and the two-sided t with 8 degrees of freedom gives 0.0170717. If an ANOVA calculation ever needs checking, collapse it to two groups and run a t test.

Which teams actually differ

The F test has said "not all equal". With 12 degrees of freedom the two-sided 5 per cent point of the t distribution is 2.1788128, so the least significant difference between any two team means is

So the finding is narrower than the F test alone suggests: team C is slower than team A by about 12 days, and neither differs detectably from team B. That is the whole result, and it is a sentence rather than a verdict.

Finally, the multiple-comparisons correction, since three comparisons is exactly the situation the opening arithmetic warned about. Bonferroni divides 5 per cent by 3, giving 1.67 per cent per comparison, and the corresponding two-sided t point on 12 degrees of freedom is 2.7794731:

The A-to-C gap of 12 days clears even that, so the one firm conclusion survives the correction. The two six-day gaps were not significant before it and are further from significance after. Note the price being paid: the threshold has moved from 8.7 days to 11.1 days, which is the cost of asking three questions instead of one, stated in days rather than in p-values.

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

Other CS1 topics explained

• Moment generating functions, and how to get moments out of them
• The central limit theorem, and when the normal approximation is safe
• Maximum likelihood estimation, step by step
• Confidence intervals: what the 95% actually means
• Hypothesis testing and p-values, without the folklore
• Simple linear regression: least squares, fitted by hand
• Generalised linear models: the three parts, and the link function
• Bayesian estimation, conjugate priors and the credibility formula inside them
• Conditional expectation, and the variance decomposition that comes with it
• Sampling distributions: what the sample mean and sample variance actually do
• The method of moments, and when maximum likelihood is worth the extra work
• The likelihood ratio test, and why the Wald and score tests exist too
• The chi-square goodness of fit test, and how to get its degrees of freedom right
• Contingency tables: testing whether two classifications are related
• Multiple linear regression, and what controlling for another variable really does

Stuck on a term rather than the idea? The actuarial glossary defines the notation used above. If something here is wrong or unclear, tell us — these pages get corrected.