What a test is doing
A hypothesis test answers one question: could the data I am looking at reasonably have arisen if a particular claim were true? If the answer is no, the claim is rejected. If the answer is yes, nothing has been proved — the data were simply not inconsistent with it.
That asymmetry is built into the machinery and is the source of most misunderstanding. A test can convict; it cannot acquit. Everything else on this page follows from taking that seriously.
The five pieces
The null hypothesis H0. The specific claim being tested, always precise enough to compute with: μ = 0, or λA = λB. It is usually the dull option — no effect, no difference — because a test is built to see whether the data can dislodge it.
The alternative H1. What you are entitled to conclude if H0 falls. One-sided (μ > 0) if only one direction is of interest and that was decided before the data were seen; two-sided (μ ≠ 0) otherwise.
The test statistic. One number computed from the data whose distribution is known if H0 is true. That last clause is the whole design: you cannot measure how surprising data are without a reference distribution to measure against.
The significance level α. How much risk of a false alarm you are willing to run, fixed before looking at the data.
The decision. Reject H0, or do not reject it. Never "accept H0".
The p-value, defined exactly
In words: if the null hypothesis were true and you ran the whole experiment again and again, what fraction of the time would you see evidence this strong or stronger, purely by chance?
Read it slowly, because every clause is load-bearing. The probability is computed assuming H0. It is a probability of data, not of a hypothesis. And "at least as extreme" means the tail of the reference distribution, not the single point observed — which is why a p-value is always an area and never a density.
A small p-value says: this data set sits far out in the tail of what H0 predicts. Either something unusual has happened, or H0 is wrong. The test takes the second view once p falls below α.
What a p-value is not
Not the probability that H0 is true. It is conditioned on H0 being true, so it cannot be a statement about whether it is. Getting the probability of a hypothesis given the data requires a prior and Bayes' theorem, which is a different part of the syllabus and a different quantity.
Not the probability the result was a fluke. The same error, differently dressed.
Not a measure of how large the effect is. A tiny, commercially meaningless difference becomes highly significant with a large enough sample, because the standard error shrinks while the difference does not. Significance answers "is it there?"; size answers "does it matter?". A complete answer reports both, which is what the confidence interval is for.
Two ways to be wrong
| H0 true | H0 false | |
|---|---|---|
| Reject H0 | Type I error, probability α | Correct, probability 1 − β |
| Do not reject | Correct | Type II error, probability β |
A type I error is a false alarm: rejecting something true. Its probability is α, and it is chosen rather than discovered — setting α = 0.05 is deciding to be wrong this way one time in twenty when the null holds.
A type II error is a miss: failing to reject something false. Its probability β is not chosen. It depends on how false H0 is, on how much data you have, and on α.
Power is 1 − β: the chance of detecting a departure of a stated size. It is not a single number — it is a function of the true value, rising from α when the truth equals H0 towards 1 as the truth moves away.
The trade-off is unavoidable at fixed sample size. Make α smaller and the rejection region shrinks, so false alarms fall and misses rise. The only way to improve both at once is more data. That is why "what sample size do I need?" is really the question "what power do I want, against what departure?".
Two routes to the same decision
Every test can be run two ways, and they are equivalent by construction.
The critical-value route. Fix α, look up the critical value, reject if the statistic is beyond it. The historical method, from when tables were all there was.
The p-value route. Compute the tail probability of the observed statistic and reject if it is below α.
They agree because the p-value is the smallest α at which you would reject. Put another way, the critical value corresponding to α = p is the observed statistic itself. That identity is a genuine arithmetic check on a test, and the worked example uses it.
The p-value route is preferred in practice because it reports the strength of the evidence rather than just the verdict. p = 0.049 and p = 0.0001 both "reject at 5%", and they are not remotely the same finding.
Where marks are lost
Writing "accept H0". Not failing to reject is not evidence of truth. A test with low power fails to reject almost everything, including falsehoods. The correct phrase is "there is insufficient evidence to reject H0 at the 5% level".
Choosing one-sided after seeing the data. Picking the direction the data happen to point halves the p-value and doubles the false-alarm rate. The side is part of the hypothesis and is fixed before the data are examined.
Halving or doubling the wrong way. A two-sided p is twice the one-sided p for a symmetric reference distribution. Getting this backwards reports a result as four times more significant than it is.
Using z where t is required. With σ estimated and n small, the z route gives a p-value that is too small — in the worked example below, 0.0052 instead of 0.0075, an overstatement of about 30%.
Not stating the conclusion in the context of the question. "Reject H0" on its own is an incomplete answer. "There is evidence at the 1% level that the method improves marks" is the answer.
Questions students ask
Why is 0.05 the threshold?
Because it became conventional, not because anything in the mathematics selects it. Roughly two standard errors is a convenient line, and it stuck. The level should match the consequences: a test that triggers a capital injection deserves a far smaller α than a test comparing two revision methods. Reporting the p-value itself and letting the reader apply their own threshold is better practice than announcing a verdict against a borrowed one.
If p = 0.06, is there nothing there?
No. There is no physical difference between 0.049 and 0.051; the cliff is an artefact of the convention. A p of 0.06 means the data lean against H0 without clearing the agreed bar. The honest report gives the p-value, the estimated size of the effect and its confidence interval, and lets all three be read together.
Does a significant result mean the effect is large?
No, and this is the error with the greatest practical cost. Significance is roughly effect size divided by standard error, and the standard error falls as 1/√n. With a hundred thousand policies a difference in claim rate far too small to price on will be overwhelmingly significant. Significance tells you the effect is probably real. Only the estimate and its interval tell you whether it is worth acting on.
A worked example
A new revision method is tried on 36 students. Each student's mark improves by some amount; the improvements average 3.2 marks with a sample standard deviation of 7.5 marks. These figures were invented for this example. Does the method work?
Setting the test up
The question is whether the method improves marks, so the test is one-sided — decided before the data, because a method that made marks worse would be abandoned rather than reported.
The standard deviation is estimated from the data, so the reference distribution is t with 36 − 1 = 35 degrees of freedom.
t = (3.2 − 0) / 1.25 = 2.56
Route one: the p-value
The p-value is the upper tail of a t distribution with 35 degrees of freedom beyond 2.56:
About 7 chances in a thousand of seeing an improvement this large, or larger, if the method did nothing at all. That is below 0.01, so H0 is rejected at the 1% level.
Route two: critical values
| Level | Critical t35 | Observed 2.56 | Decision |
|---|---|---|---|
| 5% one-sided | 1.6895725 | beyond it | reject |
| 1% one-sided | 2.4377225 | beyond it | reject |
The same conclusion, reached without computing a tail probability.
The check that ties the two routes together. If the p-value is right, then the critical value for a test at exactly α = 0.0074710 must be the observed statistic itself. Looking up the 99.25290th percentile of t35 gives 2.5600000. It agrees to seven decimal places, which confirms both the p-value and the critical values above: they are one calculation read from opposite ends.
What the z route would have said
Treating 7.5 as if it were a known population value and using the normal gives p = 0.0052336 instead of 0.0074710. The conclusion is unchanged here, but the evidence has been overstated by about 30% — the t distribution's heavier tails are exactly the allowance for having estimated s from only 36 observations.
The other error: how likely was this test to miss?
Power asks the reverse question. Suppose the method truly improves marks by 3 on average. How often would a 5% one-sided test detect it?
For this part treat the standard deviation of 7.5 as known, so the reference distribution is normal. (The exact power of a t test needs a non-central t; the normal version is the standard calculation and is accurate enough to plan with.)
The test rejects when the observed mean improvement exceeds 1.645 × 1.25 = 2.05625 marks. If the truth is μ = 3, the sample mean is centred on 3 with the same standard error of 1.25, so a miss occurs when it falls below 2.05625:
β = Φ(−0.755) = 0.2251245
power = 1 − β = 0.7748755
Check it the other way, by computing the power directly rather than through β: power = P(Z > 1.645 − 3/1.25) = P(Z > −0.755) = 0.7748755. The two agree exactly.
So even against a real improvement of 3 marks, this test would miss it about 22% of the time. Power against other alternatives, same test:
| True improvement | Power |
|---|---|
| 2 marks | 0.4821 |
| 3 marks | 0.7749 |
| 5 marks | 0.9907 |
Against a 2-mark improvement the test is barely better than a coin toss. That is the practical content of power, and it is why "we found no significant effect" from a small study is a much weaker statement than it sounds.
To reach the conventional 80% power against a 3-mark improvement would need n = ((1.645 + 0.8416) × 7.5 / 3)² = 38.64, so 39 students — only three more than were used. Here the study was adequately sized. Against a 2-mark improvement the same formula demands 87.
Where this fits in CS1
This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.
Other CS1 topics explained
- Moment generating functions, and how to get moments out of them
- The central limit theorem, and when the normal approximation is safe
- Maximum likelihood estimation, step by step
- Confidence intervals: what the 95% actually means
- Simple linear regression: least squares, fitted by hand
- Generalised linear models: the three parts, and the link function
- Bayesian estimation, conjugate priors and the credibility formula inside them
- Conditional expectation, and the variance decomposition that comes with it
- Sampling distributions: what the sample mean and sample variance actually do
- The method of moments, and when maximum likelihood is worth the extra work
- The likelihood ratio test, and why the Wald and score tests exist too
- The chi-square goodness of fit test, and how to get its degrees of freedom right
- Contingency tables: testing whether two classifications are related
- Analysis of variance: comparing three or more means at once
- Multiple linear regression, and what controlling for another variable really does
Stuck on a term rather than the idea? The actuarial glossary defines the notation used above. If something here is wrong or unclear, tell us — these pages get corrected.