The model, and what is random in it
Simple linear regression explains one variable with another:
In words: each response is a straight-line function of x, plus an independent random error with mean zero and the same variance everywhere.
Read the roles carefully, because the asymmetry is the point. The xi are treated as fixed and known — chosen, not sampled. The randomness lives entirely in ε, and therefore in Y. This is why regressing Y on x and regressing x on Y give different lines: they are different models, not two views of one.
Three further assumptions are smuggled into that one line, and every regression diagnostic in the syllabus is a check on one of them: the relationship really is linear; the error variance σ² is the same at every x; and the errors are independent of one another.
Where the least squares line comes from
For any candidate line, each point has a vertical miss. Least squares picks the line making the total of the squared misses as small as possible:
Differentiate with respect to each parameter, set both to zero, and the two resulting equations solve to
where the three sums of squares and cross-products are
Two facts fall out of the algebra and both are useful checks. The formula for α̂ rearranges to ȳ = α̂ + β̂x̄, so the fitted line always passes through the point of averages. And the residuals always sum to exactly zero. Neither is an assumption; both are consequences of where the derivatives were set to zero, and a fit failing either has an arithmetic error in it.
Each sum also has an equivalent raw-total form, which is faster on a calculator and gives a completely independent route to the same number:
Splitting the variation
The total variation in y divides exactly into the part the line explains and the part it does not:
(total) = (explained by the regression) + (residual)
which gives the coefficient of determination:
In simple linear regression — one explanatory variable only — this is exactly the square of the correlation coefficient r, which gives a free check on both.
What R² does not tell you is worth more than what it does. It does not say the model is correct: a perfectly curved relationship can produce a high R² while the straight line is plainly the wrong shape. It does not say the relationship is causal. And it never falls when another explanatory variable is added, which is why multiple regression uses an adjusted version instead.
Is the slope real?
A slope estimated from data is never exactly zero, even when x explains nothing. The test asks whether it is further from zero than noise would produce. The estimated error variance is
with n − 2 degrees of freedom — two have been spent estimating the intercept and the slope. Then
Look at where Sxx sits: in the denominator of the standard error. Spreading the x values further apart makes the slope more precisely estimated, which is why a designed experiment chooses its x values at the extremes rather than bunching them.
Where marks are lost
Dividing by n − 1 instead of n − 2. Regression has estimated two parameters, not one.
Reading R² as proof of a good model. Always look at the residuals as well. A clear pattern in them — a curve, or a fan opening to the right — says the straight line or the constant-variance assumption has failed, however high R² is.
Extrapolating. The model is evidence about the range of x that was observed and nothing outside it. Predicting at an x far beyond the data is arithmetic without warrant.
Reversing the variables. Regressing x on y does not give the inverse of the first line. Decide which variable is the response before computing anything.
Treating association as causation. The algebra is identical whichever way the causal arrow points, or if it points from somewhere else entirely at both variables.
Questions students ask
Why squared errors rather than absolute ones?
Three reasons, in increasing order of importance. Squares are differentiable everywhere, so the minimum is found by calculus rather than by search. Squaring gives a unique answer where absolute deviations can give a whole range of equally good lines. And under the normal error assumption, minimising squared error is exactly maximum likelihood — least squares is not a separate principle, it is maximum likelihood wearing different clothes.
What does the intercept mean?
Formally, the expected response when x = 0. Often that is meaningless, because x = 0 is nowhere near the data — in the example below it would be a student studying zero hours, which was never observed. The intercept is usually a positioning constant rather than a quantity to interpret, and testing whether it is zero is rarely an interesting question.
How does this become a generalised linear model?
By relaxing two things at once. Ordinary regression requires the response to be normal and models the mean directly. A generalised linear model lets the response come from any exponential-family distribution — Poisson for counts, gamma for claim amounts — and models a function of the mean linearly instead. Ordinary regression is the special case where that distribution is normal and that function is the identity.
A worked example
Eight students report hours of study a week; their mock marks are recorded. Both columns were invented for this example.
| Hours x | 4 | 6 | 8 | 10 | 12 | 14 | 16 | 18 |
|---|---|---|---|---|---|---|---|---|
| Mark y | 40 | 49 | 51 | 60 | 61 | 70 | 71 | 78 |
The three sums, computed two ways
The means are x̄ = 88/8 = 11 and ȳ = 480/8 = 60, both exact. By deviations:
Syy = 400+121+81+0+1+100+121+324 = 1148
Sxy = 140+55+27+0+1+30+55+126 = 434
Now the same three from raw totals, which share no intermediate arithmetic with the above. With ∑x = 88, ∑y = 480, ∑x² = 1136, ∑y² = 29948 and ∑xy = 5714:
Syy = 29948 − 480²/8 = 29948 − 28800 = 1148
Sxy = 5714 − (88)(480)/8 = 5714 − 5280 = 434
All three agree. Everything below rests on these numbers, so this is the check worth doing first.
The fitted line
α̂ = 60 − 2.5833333 × 11 = 60 − 28.4166667 = 31.5833333
Each extra hour of study a week is associated with about 2.58 more marks.
Residuals, and the two checks they must pass
| x | y | Fitted | Residual |
|---|---|---|---|
| 4 | 40 | 41.9166667 | −1.9166667 |
| 6 | 49 | 47.0833333 | 1.9166667 |
| 8 | 51 | 52.2500000 | −1.2500000 |
| 10 | 60 | 57.4166667 | 2.5833333 |
| 12 | 61 | 62.5833333 | −1.5833333 |
| 14 | 70 | 67.7500000 | 2.2500000 |
| 16 | 71 | 72.9166667 | −1.9166667 |
| 18 | 78 | 78.0833333 | −0.0833333 |
First check: the residuals sum to exactly 0, as least squares guarantees.
Second check: squaring and adding them gives SSres = 26.8333333. The shortcut formula, which never touches an individual residual, gives
The two agree to seven decimal places. A third confirmation: β̂Sxy + SSres = 1121.1666667 + 26.8333333 = 1148 = Syy, so the variation splits exactly as the decomposition requires. And the line passes through the averages: at x = 11 the fitted value is exactly 60.
How much is explained
Check it through the correlation instead:
r² = 0.9766260
Identical. About 97.7% of the variation in these marks is accounted for by study hours — which, being invented data, says more about how the example was built than about students.
Is the slope real?
se(β̂) = 2.1147629/√168 = 2.1147629/12.9614814 = 0.1631575
t = 2.5833333/0.1631575 = 15.8333715 on 6 degrees of freedom
The two-sided p-value is about 0.000004. The slope is firmly non-zero.
Check by a second statistic. There is an independent test of the same thing, through the correlation: t = r√(n−2)/√(1−r²). Substituting r = 0.9882439 gives 15.8333715 — the same value to seven decimal places. Testing "the slope is zero" and testing "the correlation is zero" are the same test, and the arithmetic proves it.
A 95% interval for the slope, with t6, 0.975 = 2.4469119:
Zero is nowhere near the interval, agreeing with the test as duality requires.
One prediction that is fine, and one that is not
At 11 hours a week, inside the observed range, the model predicts 31.5833333 + 2.5833333 × 11 = 60.0 marks. Reasonable.
At 30 hours a week it predicts 109.1 marks — on a paper marked out of 100. The arithmetic is correct and the answer is nonsense, because 30 hours is far outside the 4-to-18 range the data cover and nothing in the fit is evidence about it. This is the clearest possible demonstration of why extrapolation is not a prediction.
Where this fits in CS1
This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.
Other CS1 topics explained
- Moment generating functions, and how to get moments out of them
- The central limit theorem, and when the normal approximation is safe
- Maximum likelihood estimation, step by step
- Confidence intervals: what the 95% actually means
- Hypothesis testing and p-values, without the folklore
- Generalised linear models: the three parts, and the link function
- Bayesian estimation, conjugate priors and the credibility formula inside them
- Conditional expectation, and the variance decomposition that comes with it
- Sampling distributions: what the sample mean and sample variance actually do
- The method of moments, and when maximum likelihood is worth the extra work
- The likelihood ratio test, and why the Wald and score tests exist too
- The chi-square goodness of fit test, and how to get its degrees of freedom right
- Contingency tables: testing whether two classifications are related
- Analysis of variance: comparing three or more means at once
- Multiple linear regression, and what controlling for another variable really does
Stuck on a term rather than the idea? The actuarial glossary defines the notation used above. If something here is wrong or unclear, tell us — these pages get corrected.