Conditional expectation explained

CS1 · Actuarial Statistics · 11 min read

The problem it solves

A motor portfolio is not one population. It is careful drivers and careless ones, mixed together. A cohort of students is those who have sat the paper before and those who have not. A year of claims is ordinary losses with the occasional large one hidden among them.

And you almost always know the parts better than you know the whole. You know the claim rate for each driver class, because that is how the pricing was built. You know the average mark for each group, because that is how the marking was done. What you are asked for is a statement about the whole: one mean, one variance, one premium.

Conditional expectation is the machinery for assembling the parts into the whole. Its second job is more interesting, and it is the reason the topic sits where it does in CS1: it splits the variation of the whole back into the part that happens inside the groups and the part that comes from the groups differing. A generalised linear model's overdispersion, a credibility premium's weight and the variance of aggregate claims are all that same split, used three times.

The definition, and the one thing to be careful about

Fix a value of Y. Then E[X | Y = y] is an ordinary average: the mean of X over the part of the population where Y takes that value. Nothing new is happening. A conditional mean is a mean of a sub-population.

Now take the value away. The symbol E[X | Y], with no value named, is a random variable, not a number. It is the rule that says "whichever group you land in, report that group's mean". Before you know the group, that report is random, because the group is random.

In words: E[X | Y = y] is a number, one per group. E[X | Y] is the whole collection of those numbers, carried around with the probabilities of landing in each group.

That distinction is the entire topic. Every result below is a statement about the random variable, and every mistake students make here comes from reading it as a number.

A warm-up with round numbers

Two groups sit a test. Ten candidates in the first average 60 marks; thirty in the second average 70. What does the combined group average?

It is not 65. That would be averaging the two group means as though the groups were the same size. The second group is three times the size, so it gets three times the say:

(10 × 60 + 30 × 70) / 40 = (600 + 2100) / 40 = 2700 / 40 = 67.5

The weights are the probabilities of landing in each group: ¼ and ¾. Written that way, ¼(60) + ¾(70) = 15 + 52.5 = 67.5, which is exactly "the expectation of the group mean" — the expectation of E[X | Y]. The arithmetic is familiar. The point is the reading.

The tower property

E[X] = E[ E[X | Y] ]

In words: average within each group first, then average those group averages, weighting each by how likely that group is. You get the same answer as averaging everything in one go.

It looks like a tautology and it nearly is — but as a working tool it is the licence to break a hard expectation into two easy ones. When the inner average is something you know (a Poisson mean, a claim size mean, a premium) and the outer one is a short weighted sum, a calculation that had no obvious route becomes two lines.

The same statement for probabilities is the law of total probability, which you have used since the first probability unit: P(A) = Σ P(A | B_i) P(B_i). The tower property is that law with a random variable in place of the event.

The variance decomposition

Variance does not obey a tower property, and here is why: the spread of the whole population has two sources, not one. Members of the same group differ from each other, and the groups differ from each other. Both make the whole look more spread out, and you have to count both:

Var(X) = E[ Var(X | Y) ] + Var( E[X | Y] )

In words: the average of the within-group variances, plus the variance of the group means. Within plus between.

Check the extremes, because they fix which term is which. If every group had an identical mean, the second term would vanish and the whole variance would be the average of the within-group variances. If every group had zero internal variation — everyone in a group identical — the first term would vanish and all the variation would be between the groups. Any real population is somewhere between those two, and the formula is just the statement that the two effects add.

One consequence to carry away: ignoring the groups never makes a population look less variable. The between term cannot be negative, so Var(X) ≥ E[Var(X | Y)] always. A model that pretends a mixed portfolio is homogeneous understates its risk, and the understatement is exactly the variance of the thing it ignored.

Where it is used after this topic

Aggregate claims. Total claims is a random number of random amounts. Condition on the number, and the two layers come apart: the mean is E[N]E[X] and the variance is E[N]Var(X) + Var(N)E[X]² — within plus between, with the claim count as the grouping variable. The worked example below does it.

Credibility. The weight given to a risk's own experience is a ratio built from exactly these two terms: variation within a risk against variation between risks. When the between-risk variation is large, the groups are genuinely different and a risk's own data deserves weight. That sentence is the whole intuition of credibility theory, and it is a sentence about this decomposition.

Overdispersion. A Poisson distribution has variance equal to its mean. Claim counts from a mixed portfolio almost never do: the between term adds to the variance and nothing adds to the mean, so the variance comes out larger. That is why real count data is overdispersed, and why the quasi-Poisson and negative binomial models exist.

Where marks are lost

Treating E[X | Y] as a number. If it were a number, Var(E[X | Y]) would be zero and the decomposition would collapse to the wrong formula. Whenever a variance of a conditional mean appears, stop and ask what is random: the answer is the group.

Dropping the between term. The most common version of this topic's error is writing Var(X) = E[Var(X | Y)] and moving on. It is always an understatement, which is the worst direction for an answer about risk to be wrong in.

Weighting the group means equally. 65 instead of 67.5. The weights are probabilities, and probabilities are rarely equal.

Mixing up which term carries which operator. It is E of a Var plus Var of an E, in that order. Var(Var(X | Y)) is not a thing that appears anywhere, and if your working has produced one, a line has gone wrong above it.

Conditioning on the wrong variable in a compound sum. For aggregate claims you condition on the number of claims, not on the amounts. Conditioning on the amounts leaves the hard part of the problem untouched.

Questions students ask

Why is E[X | Y] called a random variable when it looks like a formula?

Because it is a function of Y, and Y is random. Substitute a particular group and you get a number; leave Y as it is and you have a quantity whose value is not yet determined. Every function of a random variable is a random variable — Y² is, e^Y is, and so is "the mean of X in Y's group".

Is this the same as the sums of squares in analysis of variance?

Yes, and seeing that is worth a great deal. Analysis of variance splits an observed total sum of squares into a between-groups part and a within-groups part. The decomposition on this page is the same split written for the population rather than for a sample: the between term becomes the between-groups sum of squares, the within term becomes the residual. One is the theory, the other is the arithmetic you do to a data set.

What if the grouping variable is continuous?

Nothing changes. The sum over groups becomes an integral over values of Y, and both formulae hold exactly as written. The discrete case is the one to learn on, because you can see the weights; the continuous case is where the formula earns its keep, since the integral version of a direct calculation is usually the one you cannot do.

A worked example

A motor book has two driver classes. Every figure here is invented for this example; it is not taken from any portfolio. Sixty per cent of policies are experienced drivers whose claim count is Poisson with mean 0.10 a year; the other forty per cent are new drivers, Poisson with mean 0.25. Pick one policy at random and let N be its number of claims.

The mean, in two stages

The inner average is given to you, one value per class. The outer average weights them by the class shares:

E[N] = 0.6(0.10) + 0.4(0.25) = 0.06 + 0.10 = 0.16

So the book averages 0.16 claims per policy per year. Note that this is not the midpoint of 0.10 and 0.25, which would be 0.175; the larger class pulls it down.

The variance, by decomposition

Within a class the count is Poisson, and a Poisson variance equals its mean, so the within-class variances are 0.10 and 0.25 — the same two numbers as the means. The within term is therefore the same weighted sum as before:

E[Var(N | class)] = 0.6(0.10) + 0.4(0.25) = 0.16

The between term is the variance of the two class means around the overall 0.16:

Var(E[N | class]) = 0.6(0.10 − 0.16)² + 0.4(0.25 − 0.16)²
= 0.6(0.0036) + 0.4(0.0081) = 0.00216 + 0.00324 = 0.00540

Var(N) = 0.16 + 0.00540 = 0.16540

Route two: the raw second moment

Forget the decomposition and do it the long way. For a Poisson with mean λ, E[N²] = λ + λ². Average that over the two classes:

E[N²] = 0.6(0.10 + 0.0100) + 0.4(0.25 + 0.0625)
= 0.6(0.11) + 0.4(0.3125) = 0.066 + 0.125 = 0.191

Var(N) = 0.191 − (0.16)² = 0.191 − 0.0256 = 0.1654

The two routes agree exactly, as they must. The decomposition is not a different answer, it is the same answer with the two sources of variation kept apart instead of added up and then corrected.

A third check, for completeness: write down the actual distribution of N for the mixed book, P(N = n) = 0.6 e^−0.10(0.10)ⁿ/n! + 0.4 e^−0.25(0.25)ⁿ/n!, and sum n P(N = n) and n² P(N = n) term by term. The sums give 0.16 and 0.191 to the last decimal place carried.

What the between term is telling you

Var(N) / E[N] = 0.16540 / 0.16 = 1.03375

The variance is 3.375 per cent larger than the mean, although every single policy in the book is Poisson. Nothing about any individual driver is overdispersed; the overdispersion is entirely the between-class term, and it would disappear if you fitted each class separately. This is the clearest possible answer to "why is real claim count data never quite Poisson?"

The same effect shows in the zero-claim probability. For the mixture:

P(N = 0) = 0.6 e^−0.10 + 0.4 e^−0.25 = 0.6(0.9048374) + 0.4(0.7788008)
= 0.5429025 + 0.3115203 = 0.8544228

A single Poisson with mean 0.16 would give e^−0.16 = 0.8521438. The mixture has slightly more claim-free policies and slightly more multi-claim policies than the Poisson that matches its mean. Heterogeneity fattens both ends.

The same decomposition one layer up: aggregate claims

Now let S be the total amount claimed on a policy in a year. The number of claims N is Poisson with mean 3, and each claim amount is independent of the rest with mean 20 and standard deviation 15, in thousands of rupees. Invented figures again.

Condition on N. Given N = n the total is a sum of n claim amounts, so its conditional mean is 20n and its conditional variance is 225n. Feed those into the two formulae:

E[S] = E[20N] = 20(3) = 60 thousand

E[Var(S | N)] = E[225N] = 225(3) = 675
Var(E[S | N]) = Var(20N) = 400(3) = 1200
Var(S) = 675 + 1200 = 1875

So the standard deviation of the annual total is √1875 = 43.3012702, or about ₹43,301.

Check it against the compound Poisson result. For a compound Poisson the variance is λ E[X²], and E[X²] = 225 + 400 = 625, so 3(625) = 1875. The two agree. Reading the first calculation next to the second shows what that textbook formula actually is: the two halves of the decomposition, already added together.

One last observation worth the line. The coefficient of variation of a single claim is 15/20 = 0.7500000. For the annual total it is 43.3012702/60 = 0.7216878 — barely smaller. Averaging three claims would have cut relative variability by √3 if the number of claims were fixed; it does almost nothing here, because most of the uncertainty is in how many claims there will be, not in how large each one is. That is the between term, doing the damage.

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

Conditional expectation, and the variance decomposition that comes with it