Maximum likelihood estimation explained

CS1 · Actuarial Statistics · 9 min read

The question the method answers

You have a model with an unknown parameter and you have some data. There are infinitely many values the parameter might take, and each one implies a different probability of seeing the data you actually saw. Maximum likelihood asks the obvious question: which value of the parameter makes the observed data most probable? That value is the estimate.

It is worth noticing how modest this is. There is no appeal to unbiasedness, no minimising of squared error, no prior belief. The whole method is one sentence, and nearly every estimator in the syllabus that was not simply guessed comes out of it.

Likelihood is not probability

The function is the same algebra read in the opposite direction, and getting this straight prevents a whole class of confusion later.

A density or mass function f(x | θ) is read as a function of the data x, with the parameter θ fixed. It describes which data are likely. The likelihood is the same expression read as a function of θ, with the data fixed at what was observed:

L(θ) = f(x₁, …, x_n | θ) = ∏_i f(x_i | θ) for independent observations

In words: the probability of getting exactly this data set, written as a function of the unknown.

Two consequences follow immediately. First, the likelihood is not a probability distribution for θ — it does not integrate to one over θ and it is not trying to. Second, only its shape matters, so any factor not involving θ can be thrown away. In a binomial likelihood the ‘n choose x’ term is a constant: dropping it moves the whole curve up or down and leaves the position of the maximum untouched.

The recipe

Five steps, and the fourth is the one that gets skipped.

1. Write the likelihood as a product over the observations.

2. Take logarithms. ℓ(θ) = ln L(θ), which turns the product into a sum.

3. Differentiate and solve. Set dℓ/dθ = 0. This equation is called the score equation.

4. Check it is a maximum. The second derivative must be negative at the solution.

5. State the estimator in terms of the data, and give it a hat.

Step 2 deserves its own justification, because students often suspect it of being a convenience. It is more than that. A product of n small numbers underflows to zero on a calculator for even moderate n, so the log is a numerical necessity as much as an algebraic one. And because the logarithm is strictly increasing, the θ that maximises ℓ is exactly the θ that maximises L — nothing is given up by working with the log.

A warm-up with round numbers

Ten policies are observed and three of them make a claim. Model each as an independent Bernoulli with claim probability p. Dropping the binomial coefficient, which does not involve p:

L(p) = p³ (1 − p)⁷
ℓ(p) = 3 ln p + 7 ln(1 − p)
ℓ′(p) = 3/p − 7/(1 − p)

Setting that to zero gives 3(1 − p) = 7p, so 3 = 10p and p̂ = 0.3. The estimate is the sample proportion, which is reassuring: a method that returned anything else here would be suspect. The second derivative is −3/p² − 7/(1−p)², negative for every p in (0, 1), so it is certainly a maximum.

What the method gives you free

Invariance

If θ̂ is the maximum likelihood estimate of θ, then g(θ̂) is the maximum likelihood estimate of g(θ), for any function g. So having estimated a rate, you have also estimated the mean, which is one over the rate, with no further work. No other estimation method in the syllabus has this property, and it saves real time in an exam.

A standard error, from the information

For large samples the estimator is approximately normal around the true value, with a variance given by the reciprocal of the Fisher information:

I(θ) = −E[d²ℓ/dθ²] and θ̂ ≈ N(θ, 1/I(θ))

In words: the information is how sharply curved the log-likelihood is at its peak. A sharp peak means the data rule out nearby parameter values firmly, so the estimate is precise; a flat peak means many values fit about as well, so it is not. Curvature is precision, and the standard error is one over the square root of it.

This is also the point at which the Cramér-Rao lower bound enters: no unbiased estimator can have a variance smaller than 1/I(θ). So the maximum likelihood estimator is not merely reasonable in large samples, it is asymptotically as good as any unbiased estimator can be.

Two versions of the information are used, and both are acceptable. Expected information takes the expectation, giving a formula in θ which is then evaluated at θ̂. Observed information simply substitutes the data into −d²ℓ/dθ² without taking an expectation. For the exponential family they often coincide.

Where marks are lost

Skipping the second-derivative check. A stationary point is not automatically a maximum, and in a two-parameter problem it can be a saddle. One line of work; it is sometimes worth a mark on its own.

Reaching for calculus when the parameter is a boundary. For a uniform distribution on (0, θ), the likelihood is θ⁻ⁿ for every θ at least as large as the biggest observation, and zero below it. It is decreasing wherever it is positive, so it is maximised at the smallest permissible θ, which is the largest observation. Differentiating gives nothing at all, because the maximum is at an edge of the range rather than at a turning point. Always ask whether the likelihood is differentiable before differentiating it.

Assuming the estimator is unbiased. It usually is not. The maximum likelihood estimate of a normal variance divides the sum of squared deviations by n, not by n − 1, and is therefore biased low. Maximum likelihood promises good behaviour as n grows, not unbiasedness at every n.

Differentiating the likelihood rather than the log-likelihood. Legitimate, and it gives the same answer, but a product rule across n factors is a long way to the same place.

Treating the asymptotic standard error as exact. It is a large-sample result. With ten observations, as in the worked example below, the resulting interval is indicative rather than precise, and saying so is part of a complete answer.

Questions students ask

Why does taking logs not change the answer?

Because the logarithm is strictly increasing: if L(θ₁) > L(θ₂) then ln L(θ₁) > ln L(θ₂), with no exceptions. The transformation preserves the ordering of every pair of values, so it preserves which value is largest. It changes the height of the peak but not its location, and only the location is being estimated.

Is maximum likelihood always the best method?

In large samples it is hard to beat. In small samples it can be clearly beaten, because it may be badly biased, and for some models the likelihood has no maximum or several. The method of moments is sometimes preferred for being simpler and more robust to a misspecified model; Bayesian estimation is preferred when genuine prior information exists and the data are thin. Maximum likelihood is the default, not a law.

What if the score equation cannot be solved by hand?

That is the normal situation for anything beyond textbook models, and it changes nothing about the principle. The maximum is found numerically instead — iteratively climbing the log-likelihood until the gradient is effectively zero. The iteratively reweighted least squares algorithm that fits a generalised linear model is exactly this, so a student who has understood this page has understood how GLM fitting works.

A worked example

An insurer records how many days ten claims took to settle. These ten figures were invented for this example.

4, 7, 2, 11, 5, 9, 3, 14, 6, 9 (days)

Settlement delays are modelled as exponential with rate λ, so the density is f(x) = λ e^−λx for x > 0. Estimate λ, confirm the estimate is a maximum, and attach a standard error.

Step 1: the likelihood

The observations total 4 + 7 + 2 + 11 + 5 + 9 + 3 + 14 + 6 + 9 = 70 days over n = 10 claims, so the sample mean is 7.0 days. Multiplying the ten densities:

L(λ) = ∏ λ e^−λx_i = λ¹⁰ e^−70λ

Only two features of the data survive into the likelihood: how many observations there were, and what they summed to. The individual values do not appear again, which is the exponential family at work.

Steps 2 and 3: logs, then differentiate

ℓ(λ) = 10 ln λ − 70λ
ℓ′(λ) = 10/λ − 70 = 0 → λ̂ = 10/70 = 0.1428571 per day

Step 4: confirm it is a maximum, two ways

Algebraically, ℓ″(λ) = −10/λ², which is negative for every positive λ. So the stationary point is a maximum, with no further argument needed.

Numerically, evaluate the log-likelihood on either side and confirm the peak sits where the algebra says:

λ	ℓ(λ)
0.10	−30.0258509
0.13	−29.5022083
0.1428571	−29.4591015
0.16	−29.5258146
0.20	−30.0943791

Every neighbouring value is lower, and the curve falls away on both sides. The two checks agree.

Step 5: the estimate, and the mean by invariance

The rate is estimated at λ̂ = 0.1428571 claims settled per day. Since the mean of an exponential distribution is 1/λ, invariance gives the maximum likelihood estimate of the mean settlement delay immediately:

mean̂ = 1/λ̂ = 70/10 = 7.0 days

Exactly the sample mean. No separate derivation was needed.

The standard error, by two independent routes

Route one: Fisher information. Since ℓ″(λ) = −n/λ² involves no data at all, the expectation changes nothing and expected and observed information coincide:

I(λ) = n/λ² = 10 / (0.1428571)² = 10 × 49 = 490
se(λ̂) = 1/√490 = 0.0451754

Route two: the delta method, through the sample mean. A different argument entirely. The variance of one exponential observation is 1/λ² = 49, so the variance of the sample mean is 49/10 = 4.9 and its standard error is √4.9 = 2.2135944 days. Now λ̂ = 1/x̄, and the derivative of 1/x̄ is −1/x̄² = −1/49, so

se(λ̂) ≈ 2.2135944 / 49 = 0.0451754

The two routes agree to seven decimal places. One came from the curvature of the log-likelihood and the other from propagating the uncertainty in the sample mean through a reciprocal; that they meet is a genuine check, not a restatement.

An approximate 95% interval

0.1428571 ± 1.96 × 0.0451754 = 0.1428571 ± 0.0885438

which is (0.0543, 0.2314) per day. Inverting both ends gives a range for the mean settlement delay of roughly 4.3 to 18.4 days.

That is a very wide range, and it should be reported as such rather than tidied away. Ten observations is a small sample, the normal approximation behind the interval is a large-sample result, and the inverted interval is visibly asymmetric because the reciprocal is a curved function. The arithmetic is right; the honest conclusion is that ten claims pin down a settlement rate only loosely.

Where this fits in CS1

This page explains one idea from CS1 Actuarial Statistics. The full CS1 syllabus, with what each part is examined on, sits on the course page, and past-paper style mock exams with full solutions are where you find out whether the idea has stuck under time pressure.

Maximum likelihood estimation, step by step