The average settles and the wobble does not

Add up many independent copies of the same random quantity. Two things are then true about the sum, and they are usually stated one after the other as though the second improves on the first.

The law of large numbers says the average of the copies settles onto the mean. The central limit theorem says the sum, once centred and scaled, has a bell-shaped distribution. The first sounds like a rough statement and the second like a refinement of it, and that reading is wrong in a way worth correcting: they are statements about the same sums under two different scalings, and each says nothing at all about the other’s scaling.

The left panel is the law of large numbers. The right panel is the central limit theorem. Nothing was changed between them except what the horizontal axis was divided by.

The arithmetic that decides it #

The reason there are exactly two interesting scalings is a single line of algebra about variances.

Variances of independent quantities add. So a sum S of n copies, each with mean μ and standard deviation σ, has mean nμ and standard deviation σ√n. That is the whole input, and everything follows from asking what happens when S − nμ is divided by something.

Divide by n, and the standard deviation becomes σ/√n, which goes to zero. The distribution collapses onto a point: the average settles.

Divide by √n, and the standard deviation is σ exactly, for every n. Nothing collapses and nothing spreads; the width is fixed, and what remains is a question about shape.

Divide by anything growing faster than √n and the result collapses; anything slower and it spreads without limit. So √n is the one scaling under which a limit shape can exist at all, and the central limit theorem is the answer to the question of what that shape is. The exponent is forced before any theorem is proved.

What each theorem actually claims #

Stated carefully, they are answers to different questions.

The law of large numbers is a statement about a single number: the average is eventually close to μ and stays close. It says nothing about the distribution of the deviation, only that it shrinks. It holds under remarkably weak conditions — a finite mean is enough for the strong form, with no variance required at all.

The central limit theorem is a statement about a distribution: the deviation, magnified by √n, has a distribution converging to a specific bell shape. It says nothing about any particular run of the experiment, and it needs a finite variance.

The gap between them shows up in what each can be used for. To promise that an estimate is close, the law of large numbers is enough, and it delivers no error bar. To put a number on the error bar, the central limit theorem is needed, and it delivers one only in the limit.

The rate, and why it is 1/√n #

The band in that figure closes like σ/√n, and that rate is the practical content of both theorems put together.

It is a harsh rate. Ten times the samples buys three times the precision; a hundred times buys ten. Nothing about the method changes the exponent — it comes from variances adding, which comes from independence, and no cleverness in the sampling touches it.

The rate is what makes estimating π by dropping needles an amusement rather than a method. The needle estimate has a standard error of about 0.8 divided by the square root of the number of needles, so three digits of π needs on the order of a million drops, and each extra digit costs a hundredfold. Every simulation-based estimate anywhere pays the same tax.

What can be improved is the constant. Reducing the variance of what is being averaged — by arranging the sampling more cleverly, or by subtracting off a quantity whose average is known — multiplies the error by a smaller number without touching the exponent. That is what the whole variance-reduction literature is about, and its ceiling is set by an exponent nobody can move.

The shape, and where it comes from #

The bell shape in the right-hand panel is not put there. It arrives.

The Galton board is the physical version of the same computation: each ball’s position is a sum of independent steps, and the heap is the distribution of the sum. The first figure on this page computes that distribution exactly, by convolving the one-step distribution with itself, so the curves there are the distribution rather than a sample from it.

Two features of the convergence are worth naming because they are not obvious from a picture.

The limit does not depend on the one-step distribution. A coin, a die, a lopsided quantity with five values — all of them, once centred and scaled by √n, converge to the same bell. Only two numbers of the original distribution survive: its mean, which is centred away, and its variance, which sets the width. Everything else is erased.

The convergence is in distribution and not pointwise. The claim is about the probability of landing in an interval, not about individual outcomes. For a sum of coin flips the possible values are integers of one parity, so the exact distribution is a spiky comb no matter how large n is; what converges is the mass in a window, and the picture makes that legible only because it is drawn as a density.

Drawn as densities, and why that matters #

The two panels of the first figure would look identical if the height plotted were the probability of each individual value, and that is a trap worth being explicit about.

A sum of n coin flips takes about n values whichever way it is scaled, so every individual probability is of size about 1/√n in both panels, and both sets of curves would shrink. What differs between the panels is how tightly the values are packed along the axis, and dividing the probability by the gap between neighbouring values — turning a probability into a density — is what puts that difference on the page.

Once that is done the contrast is stark. On the left the density climbs — 0.75, then 1.57, then 3.18 at the peak for the three curves drawn — because the same total probability is being squeezed into a narrowing interval. On the right it settles on 0.399, which is the height of the standard bell curve, and the figure asserts that it does.

The 0.399 is 1/√(2π), and the appearance of π in a question with no circle in it is the standing oddity of the subject. It comes from the integral that normalises the bell curve, and no amount of staring at a Galton board produces it.

The same two scalings, seen as a walk #

A random walk is the sum before it is divided by anything, and it makes the two scalings into two pictures of one path.

The position after n steps is a sum of n independent steps, so everything above applies unchanged. Divide the position by n — asking for the average step — and the answer settles on zero: a walk’s average step tends to nothing, which is the law of large numbers and is the least interesting thing about a walk. Divide by √n and the picture stops shrinking, and what is left is the shape the walk’s endpoint takes.

The two curves drawn over the paths are ±√n, and the walks cross them repeatedly. That is the correct behaviour and worth dwelling on: the typical distance grows like √n, and typical is not a ceiling. Exactly how far beyond the curves a walk strays, infinitely often, is settled by a third theorem sharper than either of the two here.

What a bound says when a limit will not #

Before either limit theorem is available, an inequality is, and it is worth seeing what it gives on the same quantity.

Chebyshev’s inequality says the average of n copies is within kσ/√n of the mean with probability at least 1 − 1/k², and that statement is true for every n, not merely in the limit. It is the shortest proof of the weak law of large numbers, it needs no shape, and it is loose by a factor of about forty against what the bell approximation gives.

So there are three statements about the same average and they sit in a clear order. The inequality is always true and weak. The law of large numbers is a limit and says only that the deviation vanishes. The central limit theorem is a limit and says what the deviation looks like, at the cost of the strongest hypotheses of the three. A practitioner needing a guarantee at a specific n reaches for the first; a practitioner needing an error bar reaches for the third and hopes n is large enough.

The same three-way choice appears wherever a random quantity has to be controlled. Counting how long until every coupon turns up has an exact answer, an inequality, and a limit shape, and which of the three is used depends entirely on what the answer is for.

Where it fails #

Both theorems need hypotheses and both fail without them, in ways that a simulation will not reveal.

Without a finite variance, the scaling is wrong. For quantities with heavy tails the sum has to be divided by n to some other power to have a limit at all, and the limit is not a bell curve but a stable distribution with its own tail exponent. Financial returns and the sizes of natural disasters are the standard examples, and the practical failure is that a sample of any size looks well-behaved right up until an outcome arrives that is larger than everything seen so far put together.

Without a finite mean, nothing settles. The Cauchy distribution’s average over n copies has exactly the same distribution as one copy, whatever n is. Averaging does not help at all, and a run of the experiment produces a plot that looks as though it is about to converge and then jumps.

Without independence, the variances do not add. Correlated pieces can have a sum whose standard deviation grows like n rather than √n, in which case the average does not settle and the whole apparatus is inapplicable. This is the failure mode most often met in practice, because independence is an assumption about the world rather than about the mathematics.

Independence is also weaker than it needs to be in one direction and stronger than it looks in another. The theorems here survive a good deal of dependence — a process that forgets where it came from quickly enough behaves like a sum of independent pieces, with an effective sample size smaller than the real one — while a mild-looking correlation that persists at every lag destroys the rate entirely. Deciding which case is at hand is a question about the process and not about probability, and it is where most misapplications of these theorems begin.

What it costs #

The exact distributions drawn here are computed by convolution: start with the one-step distribution and combine it with itself n times, keeping every reachable value and its probability. For a sum of 64 coin flips that is 65 values and a few thousand multiplications — cheap, exact, and free of sampling error.

That approach stops working when the one-step distribution has many values, since the number of reachable sums multiplies. The standard alternative is to work with a transform: the characteristic function of a sum is the product of the characteristic functions, so convolution becomes multiplication, and the whole central limit theorem is proved in that language in half a page. What the transform costs is that the answer comes back as a function rather than as a list of probabilities, and getting the probabilities back is another transform.

The proof by transforms is the standard one for a reason worth stating: it explains why only the mean and the variance survive. Taking logarithms of the transform turns the product into a sum, and expanding it shows that everything beyond the second term is divided away by the scaling. The erasure of the original distribution’s shape happens visibly, in one line of a series expansion.

What the picture cannot show #

Both panels are drawn at three values of n, and both theorems are statements about the limit. Nothing in a picture of n = 64 establishes what happens at 64 million, and the convergence in the right-hand panel is deliberately slow to look at: the curve for four terms is already close to the curve for sixty-four, which flatters the theorem.

The figures also draw one starting distribution at a time. That the same bell arrives whatever the starting distribution — the fact that makes the theorem astonishing rather than merely true — needs several panels or a claim, and this page makes it as a claim.

And nothing here shows the failures. A heavy-tailed sum drawn at these sizes looks like the well-behaved case; the divergence lives in the tail, out where a drawn curve has already reached the axis.

The ladder from here #

Rungs above: the strong law of large numbers, which promises that almost every individual run converges rather than that the distribution does, and needs a different proof. The Berry–Esseen theorem, which puts a rate on the central limit convergence — the error is of size about 1/√n, so the approximation improves as slowly as the estimate does. Stable distributions and the generalised limit theorem for heavy tails. The law of the iterated logarithm, which says exactly how far a running average keeps straying from its limit and is sharper than either theorem here. Large deviations, where the probability of a sum being far out decays exponentially and the exponent is a computable function. And the martingale central limit theorem, which drops independence for something weaker.

The shape of the idea #

The manoeuvre this essay is built on is worth extracting: when a quantity does not converge, divide it by something and ask whether the result does.

Choosing the divisor is the whole art. Divide by too much and the answer is zero, which is true and useless; divide by too little and it is infinite; and the one scaling in between is where a limit shape lives. The scaling exponent is found before the limit is, by counting how the size of the thing grows, and here it is one half because variances add.

The same manoeuvre appears wherever a limit is extracted from something that grows. It is how the derivative is defined — a difference that goes to zero, divided by another that goes to zero at just the right rate — and how a fractal’s dimension is measured, by asking which power of the box size keeps the count finite. In every case the exponent is the interesting number and the limit is the reward for having got it right.