The nearest point is not the best guess

The nearest point of a flat thing turned an unanswerable question into a shadow. A set of measurements that no line passes through has, nevertheless, a nearest point on the space of lines, and that nearest point is the orthogonal projection of the data — the least-squares fit. Its whole justification was geometric: among all the points the model allows, report the one closest to what was seen.

This essay takes the simplest case of that rule, where the model allows everything. Suppose there are $p$ unknown numbers $\theta_1, \ldots, \theta_p$ and each is measured once, with an error that is normally distributed, of mean nought and variance one, independently of the others. The measurements form a point $X$ in $p$-dimensional space, and the unknowns form a point $\theta$. The model says $\theta$ could be anywhere, so the nearest allowed point to $X$ is $X$ itself. Report the measurements. It is hard to see what else anybody could do, and for two centuries nobody did anything else.

In 1956 Charles Stein proved that in three or more dimensions this is a mistake — that there is another rule which, measured by the total squared error, is better than reporting $X$ whatever $\theta$ is. In 1961 he and Willard James wrote one down.

A rule that cannot lose, and then loses #

The measurement $X$ is about as respectable as an estimate can be. It is unbiased: its average, over repeated measurement, is exactly $\theta$. It is the maximum-likelihood estimate. It is the least-squares estimate. Its expected squared error, $\mathbb{E}\,|X - \theta|^2$, is the sum of the $p$ variances, which is exactly $p$, and that number does not depend on $\theta$ at all — the flat line in the figure. And in one dimension it is provably impossible to beat: any rule that does better than $X$ for some value of $\theta$ does worse for another.

James and Stein’s rule is

$$\hat\theta_{\text{JS}} = \left(1 - \frac{p - 2}{|X|^2}\right) X .$$

It takes the measurement and pulls it straight towards the origin, by a fraction that depends on how long the measurement is: a short $X$ is pulled hard, a long one hardly at all. Its expected squared error can be computed exactly. The squared length $|X|^2$ is a sum of squares of normal variables with means $\theta_i$, a non-central chi-square, and the error of the rule works out to

$$\mathbb{E}\,|\hat\theta_{\text{JS}} - \theta|^2 = p - (p - 2)^2\, \mathbb{E}\!\left[\frac{1}{|X|^2}\right],$$

where the last expectation is a sum of terms $1/(p - 2 + 2k)$ weighted by a Poisson distribution of mean $|\theta|^2/2$. The figure sums that series. Because the subtracted term is positive, every curve lies below $p$, at every $\theta$.

The gain depends on where the truth is. When $\theta$ is at the origin the error is exactly $2$, in every dimension, so with ten unknowns the shrunk estimate has a fifth of the measurement’s error and with twenty-five unknowns a twelfth. When $\theta$ is far from the origin the gain fades towards nothing. But it never becomes a loss, and that one-sidedness is what makes the result a paradox rather than a trade-off. A statistician who uses $X$ is choosing a rule that is beaten everywhere, which in the language of the subject makes $X$ inadmissible.

Why the measurement is too long #

The first step to understanding the rule is to notice that $X$ is systematically too long. Its expected squared length is

$$\mathbb{E}\,|X|^2 = |\theta|^2 + p,$$

since each coordinate contributes $\theta_i^2$ plus a variance of one. The measurement points roughly the right way and overshoots. In one dimension the overshoot is a single unit of variance against whatever $\theta^2$ is; in a hundred dimensions it is a hundred units, and unless the truth is very large, most of the measurement’s length is noise.

The geometry is worth seeing directly, because it is what turns the formula into a picture. The noise $X - \theta$ is a random vector with independent coordinates, and in many dimensions a random vector is nearly perpendicular to any fixed direction. That is the phenomenon of a sphere that is nearly all equator: measured from any pole, almost all of a high-dimensional sphere’s surface lies within a thin band around the equator, so a random direction makes an angle near ninety degrees with any given one. The noise’s component along $\theta$ is a single normal variable of size about one, while its full length is about $\sqrt p$.

So the three points $0$, $\theta$ and $X$ form, approximately, a right-angled triangle with the right angle at $\theta$. The figure draws one such triangle in the plane the three points span, for fifty unknowns.

This is a picture from the dot product is a shadow, with the roles of the arrows changed. The multiple of $X$ nearest to $\theta$ is the shadow of $\theta$ on the line through $X$: the foot of the perpendicular dropped from $\theta$ onto that line. Its length along $X$ is $\langle X, \theta\rangle / |X|^2$ times $X$, and in the drawn example that factor is 0.379 — the best available correction shrinks the measurement to well under half its length. Nobody can compute that factor, because it needs $\theta$. But the right angle says what it approximately is. With the noise perpendicular to $\theta$, $\langle X, \theta \rangle \approx |\theta|^2$ and $|X|^2 \approx |\theta|^2 + p$, so

$$\frac{\langle X, \theta\rangle}{|X|^2} \approx \frac{|X|^2 - p}{|X|^2} = 1 - \frac{p}{|X|^2}.$$

The unknown has cancelled. The shadow of the truth on the measurement can be read off the measurement’s own length, and the James–Stein rule is that reading, with $p$ adjusted to $p - 2$. In the drawn example the rule shrinks by 0.391 against the shadow’s 0.379, and lands 3.70 from the truth where the measurement sat 6.65 away.

The window of shrinking that helps #

The replacement of $p$ by $p - 2$ comes from a calculation that is short enough to state. Try the general rule $(1 - a/|X|^2)X$ with any constant $a$. Expanding its squared error and using an identity of Stein’s for normal variables — integration by parts in each coordinate, which converts the cross term $\langle X - \theta, X/|X|^2 \rangle$ into $(p - 2)/|X|^2$ on average — gives

$$\mathbb{E}\,\Big|\Big(1 - \frac{a}{|X|^2}\Big)X - \theta\Big|^2 = p - \big(2a(p - 2) - a^2\big)\, \mathbb{E}\!\left[\frac{1}{|X|^2}\right].$$

That is a parabola in $a$, and the next figure draws it.

The bracket $2a(p - 2) - a^2$ is positive for every $a$ strictly between $0$ and $2(p - 2)$, so every one of those rules beats the measurement at every $\theta$, and the best constant in the family is the vertex of the parabola, $a = p - 2$. That is the whole derivation of the James–Stein rule. At $a = 2(p - 2)$ the bracket is nought again and the rule exactly ties the measurement — a curious fact that the figure checks — and beyond it shrinking does harm.

The geometric argument above said the right amount was $p$, and the exact calculation says $p - 2$. The difference is the price of using one noisy length to estimate how much of itself is noise: $1/|X|^2$ is not the reciprocal of its average, and the integration by parts accounts for that exactly. In large dimensions the two agree to within the noise, which is why the triangle is a fair picture of what the rule does.

Why it starts at three #

When $p = 2$, the window $(0, 2(p - 2))$ is empty, and the formula offers no rule that helps. When $p = 1$ or $p = 2$ there is a deeper failure: $\mathbb{E}[1/|X|^2]$ is infinite, because a normal point in one or two dimensions lands near the origin often enough for the reciprocal of its squared length to have no average. Any rule of this shape then has infinite error. The paradox does not weaken gradually as the dimension falls; it switches off between two and three, and for one and two unknowns the measurement really is unbeatable.

That threshold has appeared before, and the coincidence is not one. A walk that comes home found that a random walk on a lattice returns to its starting point with certainty in one and two dimensions and escapes with positive probability in three. Lawrence Brown proved in 1971 that the two facts are the same fact. Whether an estimate of the mean of a normal vector is admissible — whether nothing beats it everywhere — can be translated into whether a particular diffusion process, built from the estimate, returns to where it started or wanders off for ever. For the plain measurement $X$ the process is ordinary Brownian motion, which is recurrent in one and two dimensions and transient in three or more. Stein’s paradox is the statistical shadow of Pólya’s theorem about walks.

The connection also explains why the origin can be any point. Nothing in the argument used the origin except as the place to shrink towards; shrinking towards any fixed point $c$, by $(p - 2)/|X - c|^2$, has the same guarantee. A good choice of $c$ gives a large gain and a bad one a small gain, but no choice gives a loss.

Twenty thousand measurements at each distance #

A formula this surprising deserves a check that shares nothing with it, of the kind pi from dropped needles made the standard one: generate the experiment many times and average. The next figure generates, for ten unknowns, twenty thousand measurements at each of seven distances of the truth from the origin, and averages the squared errors of three rules: the measurement itself, the James–Stein estimate, and its positive part, which uses $\max\!\big(0, 1 - (p - 2)/|X|^2\big)$ so that a very short measurement is pulled to the origin and no further.

The averages land on the series, within their sampling error, at every distance: 2.01 against the exact 2 at the origin, 4.39 against 4.38 at distance two, 9.05 against 9.09 at distance eight. The measurement’s own average sits at ten throughout, as it must.

The positive part is the more instructive line. When $|X|^2 < p - 2$ the James–Stein factor is negative, and the rule reflects the measurement through the origin — it reports a point on the far side of nought from what was seen, which no sensible procedure would do. Clamping the factor at nought removes that, and the squares in the figure show it helps most where the truth is near the origin, bringing the error there from 2.01 down to 1.26. So the rule that first exposed the measurement as inadmissible is itself inadmissible, beaten by an obvious repair. That is not a defect in the theorem. The theorem’s content is that the measurement can be beaten, and once that is known the question becomes how well, which has a long tail of refinements.

The total improves while one quantity worsens #

The theorem is about the total squared error over all $p$ coordinates, and it is worth seeing what it does not say. The next figure puts one unknown far from nought and the other nine at nought, and measures the error coordinate by coordinate.

The large coordinate is dragged towards nought with the rest and is estimated worse, with an error of 2.39 against the measurement’s 1. The nine small ones are estimated better, at 0.67 each. The total, 8.45 against 10, is what the theorem is about. Nobody who cares about one particular quantity has been offered anything: for that quantity alone, the measurement is admissible, because it is a one-dimensional problem.

This is where the paradox earns its reputation. The ten unknowns need have nothing to do with each other — a price of tea, a rainfall, the mass of a beetle — and the rule still pools them, because its shrinking factor depends on the length of the whole vector. The joint estimate is better in total than ten separate ones, even though no physical connection links them. Bradley Efron and Carl Morris, who did most to explain the result to a wide audience, put the resolution plainly: the improvement is a statement about a sum of errors, and whether that sum is the right thing to minimise is a question about the purpose of the estimates, which the mathematics cannot settle.

Eighteen rates pulled towards their average #

The version used in practice shrinks not towards a fixed point but towards the average of the measurements themselves. With $p$ measurements and their average $\bar X$, the rule multiplies each measurement’s distance from $\bar X$ by $1 - (p - 3)\sigma^2/S$, where $S = \sum (X_i - \bar X)^2$ is their spread and $\sigma^2$ the noise variance. Subtracting the average is a projection: it removes each measurement’s component along the direction $(1, 1, \ldots, 1)$, exactly the single subtraction of one subtraction clears a direction, and leaves a vector in a space of one dimension fewer. The rule is James–Stein applied in that smaller space, so the guarantee now needs four or more measurements rather than three.

The figure is the shape of the famous example Efron and Morris used in 1975, built here from invented numbers rather than their data: eighteen success rates, each estimated from a small sample, so that the measured spread is largely noise. The shrinking factor comes out at 0.353 — the data say that roughly two-thirds of their own spread is noise — and the total squared error falls from 0.0165 to 0.0088, close to half. The rates that measured highest and lowest are pulled hardest, which is the formal version of regression towards the mean: an extreme measurement is extreme partly because its noise happened to be large, and regression is a square completed halfway drew the same pull for two correlated quantities.

Read this way, the rule has a second derivation. If the true rates were themselves drawn from a normal distribution around some centre, then Bayes’ theorem would give each one a posterior mean that is exactly a weighted average of its measurement and the centre, with weights set by the ratio of the noise variance to the spread of the rates. The James–Stein factor estimates that ratio from the data. Efron and Morris called this empirical Bayes: the prior is not assumed but read off the collection, and the paradox is the observation that this is possible only when there is a collection to read it off.

More coordinates make the shrinking exact #

The right-angle argument was approximate, and the approximation improves with dimension. The last figure fixes the ratio $|\theta|^2/p$ and lets the number of unknowns grow.

The dashed lines are what an oracle would achieve if it knew $|\theta|$ and used the best fixed multiple of $X$, which is $|\theta|^2/(|\theta|^2 + p)$ and gives an error of that same share of $p$. As $p$ grows the James–Stein curves fall onto those lines. The reason is the average settling while the wobble does not: $|X|^2/p$ is an average of $p$ squared coordinates, and its fluctuation shrinks like $1/\sqrt p$ — the same bound on how far an average strays from its mean that every concentration argument starts from. In two hundred dimensions the measurement’s length reveals, almost exactly, how much of it is signal, and estimating the oracle’s factor costs almost nothing.

The size of the gain is set by the ratio. With a weak signal, $|\theta|^2 = p/4$, the shrunk estimate has a fifth of the measurement’s error; with a strong one, $|\theta|^2 = 4p$, it has four-fifths. Shrinking matters most exactly when the noise dominates, which is when a single measurement says least.

What a total of squared errors does not promise #

The figures measure one thing, the expected total squared error, and the guarantee is about that quantity and no other. The coordinate-by-coordinate figure already showed one consequence: no individual coordinate is promised anything, and a coordinate far from where the rule shrinks is estimated worse. Two more limits are worth stating.

The first is the noise model. Everything above assumes the errors are normal, independent and of equal, known variance. With unequal variances, rules that beat the measurement everywhere still exist in three or more dimensions, but no single one is best and the formulas change; with heavy-tailed errors the length $|X|$ no longer measures the noise reliably, because one wild coordinate can dominate it. The picture of a right-angled triangle is a picture of normal noise, whose coordinates are independent and individually small.

The second is the meaning of better. A rule that beats the measurement in expected squared error does not produce narrower honest error bars around each coordinate, and confidence statements built naively around the shrunk values are wrong in the coordinates that were pulled furthest. The figures show averages over repeated measurement, and an experimenter who measures once holds one draw, in which — as the shadow figure illustrates — the shrunk point is usually better and occasionally worse.

Still open: the best rule of this kind #

The measurement is beaten by James–Stein, James–Stein is beaten by its positive part, and the positive part is itself inadmissible: Qiang Shao and William Strawderman exhibited in 1994 a rule that improves on it at every $\theta$, by a very small margin. The chain of improvements does not obviously end in a rule that is both admissible and simple. Admissible rules that beat the measurement do exist — Brown’s correspondence characterises them through the recurrence of their associated diffusions, and rules derived from suitable priors, such as Strawderman’s of 1971, are among them — but they are not of the elementary shrink-by-a-formula kind, and whether an admissible rule as simple as the positive part exists is a question that, as far as the literature records, has no settled answer.

The practical question behind it is where to shrink to. The guarantee holds towards any point, the gain depends heavily on the choice, and the empirical-Bayes version chooses the average because that is what the data suggest. In problems with structure — rates that depend on a known covariate, measurements on a grid where neighbours resemble one another — the target can be a fitted model rather than a single number, and that is the nearest point of a flat thing returning one level up: shrink towards the projection of the measurements onto a model, by an amount the residuals choose. How to pick the model is not a theorem. What the right angle guarantees is only that, once there are three or more unknowns, refusing to shrink at all is never the best choice.