The best middle when the shape is unknown

The middle value has a bell of its own compared two ways of estimating the centre of a symmetric distribution from a sample: the mean and the median. Each has a bell-shaped sampling distribution in large samples, with widths that depend on the shape of the data. For normal data the mean is narrower, by a factor whose square is $\pi/2$; for Laplace data the median is narrower, by a factor of two in variance; for Cauchy data the mean has no width at all to speak of, since it is as spread as a single observation. The essay ended on the question practical statistics actually faces: the shape is not known. Which single estimate of the centre does best across a whole family of shapes?

The trimmed means interpolate between the two. Sort the sample, discard a share $\alpha$ of the smallest observations and the same share of the largest, and average what is left. With $\alpha = 0$ that is the mean; as $\alpha$ approaches one half it becomes the median. Each trimmed mean has its own large-sample variance, computable exactly for any symmetric shape, and comparing them across shapes answers the question as well as it can be answered.

The variance of a trimmed mean #

For a symmetric distribution with density $f$, let $q$ be the point above which a share $\alpha$ of the distribution lies. In large samples the $\alpha$-trimmed mean of $n$ observations is approximately normal about the centre, with variance $\sigma^2_\alpha / n$, where

$$\sigma^2_\alpha = \frac{1}{(1 - 2\alpha)^2}\left(\int_{-q}^{q} x^2 f(x)\,dx + 2\alpha q^2\right).$$

The formula has a plain reading. The trimmed mean behaves like the mean of a winsorised distribution, in which every observation beyond $q$ is pulled in to $q$ and every one below $-q$ to $-q$: the integral is the variance of the middle part and $2\alpha q^2$ the contribution of the pulled-in tails. The factor $(1 - 2\alpha)^{-2}$ accounts for averaging only the fraction $1 - 2\alpha$ of the sample that is kept. The figures evaluate it by numerical integration for each shape, finding $q$ by bisection on the tail area.

To turn a variance into a verdict it needs a yardstick, and the yardstick is the Fisher information $I = \int f'(x)^2 / f(x)\,dx$. No estimator of the centre that is unbiased in large samples can have variance below $1/(nI)$, by the Cramér–Rao inequality, and for each of these shapes some estimator reaches it. The efficiency of the trimmed mean is $1/(I\sigma^2_\alpha)$, a number between 0 and 1: the share of the information in the sample that the estimator actually uses.

A bell even for the Cauchy #

The trimmed mean has a bell-shaped sampling distribution for every one of the seven shapes, the Cauchy included, and that is not a small point. The plain mean’s bell comes from the central limit theorem, which needs a finite variance; for Cauchy data there is none, and the mean of any number of observations is as spread as one of them, the shape that averaging leaves alone. The trimmed mean escapes that because it never sees the tails. Its value depends on the sample only through the observations between the $\alpha$ and $1 - \alpha$ sample quantiles, and the sample quantiles of any distribution with a positive density there are themselves approximately normal, with variance of order $1/n$. A smooth function of approximately normal quantities is approximately normal, so the trimmed mean obeys a central limit theorem whenever the density is positive at its trimming points — a far weaker condition than a finite variance.

That is also why the variance formula has the shape it has. The tails beyond $q$ enter only through the single number $q$, the position of the cut, and the distribution’s behaviour beyond the cut is irrelevant: a trimmed mean cannot tell a Cauchy tail from a normal one, provided the middle is the same. The price is the information the tails carry when they are well behaved, which is the normal shape’s loss of 6 per cent at a 10 per cent trim.

Seven shapes, and what each prefers #

The hero runs the trim from nought to one half for seven shapes chosen to span light and heavy tails. For normal data the mean is perfect, efficiency 1, and every trim costs a little: 0.943 at 10 per cent from each end, 0.837 at 25, and $2/\pi = 0.637$ for the median. For the logistic, slightly heavier-tailed, the mean is already only 0.912 efficient and a light trim does better, 0.994 at 10 per cent. For the normal with 10 per cent contamination — nine observations in ten from the standard normal, one in ten from a normal three times as wide, John Tukey’s model of a well-behaved process with occasional gross errors — the mean falls to 0.698, and trimming 10 per cent restores 0.969.

The heavier shapes push the best trim further in. For the $t$ distribution with five degrees of freedom the best is about 14 per cent, for three degrees about 22 per cent, and for the Cauchy — the $t$ with one degree — the best trimmed mean takes 38 per cent from each end and reaches 0.878, while the plain mean’s efficiency is nought. For the Laplace, whose sharp peak makes the median exactly optimal, the efficiency climbs all the way to 1 at the median. Every shape has its own best trim, and none of them is at either end except the two extremes that define the ends: the normal at the mean and the Laplace at the median.

Why peaks favour the median #

The median’s variance has an even simpler formula than the trimmed mean’s: $1/(4f(0)^2)$, which depends only on the height of the density at the centre. A tall, sharp peak means that many observations fall close to the centre, and the middle observation of the sample is pinned there tightly; the tails do not enter at all. The Laplace density, $\frac12 e^{-|x|}$, comes to a sharp point at the centre, and for it the median is exactly optimal. The normal density is rounder at the top, its peak lower for the same variance, and there the median loses the factor $2/\pi$.

The mean’s variance is the opposite: it depends only on the spread, $\int x^2 f$, and not on the peak at all. So the two estimators are sensitive to different features of the shape, and a trimmed mean, which averages the middle part, is sensitive to both in proportion to how much it trims. That is why its efficiency curves in the hero cross so freely: a shape with a sharp peak and heavy tails, like the Laplace or the $t_3$, wants a heavy trim, and a shape with a round peak and light tails, like the normal, wants none.

The best guarantee #

If nothing is known about which of the seven shapes produced the data, the relevant measure for each trim is its worst efficiency over all of them.

The plain mean guarantees nothing, since the Cauchy makes its efficiency nought. As the trim grows, the worst case is the Cauchy until about 28 per cent, the guarantee rising steeply; beyond that the normal becomes the worst case, the guarantee falling slowly to the median’s 0.637. The peak is a minimax compromise: trimming 28 per cent from each end keeps every one of the seven shapes above 81 per cent efficiency. Over the plausible middle range of trims, 20 to 35 per cent, the guarantee stays above 0.69.

The shape of the curve is worth reading as well as its peak. The rise on the left is steep because the Cauchy’s efficiency climbs quickly with the first few per cent of trimming — the most extreme observations are by far the most damaging, and removing them buys the most. The fall on the right is gentle because the normal loses information slowly as more of its well-behaved middle is discarded. A guarantee curve that rises steeply and falls gently means that erring towards a heavier trim is cheaper than erring towards a lighter one, which is the practical advice the figure gives. The answer depends on the family. Leave out the Cauchy and the worst case for small trims is the $t_3$ or the Laplace, and the best compromise moves to a lighter trim; add shapes with even heavier tails and it moves heavier. This is the honest form of the conclusion: there is no best estimator of the centre without some statement of which shapes are possible, and for any reasonable family the best compromise is a trimmed mean somewhere in the middle, not either end.

Bickel’s floor #

There is also a guarantee in the other direction, against the plain mean, that holds for every symmetric shape at once.

Peter Bickel proved in 1965 that the ratio of the mean’s variance to the $\alpha$-trimmed mean’s is at least $(1 - 2\alpha)^2$ for every symmetric distribution with a variance. So trimming 10 per cent from each end can cost at most a factor of 0.64 in variance against the mean, and for any shape where the mean is not already best it costs less or gains. The figure confirms the bound for the six shapes with a finite variance: the normal stays closest to it, since for normal data the mean is optimal and every trim is a loss, and every heavier-tailed shape sits well above, the ratio rising above 1 wherever trimming helps. The Cauchy is off the top of the figure: its mean has infinite variance and every trim is an infinite improvement.

The bound is what makes trimming a safe default. A statistician who trims 10 per cent and turns out to have had normal data has lost at most 36 per cent of the variance’s worth — in fact only 6 per cent for the normal itself, since the bound is far from tight there — while one who uses the mean and turns out to have had contaminated data can lose everything.

The formula and the samples #

The variance formula is a large-sample statement, and samples are finite.

Simulating 3,000 samples of 100 observations each, for normal, Laplace and $t_3$ data and trims from 5 to 40 per cent, the measured variances agree with the formula to within a few per cent: the largest relative gap is 4.4 per cent, for the $t_3$ with the lightest trim, where the occasional very large observation that survives a 5 per cent trim makes the sample variance itself noisy. How quickly a large-sample formula becomes accurate is itself a measurable thing — how fast the bell arrives for the plain mean is governed by the third moment of the data, through the Berry–Esseen theorem, and for heavy-tailed data that moment can be large or infinite. The trimmed mean sidesteps the problem in the same way it sidesteps the variance: the observations that make the third moment large are the ones it discards, so its approach to the bell is governed by the middle of the distribution, where every shape here is well behaved. The formula is already a reliable guide at a hundred observations, which is what makes the efficiency comparisons of the earlier figures useful in practice rather than only in the limit.

On heavy-tailed data #

What the efficiencies mean for a single sample is clearest in the sampling distributions themselves.

For $t_3$ data, samples of 50, the middle half of the mean’s sampling distribution is 0.319 wide, the median’s 0.252 and the 20 per cent trimmed mean’s 0.240. The mean’s distribution is not only wider but has heavier tails, because a single extreme observation moves it a long way. The trimmed mean beats the median here because it still averages the middle sixty per cent of the sample, using more information than the median while discarding the extremes that harm the mean.

For Cauchy data the contrast is extreme. The mean of fifty Cauchy observations has exactly the distribution of one observation, an average that never settles, and its middle half is about 2 wide — the figure’s histogram is almost flat across the drawn range. The median’s middle half is 0.299 and the 20 per cent trimmed mean’s 0.324; for this shape a heavier trim does better still, as the hero showed, with the best near 38 per cent.

Trimming in practice #

Trimmed means are older than the theory that justifies them. Judges’ scores in diving and gymnastics have long been combined by discarding the highest and the lowest and averaging the rest, a trimmed mean chosen to blunt the effect of a single biased or careless judge, and the reasoning is exactly the reasoning above: a single wild value should not be able to move the result far. In the analysis of measurements the same practice appears as rejecting outliers, usually with a rule decided after looking at the data, which makes its properties hard to state. A trimmed mean is the version of that practice whose properties can be stated in advance, which is why it is the version a statistician can recommend.

The law of large numbers guarantees that every one of these estimators settles at the centre of a symmetric distribution as the sample grows, Cauchy excepted for the plain mean. The question this essay answers is how fast, and the answer is that the speed depends on the shape in ways no single estimator can match everywhere — which is why the choice of how much to trim is a choice about which shapes are plausible, made before the data are seen.

Seven shapes chosen, not all shapes #

Seven shapes are a choice. The minimax trim of 28 per cent and its guarantee of 0.814 are properties of this family. A different family gives a different compromise, and no family covers everything real data can do — skewed data, in particular, are outside the whole framework, since the centre of a skewed distribution is not well defined and the trimmed mean estimates a different quantity from the mean.

Large-sample efficiencies. The efficiencies come from asymptotic variances. The simulation checks them at a hundred observations; for very small samples the comparisons can change, and the median’s advantage on heavy tails is larger in small samples than the asymptotics suggest.

Trimmed means are not the only compromise. Peter Huber’s M-estimators of 1964, which down-weight large deviations smoothly instead of discarding them, have a cleaner minimax theory against contamination, and the large robustness study of 1972 at Princeton compared dozens of estimators across many shapes. Trimmed means are the simplest member of the family and close to the best in most comparisons, which is why they are drawn here.

Still open: adapting to the sample #

The sharpest version of the question is adaptive: can an estimator choose its own trim from the data and be nearly as efficient, for every shape, as the best trimmed mean for that shape? In the limit of large samples the answer is yes — the shape can be estimated, and an adaptive estimator can match the best one for every symmetric shape, a result associated with Charles Stein’s work on adaptive estimation in the 1950s and made precise by later authors. For samples of practical size it is not settled. Choosing the trim from the data costs some of the information the choice is meant to save, and how much — and whether a simple rule, such as choosing the trim from how heavy the sample’s own tails look, comes close to the best possible adaptive rule at sizes of fifty or a hundred — is a question with useful partial answers and no complete one. The tail is not a bell, and the estimators that deal with tails well are still being compared one sample size at a time.