Holding one still turns the other over

Regression is a square completed halfway completed the square in one variable of a two-variable bell and found the centre of each slice on a line flatter than the bell’s own axis. Its last section noted that the same completion works in any number of variables, and that in more than two it produces something that has no counterpart with two: a quantity positively correlated with an outcome can predict it negatively once another quantity is held fixed.

That is not a curiosity of badly behaved data. It happens with perfectly bell-shaped variables, it follows from the same few lines of algebra, and it is common. This essay says exactly when it happens, how often it happens among all the correlations three quantities can have, how its frequency grows with the number of quantities, and why a sample is least able to see the reversal precisely where the reversal is most likely to be real.

The completion with two predictors #

Take three quantities, standardised so that each has mean zero and spread one: two predictors $x_1$ and $x_2$, correlated with each other at $\rho$, and an outcome $y$, correlated with them at $r_1$ and $r_2$. If the three are jointly bell-shaped, the best prediction of $y$ from both predictors is a straight line in them, $\beta_1 x_1 + \beta_2 x_2$, and the coefficients come from completing the square in $y$ given the other two, exactly as the single-predictor case did — the same move completing the square, by completing a square drew as tiles, now made in an exponent, where one number under every bell showed it costs nothing but a shift and a squeeze. Written out,

$$\beta_1 = \frac{r_1 - \rho\, r_2}{1 - \rho^2}, \qquad \beta_2 = \frac{r_2 - \rho\, r_1}{1 - \rho^2}.$$

The same formulas are the least-squares coefficients for any data with those correlations, bell-shaped or not; the bell is only what makes the line the best possible prediction rather than the best straight one.

The numerator of $\beta_1$ is the whole story. The coefficient starts from $x_1$’s own correlation with $y$ and subtracts $\rho\, r_2$, the part of that correlation that $x_1$ owes to travelling with $x_2$. If that borrowed part is larger than the whole — if $\rho\, r_2$ exceeds $r_1$ while $r_1$ is positive — then $\beta_1$ is negative. With $\rho = 0.8$, $r_1 = 0.3$ and $r_2 = 0.6$, the borrowed part is $0.48$, larger than $0.3$, and

$$\beta_1 = \frac{0.3 - 0.48}{1 - 0.64} = -0.5.$$

A quantity that rises with $y$ predicts it falling, by half a standard deviation per standard deviation, once $x_2$ is held still.

The reversal in data #

The numbers above are not a contrivance. They describe a perfectly ordinary population, and a sample from it shows the reversal plainly.

Across all six hundred points $y$ rises with $x_1$, at a slope of $0.31$, just as the correlation of $0.3$ says it should. Now split the points into three bands by their value of $x_2$ — below $-0.6$, between, and above — and fit a line inside each. All three slopes are negative. Points with high $x_1$ tend to have high $x_2$, which brings high $y$, and that is the entire source of the overall rise; among points with similar $x_2$, more $x_1$ goes with less $y$.

The band slopes are shallower than the population’s $-0.5$, because a band only roughly holds $x_2$ fixed and the variation left inside it leaks some of the borrowed correlation back in. Narrower bands would come closer, and the exact way to hold $x_2$ fixed is not a band at all but a subtraction.

The shape of the picture is the one usually told as Simpson’s paradox, where a comparison that holds inside every group reverses when the groups are pooled. Every fraction exactly once found the discrete version of it in the mediants of the Stern–Brocot tree, where a pooled ratio can sit on the wrong side of both of the ratios it pools. The continuous version needs no groups: the grouping variable is $x_2$, and it is measured rather than counted.

Holding still is a subtraction #

Predict $x_1$ from $x_2$ by a straight line and keep only what that line misses; do the same for $y$. The leftovers of $x_1$ are the part of $x_1$ that has nothing to do with $x_2$, and the leftovers of $y$ are the part of $y$ that $x_2$ does not explain. Plotted against each other they fall along a line of slope $-0.502$, and that number is not merely close to the coefficient of $x_1$ in the full fit — it is equal to it, to every digit. This is the theorem of Ragnar Frisch and Frederick Waugh from 1933, rediscovered by Michael Lovell thirty years later: the coefficient of one predictor in a multiple regression is the slope between the two sets of leftovers.

In the language of this subject it is a statement about the order of completion. Completing the square first in $x_2$ and then in $x_1$ gives the same answer as completing in both at once, for the same reason the law of inertia found every order of completion leaving the same signs behind: each step is a change of variables that leaves the quadratic form unchanged. Geometrically it is a projection. The nearest point of a flat thing described least squares as dropping a perpendicular onto the space the predictors span; the leftovers are perpendiculars dropped onto the space $x_2$ alone spans, and the coefficient of $x_1$ is the slope of one perpendicular against the other.

The subtraction shows what “holding $x_2$ fixed” means and what it does not. It does not mean that $x_2$ could actually be held fixed while $x_1$ changes; in many applications it cannot. It means comparing values of $x_1$ that differ in the part of $x_1$ that $x_2$ does not predict, and the answer is a property of the correlations alone.

The wedge where the sign turns over #

For a fixed correlation $\rho$ between the predictors, the pairs $(r_1, r_2)$ that the outcome can have with them are limited: the matrix of all three correlations has to be a possible one, which means its determinant cannot be negative. That condition,

$$r_1^2 + r_2^2 - 2\rho\, r_1 r_2 \le 1 - \rho^2,$$

fills a tilted ellipse, the one in the hero figure. The coefficient $\beta_1$ has the opposite sign to $r_1$ exactly when $r_1$ lies between $0$ and $\rho\, r_2$ — in the hero figure, between the vertical axis and the line $r_1 = \rho\, r_2$. That is a double wedge through the centre of the ellipse, and $\beta_2$ has the mirror-image wedge between the horizontal axis and the line $r_2 = \rho\, r_1$.

How much of the ellipse does a wedge take? The ellipse is a circle drawn in a slanted coordinate system, the one in which its own quadratic form measures lengths. In that system the two lines bounding the wedge meet at an angle whose cosine works out to $\sqrt{1 - \rho^2}$, so the angle is $\arcsin |\rho|$, and a double wedge of angle $\theta$ takes the fraction $\theta/\pi$ of a circle. So the share of possible correlations that reverses $x_1$’s sign is exactly

$$\frac{\arcsin|\rho|}{\pi},$$

which at $\rho = 0.6$ is $20.5\%$ — the share the hero figure measures cell by cell.

The two wedges never overlap. Inside $\beta_1$’s wedge, with $r_1$ positive, $r_2$ exceeds $r_1/\rho$, which is larger than $\rho\, r_1$, so $\beta_2$ keeps its sign. With two predictors, at most one coefficient can turn over, never both. Geometrically the two wedges sit on opposite sides of the diagonal $r_1 = r_2$, each hugging its own axis, and the predictors’ correlation $\rho$ opens both by the same amount without ever letting them meet. Only with a third predictor can a single correlation be outweighed by borrowed parts from two directions at once.

The fence round the possibilities #

The ellipse in the hero figure is one slice of a solid. Three quantities have three correlations, and a triple $(\rho, r_1, r_2)$ is possible only if the matrix

$$\begin{pmatrix} 1 & \rho & r_1 \\ \rho & 1 & r_2 \\ r_1 & r_2 & 1 \end{pmatrix}$$

can be the correlation matrix of real quantities, which means its quadratic form can never be negative. The signs no completion can change gave the test: complete the square in the form, variable by variable, and every coefficient that comes out must be positive or zero. Completing in the first variable leaves a $1$; in the second, $1 - \rho^2$; in the third, the determinant divided by $1 - \rho^2$. So the condition is that the determinant, $1 + 2\rho r_1 r_2 - \rho^2 - r_1^2 - r_2^2$, is not negative, and each slice at fixed $\rho$ is the ellipse already drawn.

Stacked over every $\rho$ from $-1$ to $1$ the slices make a solid that statisticians call the elliptope: a puffed-up tetrahedron with four corners at the matrices whose correlations are all $\pm 1$, straight edges between the corners and no flat faces anywhere. Its volume is $\pi^2/2$, a fraction $\pi^2/16 \approx 0.617$ of the cube of side two in which all three correlations live, and that fraction is what the sampling above recovered: of 200,000 points drawn in the cube, 61.7% landed inside. The slices are widest at $\rho = 0$, where the ellipse is a circle of radius one, and shrink to nothing at $\rho = \pm 1$, where two predictors that are the same quantity must have the same correlation with everything.

That shrinking is why the average reversal rate is only one in seven although a wedge can approach half its slice. The slices with wide wedges are the narrow ones near $\rho = \pm 1$, and they hold little of the solid’s volume; most possible matrices have moderately correlated predictors, where the wedges are modest.

About one coefficient in seven #

The wedge share depends on $\rho$, so the natural next question is how often a reversal happens among all the correlation matrices three quantities can have.

Uncorrelated predictors never reverse: at $\rho = 0$ the wedge has no width, and each coefficient is just its own correlation. As $\rho$ grows the wedge opens, to a sixth of the ellipse at $\rho = 0.5$ and a third at $\rho = 0.87$, and as $\rho$ approaches one it approaches half — nearly highly correlated predictors reverse one coefficient or the other almost every time.

To average over all possible matrices, weight each slice by its size. The ellipse at correlation $\rho$ has area $\pi\sqrt{1 - \rho^2}$, so the average share is

$$\frac{\int_{-1}^{1} \frac{\arcsin|\rho|}{\pi}\,\pi\sqrt{1-\rho^2}\,d\rho}{\int_{-1}^{1} \pi\sqrt{1-\rho^2}\,d\rho} = \frac{1}{4} - \frac{1}{\pi^2} \approx 0.1487.$$

The integral in the numerator becomes $\int_0^{\pi/2} t\cos^2 t\,dt = \pi^2/16 - 1/4$ after the substitution $\rho = \sin t$. So a correlation matrix drawn uniformly from all valid ones reverses a given coefficient with probability exactly $1/4 - 1/\pi^2$, one time in about seven, and reverses one of the two with probability $1/2 - 2/\pi^2$, about three times in ten. Drawing 200,000 points in the cube of possible correlations and keeping the 123,332 that form valid matrices — the expected share is $\pi^2/16$, the volume of the set of valid matrices — gives a measured share of $14.69\%$ per coefficient and $29.6\%$ for either, against the exact $14.87\%$ and $29.7\%$, and not one matrix that reverses both.

Uniform is a choice, and real data are not drawn uniformly from all correlation matrices. It is the natural reference point, the one with no preference for any pattern, and it makes the point that reversals are not rare corners of the space of possibilities but a seventh of it.

Many predictors, many reversals #

With more predictors the completion is the same in matrix form — the coefficients are the inverse of the predictors’ correlation matrix applied to their correlations with $y$ — and the restriction that only one sign can turn over disappears.

Drawing correlation matrices uniformly in higher dimensions takes care, since most symmetric matrices with ones on the diagonal are not valid correlation matrices at all. The onion method of Lewandowski, Kurowicka and Joe builds one row at a time from a beta-distributed length and a uniform direction, and produces exactly the uniform distribution. With two predictors it reproduces $1/4 - 1/\pi^2$ to the third decimal. With three, each coefficient reverses in $22\%$ of matrices; with six, $30\%$; with twenty-four, $39\%$. The share of matrices in which some coefficient reverses climbs faster still, past $90\%$ at six predictors and to within a fraction of a percent of certainty by twelve.

The trend points towards one half, a coin toss, and its meaning is that with many predictors a quantity’s correlation with the outcome says less and less about the sign it will carry in the fit. Each coefficient is the correlation minus a sum of borrowed parts, one for every other predictor, and when there are many of them the borrowed parts are as large as the original. A table of correlations, read one column at a time, is a poor guide to a multiple regression, and the poorer the more columns there are.

Where the sign is least certain #

A population’s coefficient is one thing; a sample’s estimate of it is another, and the reversal sits exactly where the estimate is weakest.

A fitted coefficient is an average of noisy terms, and like any such average it scatters round its target with a spread that shrinks like one over the square root of the sample size. That spread also grows with the predictors’ correlation, by the factor $1/\sqrt{1 - \rho^2}$ in its spread — the variance inflation of the regression textbooks. When $x_1$ and $x_2$ are nearly the same quantity the data can say a great deal about their combined effect and very little about how to divide it between them. In the simulation the population coefficient of $x_1$ is $-0.2$ and that of $x_2$ is $+0.5$; with independent predictors a sample of fifty gets $x_1$’s sign right 95% of the time, and at $\rho = 0.95$ only two times in three.

Meanwhile, once $\rho$ passes $0.4$, $x_1$’s correlation with $y$ in this population is positive — $-0.2 + 0.5\rho$ — and the negative coefficient is a genuine reversal. So the reversals are concentrated at high $\rho$, where the wedge is wide, and high $\rho$ is where samples are worst at telling a real reversal from noise. A reported reversal at high correlation is exactly as likely to be the population’s as an accident of the sample, until the sample is large enough to say which.

What the completed square does not decide #

Everything above is a statement about correlations, and correlations are symmetric: they do not say which quantity influences which. A negative coefficient for $x_1$ with $x_2$ held fixed is a fact about the prediction of $y$ from the two, and whether it describes what would happen if someone changed $x_1$ depends on how the three are actually connected. If $x_2$ is caused by $x_1$ — a medicine’s dose $x_1$ affecting a blood level $x_2$ that affects recovery $y$ — then holding $x_2$ fixed removes part of $x_1$’s real effect, and the reversed coefficient answers a question nobody asked. If $x_2$ is a common cause of both, holding it fixed is exactly right. The algebra is identical in both cases, and no amount of it distinguishes them.

The uniform average is also only a reference. In a field where predictors are chosen because each is believed to matter, or where they are measurements of overlapping things, the correlations are not drawn from all possibilities, and the reversal rate can be much higher or much lower than one in seven.

Still open: telling a real reversal from an artefact #

The arithmetic of reversal is settled: the wedge, its angle, its average, and its growth with the number of predictors all follow from completing a square. What is not settled is the inference.

Given a sample showing a reversal, how should it be decided whether the population has one? A confidence interval for the coefficient that excludes zero settles the sign in the narrow statistical sense, but the intervals widen exactly where reversals concentrate, and at the sample sizes common in practice they often include zero there. Methods that pool information across many related regressions, or that bring in prior knowledge about which signs are plausible, change the answer, and how much they should be trusted depends on the field. For many predictors there is also a question with no clean answer here: the share of reversals in uniformly drawn matrices is measured up to twenty-four predictors and is still climbing, at a slowing rate, towards something no larger than one half. Whether the limit is one half, and how fast it is approached, is not something this computation can settle, and no formula for the share at three predictors — the first case where two coefficients can reverse at once — has been found here, though the three-predictor share, about 22%, is measured to two decimals.

The borrowed part can exceed the whole #

A predictor’s coefficient, with another held fixed, is its correlation with the outcome minus the part it borrows from the other — and the borrowed part can be larger than the whole. With two predictors correlated at $\rho$, the possible correlations that make it so fill a double wedge of angle $\arcsin|\rho|$ in an ellipse, a share $\arcsin|\rho|/\pi$ of all possibilities; averaged over every correlation matrix that is $1/4 - 1/\pi^2$, about one coefficient in seven, and the two wedges never overlap, so both coefficients never reverse together. With more predictors reversals multiply, to 39% of coefficients with twenty-four. Holding a variable fixed is a subtraction, the coefficient is the slope of what the subtraction leaves, and the reversals live at high correlation, where a sample is least able to tell them from noise.