Evidence that can be multiplied

Peeking until the answer is yes watched one coin, one toss at a time, and found that a p-value checked after every toss will eventually call any fair coin biased, while a Bayes factor checked the same way passes twenty to one at most one time in twenty. The guarantee came from Jean Ville’s inequality for a quantity whose average under the null stays at one however long it is watched.

That essay was about a single study. Science is made of several. A coin is tossed forty times in one laboratory and forty in another; a drug is tested in one trial, and a second trial is run because the first looked promising; a meta-analysis gathers what was published. Each step raises the same question at a larger scale: how should evidence from separate studies be put together, and what happens to the guarantee when the decision to run the next study depends on how the last one came out?

The three curves in the figure are three answers. One of them is classical and works only if the number of studies is fixed before the first begins. One is valid under any plan whatever and wastes evidence. The third is valid under any plan and wastes nothing, and it is the subject of this essay.

A number whose average is one #

The guarantee of the earlier essay rested on one property of the Bayes factor, and it is worth separating the property from the Bayes factor. Call a non-negative statistic $E$ an e-value for a null hypothesis if its expected value is at most one whenever the null is true:

$$\mathbb{E}_{\text{null}}[E] \le 1.$$

Markov’s inequality, the bound on how far from its average a non-negative quantity can be, then says that $E \ge 20$ happens with probability at most $1/20$. So an e-value of twenty is evidence against the null at the 5% level, and in general $1/E$ is a valid p-value. Nothing about this needs the e-value to be a Bayes factor, but every Bayes factor of a fixed alternative against the null is one: its expectation under the null is a sum, over every possible outcome, of the alternative’s probability of that outcome, which adds to one.

For the coin, the Bayes factor of a prior on the bias against a fair coin is a ratio of probabilities of the tosses observed. With every bias equally likely beforehand it is $2^n H!\,T!/(n + 1)!$ after $H$ heads and $T$ tails — the same number that a bettor who learns the coin toss by toss ends up holding as wealth. That is not a coincidence, and it is the best way to think about an e-value: it is the wealth of a gambler who started with one unit, bet against the null at fair odds, and was not allowed to borrow. Under the null the bets are fair, so the expected wealth stays at one; under an alternative the gambler wins.

Products of e-values are e-values #

Suppose a first study yields an e-value $E_1$, and a second study, run afterwards, yields $E_2$ — where the second study’s e-value may be computed in a way that depends on what the first one found, so long as it is an e-value given the first study’s data. Then

$$\mathbb{E}[E_1 E_2] = \mathbb{E}\big[E_1\, \mathbb{E}[E_2 \mid \text{study 1}]\big] \le \mathbb{E}[E_1] \le 1.$$

The product is an e-value. More than that: the running product after each study is a non-negative quantity whose expected next value never exceeds its current value — a supermartingale — and Ville’s inequality applies to it exactly as it applied to the Bayes factor watched toss by toss. However the decision to continue is made, the product passes twenty with chance at most one in twenty. In the gambling picture, each study is a new round of fair bets made with whatever wealth the previous rounds left; a gambler cannot turn fair bets into a profit by choosing when to stop.

The figure’s two e-value curves differ only in how the second study’s e-value is chosen. The green curve uses a fresh uniform prior for every study, as if each study were the first. The orange curve uses, as each study’s prior, the posterior from all the tosses before it. The second product telescopes: the probability of the twelve studies’ tosses under the learned alternatives, multiplied together, is the probability of all 480 tosses under the original uniform prior, so the product of learned e-values is exactly the Bayes factor of the pooled data, which the figure confirms to eight decimal places. It passes twenty at the seventh study and ends near $10^{4.3}$.

The fresh-prior product is valid too, and it ends at 2.4. Every study had to learn the coin’s bias from scratch, and learning costs about $\tfrac12 \ln n$ per study, the price measured for the bettor who did not know the coin. Twelve studies pay it twelve times. Pooling pays it once. Multiplying e-values is legitimate in either form, and it wastes evidence unless each study is allowed to know what the others found.

Fisher’s combination needs a plan #

The blue curve combines the twelve p-values the classical way. Ronald Fisher observed in 1932 that if $k$ independent p-values are each uniform under the null, then $-2\sum \ln p_i$ has a chi-squared distribution with $2k$ degrees of freedom, and a combined p-value can be read from it. On this run it passes 0.05 at the tenth study.

Fisher’s method makes a promise only about a number of studies decided beforehand. The figure below asks what happens in the more realistic situation where another study is run whenever the evidence so far falls short.

With the number of studies fixed in advance, Fisher’s method holds its level; on these discrete tosses it is conservative, wrongly rejecting a fair coin 1.4% of the time with ten studies. With another study run whenever the combined result falls short, it rejects 5.3% of fair coins by the second study and 8.7% by the tenth, and the share keeps climbing as more studies are allowed. That is the mechanism of peeking, working at the scale of studies instead of tosses: each look is a fresh chance for a combined p-value that is uniform under the null to dip below the line.

The product of learned e-values, watched under exactly the same rule, wrongly claims bias on 0.7% of programmes, and Ville’s inequality guarantees it would stay below 5% under any rule at all — a rule that stops when the evidence looks bad, one that continues only after promising results, one that adds studies until a grant runs out. Nothing about the rule has to be stated in advance or even known afterwards.

That last property is what makes e-values attractive for science as it is actually conducted. Very few research programmes fix the number of studies before the first result is in, and very few meta-analyses can know why a study was run or not run. A p-value-based meta-analysis is formally valid only under assumptions about that history which nobody can check. A product of e-values is valid without them.

The rate at which evidence grows #

If e-values cost nothing in validity, they must cost something somewhere, and the natural place to look is in how fast they grow when the null is false.

The logarithm of an e-value grows, under an alternative, at an expected rate per toss, and no e-value can grow faster than the likelihood ratio at the true chance, whose rate is $\ln 2 - H(\theta)$, the Kullback–Leibler divergence of the true coin from a fair one. That is the same number as Kelly’s growth rate for a bettor who knows the coin — again because an e-value is a gambler’s wealth — and the same number that decides which of two wrong hypotheses a posterior settles on.

The likelihood ratio staked on a single alternative, 0.6, achieves the best rate exactly at 0.6 and falls away on either side. At a true chance of 0.8 it grows at 0.101 nats a toss where 0.193 was possible, because it keeps betting as though heads came six times in ten. The uniform-prior Bayes factor never commits to an alternative, and its rate rises towards the best rate at every chance at once as the tosses accumulate: after a hundred tosses of a 0.6 coin it has grown at 0.004 a toss on average, after a thousand at 0.017, against the best 0.020. The shortfall is the $\tfrac12 \ln n$ of learning, spread over $n$ tosses.

What a guarantee under any plan costs in tosses #

Growth rates are a long-run measure. The practical question is how many tosses a researcher needs to detect a biased coin with reasonable reliability, and how that compares with a test whose sample size is fixed in advance.

The fixed test needs 210 tosses to detect a 0.6 coin with 80% probability; the watched Bayes factor needs 376. For a 0.55 coin the numbers are 807 and 1,684; for a 0.8 coin, 23 and 34. The guarantee under any plan costs between 1.4 and 2.1 times as many tosses, more for the subtler biases. That is the honest exchange rate. It is paid in data rather than in false alarms, and it buys two freedoms the fixed test does not have — to stop early when the evidence is overwhelming, which on a strongly biased coin often happens long before the fixed sample size, and to continue when it is not, without any adjustment.

The cost is not a defect of the particular e-value either. Part of it is the price of not committing to an alternative, which a researcher who really did know the effect size could avoid by staking on it, and part of it is that a threshold of twenty is a demand that holds under every possible stopping rule, including adversarial ones. A test that is promised to be used only once can afford a lower bar.

What a p-value is worth as evidence #

E-values and p-values can be converted into each other, and the conversion is deeply asymmetric.

In one direction it is free: $1/E$ is a valid p-value, so an e-value of twenty is worth a p-value of 0.05. In the other direction, a function $f$ turns a p-value into an e-value if its average over a uniform $p$ is at most one, and the most any of the standard family $\kappa\, p^{\kappa - 1}$ can give at a particular $p$ is

$$\frac{1}{-e\, p \ln p},$$

which at $p = 0.05$ is 2.46. A result significant at 5% is worth at most about two and a half to one against the null as betting evidence; a result at $p = 0.005$ is worth about fourteen to one, and one at $p = 0.001$ about fifty-three. The same bound, found by Thomas Sellke, Maria Bayarri and James Berger in 2001 as the most a p-value could imply for a Bayes factor, is one of the strongest arguments made for lowering the conventional threshold to 0.005, and it is the general form of the disagreement that a result at five per cent that favours the null computed for one coin.

The asymmetry is the price of the p-value’s great convenience: it needs no alternative. An e-value is a bet, and a bet has to be on something. A p-value measures only how surprising the data are under the null, and surprise without a direction is cheap evidence.

No e-value is best against every coin #

The bet an e-value represents is a choice, and the choice decides what it is good at.

All four e-values in the figure respect the guarantee at a fair coin: 3.5%, 1.2%, 3.1% and 3.3% of fair coins pass twenty within two hundred tosses. Away from a half they part company. The likelihood ratio staked on 0.6 is the most powerful of the four for coins slightly biased towards heads — it detects a 0.56 coin 37% of the time, against 15% for the uniform prior — and it can never detect a coin biased towards tails at all, since it bets on heads at every toss. Among the three priors, the one with $a = 20$, concentrated near a half, beats the uniform prior at every chance on the axis and wrongly passes twenty least often; the prior with $a = \tfrac12$, which leans towards coins that almost always land one way, is the weakest. With two hundred tosses to spend, the biases worth betting on are the ones between a few hundredths and a fifth: smaller ones cannot be found in time by any method, and larger ones are found quickly by every method. A prior that puts its weight where the budget can make a difference does better than one spread evenly over biases that will never be in doubt — the same reasoning as choosing a sample size, made as a bet.

This is not a weakness of e-values but a fact about evidence that p-values hide. Every test is more powerful against some alternatives than others; a p-value’s power curve is fixed by the choice of test statistic, which is equally a choice, but it is made once and is rarely stated as a bet. An e-value makes the choice explicit — the prior is written down — and it pays for a wrong choice in growth rate rather than in validity: a badly chosen e-value is weak, never misleading.

The learned product of the first figure is the natural way to make the choice well over a sequence of studies. Each study bets on what the previous studies suggest, with exactly the confidence they justify, which is how evidence measured in decibans accumulates across separate sources: as a sum of logarithms, with nothing thrown away at the joins.

Averaging is always allowed #

There is a second way to combine e-values, and it needs even less. The average of any e-values for the same null is an e-value, whatever the dependence between them, simply because an average of expectations at most one is at most one. Five analyses of one data set — five priors, five test statistics, five researchers’ preferences — can be averaged, and the average keeps the guarantee.

What cannot be done is to pick the largest. The maximum of several e-values computed on the same data is not an e-value, for the same reason that reporting the most significant of several analyses inflates a p-value: under the null, each analysis has its own small chance of a lucky excess, and taking the maximum collects all of them. Nor can e-values computed on the same data be multiplied as though they came from separate studies; the product rule needed each factor to be an e-value given the ones before it, and a second analysis of tosses already counted is no such thing. The gambling picture makes both rules obvious. Averaging is splitting one’s money among several gamblers, each betting fairly; multiplying is letting one gambler reinvest winnings in new bets on new tosses; taking the maximum, or multiplying analyses of old tosses, is placing a bet after seeing how the coin landed.

These two rules — multiply across fresh data, average across analyses of the same data — are the whole arithmetic of e-values, and both follow from one line about expectations. By comparison, the rules for combining p-values depend on how the studies are related and are different for every combination method. It is the same contrast that ran through Kelly’s rule and through a fair game between two barriers: quantities that are fair bets compose simply, and quantities that are not have to be corrected one case at a time.

What the computations assume #

Every figure here concerns one coin, studies of fixed size, and tosses independent of one another. The guarantees hold far more generally — Ville’s inequality needs only that the running product is a non-negative supermartingale under the null, which holds for e-values computed in sequence under any dependence the null allows — but the measured numbers, the 8.7%, the factor of 2.1, the 2.46, belong to the setting drawn.

Two things the figures cannot show deserve to be said plainly. First, the null hypothesis here is simple: a fair coin is one fully specified distribution. Composite nulls — “the coin’s bias is at most 0.55”, or “the two treatments have the same effect, whatever it is” — are where e-values become genuinely hard, because an e-value must then have expectation at most one under every distribution in the null, and constructing powerful ones is an active subject. Second, e-values do not repair a study that is badly designed or a measurement that is biased; they repair one specific failure, the dependence of a statistical guarantee on a plan that nobody can verify.

Still open: the right alternative for a composite null #

The lead question that the essay on peeking ended on — which guarantee to ask for — has a partial answer here: ask for an e-value, and choose the bet according to the alternatives that matter. What is not settled is how to choose well when the null itself is a family. Peter Grünwald, Rianne de Heide and Wouter Koolen showed in 2019 that for composite nulls there is a growth-rate-optimal e-value, obtained by a reverse information projection of the alternative onto the null, and that it exists under broad conditions; but it can be hard to compute, it depends on which alternative is targeted, and for many standard problems — testing whether two proportions are equal, or whether a regression coefficient is zero with the noise level unknown — the practical recipes are recent and their power relative to classical tests is being mapped case by case. Whether e-values can replace p-values in routine practice without a large loss of power is, at the time of writing, an empirical question being answered one problem at a time.

Multiply, and say what was bet on #

An e-value is a gambler’s wealth from fair bets against the null, and that single fact gives it everything a p-value lacks across studies. Products of e-values are e-values, so evidence multiplies from study to study, and Ville’s inequality keeps the chance of ever reaching twenty below one in twenty under any rule for deciding whether to continue — where Fisher’s combination, with another study run whenever the evidence fell short, reached 8.7%. When each study’s bet is learned from the ones before, the product is exactly the evidence of the pooled data. The guarantee costs 1.4 to 2.1 times as many tosses for the same power, a p-value at 0.05 is worth at most 2.46 to one as a bet, and no e-value is best against every coin: the prior is a stated choice of which alternatives to be good at finding.