A share that depends on the average

Multiplying makes the digit one common explained Benford’s law for products. Multiply a few random numbers together and the product’s first digit is 1 about 30 per cent of the time and 9 under 5 per cent, because the logarithm of a product is a sum, sums spread out, and a logarithm spread evenly over many units has its fractional part spread evenly too — which is exactly the statement that the first digit is $d$ with probability $\log_{10}(1 + 1/d)$. It ended by listing which sequences are known to follow the law: the powers of 2, the Fibonacci numbers, the factorials.

The most obvious sequence of all was left off the list: the counting numbers $1, 2, 3, \ldots$ themselves. Do they follow Benford’s law? The question sounds as if it must have an answer, and the answer turns out to depend entirely on what “share” means. Counted in the ordinary way, the share of numbers beginning with 1 has no value at all. Averaged in one of several natural ways, it is exactly $\log_{10} 2$.

A share that never settles #

Count the numbers from 1 to $N$ that begin with the digit 1. At $N = 9$ there is one of them, a ninth. At $N = 19$ there are eleven, more than half. At $N = 99$ there are still eleven, back to a ninth. At $N = 199$ there are 111 out of 199. The count is easy to write down exactly — it is a sum over the decades of how much of each block $10^j, \ldots, 2 \cdot 10^j - 1$ lies below $N$ — and the share it gives swings for ever between two values.

When $N$ is one less than a power of ten, every decade is complete, and in each complete decade the numbers beginning with 1 are exactly a ninth of the numbers with that many digits; the share is just over a ninth. When $N = 2 \cdot 10^j - 1$, the latest decade’s numbers beginning with 1 have all been counted and none of the rest, and the share is $(10^{j+1} - 1)/9$ divided by $2 \cdot 10^j - 1$, which tends to $5/9$. Between those two moments the share climbs and falls along the sawtooth in the figure, and at a million million the swing is exactly what it was at a hundred: from 0.1111 to 0.5556.

So the set of numbers beginning with 1 has no natural density: the ordinary share among the first $N$ numbers does not converge. The average settles and the wobble does not is about sums of random quantities, whose averages converge; this is a fixed, completely regular set whose share does not, because its structure is set on a logarithmic scale — decades — while the counting is done on an ordinary one.

Averaging the average #

A share that oscillates can sometimes be rescued by averaging it. Take the share up to $N$, and average it over $N = 1, 2, \ldots, M$: this is the Cesàro mean, the standard way of assigning a value to a sequence that swings.

The averaged share still swings, through 0.127 instead of 0.444, and averaging again leaves 0.042, and again 0.014. Each averaging smooths the sawtooth into a gentler wave around $\log_{10} 2$, and each wave still has the period of a decade and still does not die out. Betty Flehinger proved in 1966 that the swings shrink to nothing only in the limit of infinitely many averagings, and that the value they shrink to is $\log_{10} 2$. No finite number of averagings gives a share.

The computation behind the figure is exact: the share was recorded for every $N$ up to ten million, and each average was taken over all of them. Each averaging acts like a filter that damps the decade-long wave by a fixed factor, roughly a third here, so three averagings leave about a thirtieth of the original swing and every further one takes off the same share again. The swing does not narrow as $N$ grows, for any of the four curves. It narrows only from one curve to the next.

Weighting by one over n #

The averaging that works in one step weights each number by $1/n$. Count the numbers up to $N$ beginning with 1, but give $n$ the weight $1/n$, and divide by the total weight $\sum_{n \le N} 1/n$, which grows like $\ln N$.

The swings die out, and each digit’s share closes in on Benford’s value. The reason is that the weight $1/n$ makes every decade count alike: the numbers from $10^j$ to $10^{j+1}$ carry total weight about $\ln 10$, whatever $j$ is, so no decade is favoured by being the latest one counted. And within a decade, the numbers beginning with $d$ run from $d \cdot 10^j$ to $(d+1) \cdot 10^j$, which carry weight about $\ln((d+1)/d)$ out of $\ln 10$ — exactly $\log_{10}(1 + 1/d)$. The logarithmic density of the numbers beginning with $d$ exists, and it is Benford’s value.

The convergence is slow, because the latest incomplete decade still contributes a swing of size about one decade’s weight divided by the total, and the total grows only like $\ln N$. At ten million the digit 1 stands at 0.311 against 0.301. But the swing does shrink, at a rate of $1/\ln N$, and that is the difference between a share that has a limit and one that does not.

A third average, the same answer #

Weighting by $n^{-s}$ for $s$ slightly larger than 1 is another standard choice, and number theory uses it constantly: it is the weighting behind the zeta function $\zeta(s) = \sum n^{-s}$, which almost no number is one used to count sums of two squares.

With $s = 2$ the weights favour the first few numbers heavily — the number 1 alone carries more than half the weight — and the share of numbers beginning with 1 is 0.644. As $s$ falls towards 1 the weights flatten, later decades count for more, and the share falls: 0.341 at $s = 1.1$, 0.305 at $s = 1.01$, 0.3031 at $s = 1.005$, heading for $\log_{10} 2$. This limit is the Dirichlet density, and a general theorem says that whenever the logarithmic density exists the Dirichlet density exists and agrees with it. If the sums are replaced by integrals the share has a closed form, $(1 - 2^{1-s})/(1 - 10^{1-s})$, whose limit as $s \to 1$ is exactly $\log 2/\log 10$.

Every digit, every swing #

The swing is not peculiar to the digit 1.

Every digit’s share swings through a decade. The digit 1’s swings from a ninth to five ninths; the digit 9’s from about a hundredth to a ninth. Benford’s value lies inside every range, but not in the middle: it lies where the logarithmic weighting puts it, much nearer the bottom for the large digits. The ordinary share is like a reading taken at a random moment in a cycle, and the logarithmic share is the cycle’s average weighted by how long each moment lasts on a logarithmic clock.

This is the same structure the longest run has no limit found in the longest run of heads, whose distribution cycled once per doubling of the number of tosses. In both cases a quantity defined on an ordinary scale is governed by a structure on a logarithmic scale, and the fractional part of a logarithm, which never settles, drives a cycle that never ends. The difference is that the longest run’s average settled while its probabilities cycled; here even the share cycles, because a share is itself an average of the wrong kind.

The primes #

The primes might be expected to differ, since they thin out across each decade. They follow the same pattern.

Counted, the share of primes beginning with 1 swings between 0.120 and 0.538 for $N$ above a million and goes on swinging, much as for the whole numbers. There is no natural density. Weighted by $1/p$ — the weighting whose total, $\sum 1/p$, grows like $\log \log N$ by Mertens’s theorem — R. E. Whitney proved in 1972 that the share tends to $\log_{10} 2$. The figure shows how slowly. With the weighting starting at 2 the share is 0.233 at a hundred million and rising; started at 11 it is 0.371 and falling. The two curves bracket the limit and close in on it as slowly as $\log \log N$ grows, which is to say barely at all: the primes 2, 3, 5 and 7, none of which begins with 1, carry more than a third of all the weight $\sum 1/p$ even at a hundred million, and their influence fades only as fast as the sum itself grows.

That slowness has the same source as the swing. The weight $1/p$ makes each decade of primes count about equally in the limit, since the primes in a decade carry weight about $\ln\ln 10^{j+1} - \ln\ln 10^j$, which is roughly $1/j$. Decade $j$ counts about $1/j$ — less as $j$ grows, but only slowly — so the early decades keep a large say for a very long time.

Which averages give Benford’s law #

The pattern is general. For any set of whole numbers defined by a condition on the leading digits, the ordinary share up to $N$ oscillates with period one decade in $\log_{10} N$ unless the condition happens to occupy the same share of every block of a decade. Any averaging that weights the decades equally in the limit — logarithmic, Dirichlet, infinitely iterated Cesàro — removes the oscillation and gives the share the logarithmic measure assigns, which for leading digits is Benford’s.

That is the precise sense in which the counting numbers “follow Benford’s law”: they do so for the averages that are invariant under rescaling, and for no others. The law is a statement about scale invariance, as the previous essay explained for products, and a share computed on an ordinary scale is not scale-invariant. A sequence like the powers of 2 follows the law in the ordinary sense because it is already spread evenly on a logarithmic scale, one term per $\log_{10} 2$ of logarithm; the counting numbers are spread evenly on an ordinary scale, and only a logarithmic average sees them evenly in the other sense.

A hierarchy of densities #

The averages in this essay form a hierarchy, and each level is stronger than the next. If a set has a natural density — its ordinary share up to $N$ converges — then its logarithmic density exists and is the same number, and so are its Cesàro means and its Dirichlet density. The implications run one way only. The numbers beginning with 1 sit exactly at the place where the hierarchy breaks: they have a logarithmic and a Dirichlet density and no natural one. Results that let a statement climb back back up the hierarchy, from a weaker average to a stronger, are called Tauberian theorems, and they always need an extra hypothesis — typically that the thing being averaged does not change too abruptly. The leading-digit indicator changes abruptly at every power of ten, and that is exactly the hypothesis it fails.

There is a lesson here about the phrase “a random whole number”. No probability distribution gives every whole number the same chance, so any statement of the form “a random number begins with 1 with probability $p$” is shorthand for one of these averages, and the shorthand hides which. For most sets that occur in practice — the even numbers, the squarefree numbers, the numbers with an even count of prime factors — all the averages agree and the choice does not matter. Sets defined by leading digits are the standard example where it does. An average that never settles showed a random quantity whose sample averages never converge because of its heavy tails; here nothing is random at all, and the averages still disagree, because of how the set is laid out along the line.

Logarithms that do and do not spread #

The cleanest way to see the difference between the counting numbers and the powers of 2 is through the fractional parts of their logarithms. A number’s first digit is decided by the fractional part of its base-ten logarithm: it begins with 1 exactly when that fractional part is below $\log_{10} 2$. For the powers $2^k$, the logarithms are $k \log_{10} 2$, the multiples of an irrational number, and their fractional parts are evenly spread in the ordinary sense — the theorem of Hermann Weyl that almost every orbit is fair set beside its counterpart for doubling. So the powers of 2 begin with 1 a share $\log_{10} 2$ of the time, counted the ordinary way.

For the counting numbers, the logarithms $\log_{10} n$ grow, but more and more slowly: from one power of ten to the next there are ten times as many numbers as in the decade before, all crowded into the same single unit of logarithm, so that the latest decade always outweighs everything before it put together. Their fractional parts are not evenly spread in the ordinary sense; they sweep from 0 to 1 once per decade, lingering ever longer, and the share below any threshold oscillates. Weighted by $1/n$, each number’s contribution is proportional to the length of logarithm it covers, and the sweep becomes uniform. A sequence growing like $n$ has evenly spread logarithms only on a logarithmic clock; a sequence growing like $2^n$ has them on an ordinary one.

Why this matters for data #

The distinction is not academic for the use Benford’s law is most often put to, the screening of financial and scientific data for fabrication. Data that span many orders of magnitude and arise from multiplicative processes follow the law in the ordinary sense, because their logarithms are spread evenly. Data that are uniformly spread over a fixed range — invoice numbers from 1 to some maximum, ages, page numbers — do not: their first digits depend on where the range ends, exactly as the share of counting numbers beginning with 1 depends on $N$. A test that expects Benford’s frequencies from such data will find a “deviation” that is nothing but the sawtooth in the hero figure, read off at one value of $N$. The law applies to data that are scale-free, and whether a data set is scale-free is a question about how it was generated, not about its digits.

Still open: the primes, more precisely #

Whitney’s theorem gives the logarithmic density of primes with each leading digit. Finer questions remain. How fast the weighted share converges, with an explicit error term, depends on the error in the prime number theorem over short ranges, and under the Riemann hypothesis the error is much better controlled than what is known unconditionally. For primes in thin sets — primes of the form $n^2 + 1$, twin primes — the logarithmic density of leading digits is expected to be Benford’s for the same reason, but proving it needs equidistribution results for those sets that are not available, the same obstacle the same bell on thinner sets met for their counts of prime factors. And for sequences mixing addition and multiplication, the open problem the previous essay ended on, even the logarithmic version is open.

Counting on the wrong scale #

The share of whole numbers beginning with 1 is a fixed property of a fixed, completely known set, and it has no value. Counted up to $N$ it swings between a ninth and five ninths for ever; averaged over $N$ it still swings, more gently, however many times the averaging is repeated a finite number of times; weighted by $1/n$, or by $n^{-s}$ as $s$ falls to 1, it settles at $\log_{10} 2$. The primes do the same, more slowly. Benford’s value for the counting numbers is real, but it belongs to the logarithmic scale on which their leading digits are defined, and an ordinary count, made on the wrong scale, never finds it.