An error with an unknown in it

The rung below drew partial sums peeling away from a curve and said, correctly, that each extra term buys a wider interval. It did not say how wide, and the difference between a picture of curves lying near each other and a statement about how near is the whole of what makes a series useful.

Taylor’s theorem supplies the statement, and its shape is strange. After $n$ terms the error is exactly

$$R_n(x) = \frac{f^{(n+1)}(\xi)}{(n+1)!}\,x^{n+1}$$

for some $\xi$ between $0$ and $x$. That is an equation, not an estimate — but it contains a number nobody can produce.

Replacing what cannot be found #

The theorem’s usefulness comes entirely from one substitution. Nobody knows $\xi$, but $\xi$ lies between $0$ and $x$, so whatever $f^{(n+1)}(\xi)$ is, it is no larger than the biggest value that derivative takes on that interval. Call that biggest value $M$. Then

$$|R_n(x)| \le \frac{M\,|x|^{n+1}}{(n+1)!},$$

and every number in that expression is available. An equality containing an unknown has become an inequality containing none, which is the manoeuvre the theorem exists for, and the price is that the answer is now a bound rather than a value.

For $\sin x$ the substitution costs almost nothing to make, because every derivative of the sine is $\pm\sin$ or $\pm\cos$ — the two functions that keep turning into each other — and so is bounded by $1$ everywhere. Taking $M = 1$ gives $|R_n(x)| \le |x|^{n+1}/(n+1)!$, which goes to nought for every $x$ whatever, and that single line is the proof that the sine series works on the whole real line.

For most functions the substitution is where all the difficulty is. Bounding a high derivative over an interval is a real problem, and it is the reason a theorem this clean is so often unusable. The exponential is the other easy case, and it is easy for the opposite reason — not because its derivatives stay small but because they are all the same function, so the maximum is wherever the function’s own is.

What the substitution gives away #

The bound is not the error and the gap between them is worth seeing, because the size of the gap is what decides whether a bound is a useful thing to have.

The two curves are parallel because both are proportional to $x^{n+1}$, and separated because the derivative at the worst point is larger than the derivative at $\xi$. So the bound gets the exponent right and the constant wrong, which is exactly the trade a bound is supposed to make: the shape of the error is preserved and the size is overstated.

That is worth stating as a principle because the alternative failure is much worse. A bound that got the exponent wrong would be useless at every scale; a bound that overstates by a factor of three is useful everywhere, since the exponent is what decides how many terms are needed and a constant factor changes the answer by less than one term.

Where the point actually is #

The unknown $\xi$ is usually left unknown, on the grounds that the whole method is to avoid needing it. That is sound practice and it hides something worth seeing.

For a function whose $(n+1)$-th derivative is monotone the defining equation can simply be solved for $\xi$, by bisection, to whatever accuracy is wanted. And the answer is not arbitrary at all.

As $x$ shrinks, $\xi/x$ approaches $1/(n+2)$. The reason is not deep and is worth having: for small $x$ the remainder is dominated by the first omitted term, so

$$\frac{f^{(n+1)}(\xi)}{(n+1)!}x^{n+1} \approx \frac{f^{(n+1)}(0)}{(n+1)!}x^{n+1} + \frac{f^{(n+2)}(0)}{(n+2)!}x^{n+2},$$

and expanding the left-hand side’s derivative about zero as $f^{(n+1)}(0) + \xi f^{(n+2)}(0)$ and matching gives $\xi = x/(n+2)$.

So the mean-value point is not somewhere unknowable in the middle; it is very near the expansion point, at a fraction of the interval that shrinks as the degree grows. The bound puts it at the far end, which is where it never is. That is the whole reason the gap in the previous figure is as wide as it is, and it says the loss from taking the worst case grows with the degree rather than shrinking.

It also settles a question the theorem’s phrasing invites and does not answer: whether $\xi$ is unique. It is not, in general — the equation asks for a point where the derivative takes a particular value, and a derivative that is not monotone can take that value several times. The figures here restrict to two functions whose relevant derivative is monotone, which is why a bisection finds the point at all and why “the” mean-value point is a defensible phrase on this page and not elsewhere. For $\sin x$ at a degree where the derivative oscillates across the interval, there are several such points, all of them equally entitled to the name, and the theorem is indifferent between them.

None of that damages the theorem, because the theorem never claimed uniqueness and the substitution that follows does not need it. It is worth noticing anyway, because a reader who has watched a curve of $\xi$ values being computed will reasonably assume the curve is well defined, and for most functions it is a curve through one of several available choices at every point.

Two functions with oppositely-behaved derivatives, and the same limit at the origin. That the limit is universal is the informative part: it is a statement about the first omitted term, which every function has, rather than about $e^x$ or $\ln(1+x)$.

The other form, which has no unknown at all #

There is a second expression for the remainder, and it is the one to reach for when the bound is not good enough.

$$R_n(x) = \frac{1}{n!}\int_0^x (x-t)^n f^{(n+1)}(t)\,dt .$$

Nothing in it is unknown. It is an exact statement, derived by integrating by parts $n$ times starting from the fundamental theorem of calculus, and Lagrange’s form follows from it in one step: the weight $(x-t)^n$ does not change sign on the interval, so the mean value theorem for integrals says the integral equals $f^{(n+1)}(\xi)$ times the integral of the weight alone, for some $\xi$ — and the integral of the weight alone is $x^{n+1}/(n+1)$.

So Lagrange’s form is the integral form with the mean value theorem applied, and the $\xi$ that cannot be found is the point that theorem asserts without producing. Everything unknown in the first form arrived at that step and is not there in the second.

The integral form is worth more than a curiosity in two situations. When the derivative can be integrated but not usefully bounded — which happens whenever it oscillates, since the oscillation cancels in an integral and does not in a supremum — the integral form gives the right answer where the bound gives a useless one. And when a sharper constant is wanted, the integral form supplies it, because it is exact.

The cost of a bound nobody can compute #

The uncomfortable fact about Taylor’s theorem in practice is that its bound needs a quantity that is usually harder to get than the thing being estimated.

$M$ is the largest value of the $(n+1)$-th derivative on an interval. Finding it means differentiating $n+1$ times, which for a function given by a formula is mechanical and produces expressions that grow explosively, and then maximising the result, which is a second problem of the same kind. For $\sin$ and $e^x$ the derivatives have a pattern and the bound is free; for almost anything else it is not.

The factorial in the denominator is the reason the method works at all when it works. $M$ can grow with $n$ — for $\ln(1+x)$ it grows like $n!$ — and whether the bound goes to nought is a race between that growth and the $(n+1)!$ underneath. For the sine, $M$ is constant and the factorial wins outright. For $\ln(1+x)$ the two nearly cancel and what is left is $|x|^{n+1}/(n+1)$, which goes to nought only for $|x| < 1$ — a sum of reciprocals weighted by a geometric factor, and the geometric factor is doing all the work.

So the radius of convergence is a statement about how fast the derivatives grow, and that is the honest general description of it. The famous case where nothing on the real line explains the radius is the case where the derivatives grow for a reason invisible on the line, and it is the subject of the next rung.

Counting terms, which is what the bound is for #

The bound’s real job is not to describe an error but to answer a question of the form how many terms, and that question is answered by solving the inequality rather than by evaluating anything.

For $\sin x$ on $[-\pi/4, \pi/4]$, with $M = 1$, the requirement $|x|^{n+1}/(n+1)! < 10^{-10}$ is satisfied first at degree $13$, where the bound is $3.9\times10^{-13}$. Pushing to the limit of double-precision arithmetic, $10^{-16}$, needs degree $16$. Those are the numbers a library implementer starts from, and they were obtained without evaluating a sine anywhere.

Now move the same question out to $x = 3$ and the answer becomes degree $21$; out to $x = 20$ and it becomes degree $71$. The count grows with the interval and not with the accuracy, which is the single most useful consequence of the factorial: buying six more decimal places at a fixed $x$ costs a handful of terms, and moving $x$ out by a factor of six costs fifty.

That asymmetry is the entire reason for argument reduction. No routine evaluates $\sin(20)$ from the series; it subtracts multiples of $2\pi$ until the argument is small, and then uses a degree-13 polynomial. The reduction is where all the difficulty and all the cleverness live, because subtracting a multiple of an irrational number from a floating-point value loses precisely the digits that matter.

There is a second reason the direct evaluation is hopeless at $x = 20$, and the bound does not show it. The terms of the series there rise before they fall: the largest is the nineteenth, at about $4.3\times10^{7}$, and the sum of the whole series is at most one. So computing $\sin(20)$ by adding terms means cancelling eight digits away, and a double carries sixteen — half the precision is gone before the answer appears. The remainder bound is about the mathematics and says nothing about the arithmetic, and a series that converges beautifully can be numerically useless, which is the same distinction Gram–Schmidt’s two orders make one field away.

For $\ln(1+x)$ the picture is worse and for the reason the bound predicts. At $x = 0.5$ reaching $10^{-10}$ takes degree $28$; at $x = 0.9$ it takes degree $169$. The bound is $|x|^{n+1}/(n+1)$ with only a single power of $n$ underneath rather than a factorial, so the convergence is geometric rather than factorial and slows to a crawl as $x$ approaches the radius. Nobody computes a logarithm this way either.

Where the theorem needs a condition #

The function must have $n+1$ derivatives on the interval, not merely at the point. That is a stronger hypothesis than the one needed to write the polynomial down, which needs only $n$ derivatives at a single point, and the difference is not pedantic: the polynomial exists for a function whose $(n+1)$-th derivative does not, and the theorem then says nothing at all about how good it is.

The interval must contain both the centre and the point being estimated. For $\ln(1+x)$ at $x = -0.9$ the interval reaches into the region where the derivatives are enormous, and the bound reflects that correctly by being enormous. A bound taken over the wrong interval is not conservative; it is wrong.

And the $\xi$ depends on $x$, on $n$, and on the function. It is a different point for every one of the 161 samples in the figures above, which is why the phrase “for some $\xi$” is doing more work than it looks like. The figures compute it separately at every point, and the curve they draw is a curve of different mean-value points rather than the path of one.

A theorem about a single derivative, generalised #

Taking $n = 0$ collapses everything above into a statement that is older and more famous than the rest of it. The polynomial is the constant $f(0)$, the remainder is $f'(\xi)\,x$, and the equation reads

$$f(x) = f(0) + f'(\xi)\,x,$$

which is the mean value theorem: somewhere in the interval the slope equals the average slope across it.

So Taylor’s theorem is the mean value theorem with more terms, and every difficulty in it is present already at $n = 0$. The point $\xi$ is unnamed there too. The theorem asserts its existence by an argument — Rolle’s — that produces no way of locating it. And the standard use is the same substitution: bound the derivative on the interval, and the mean value theorem becomes a bound on how much a function can change.

That descent is worth making because it says where the strength of the result is. A tangent line is the flat map that fits closest — a slope read at a single point — and the mean value theorem is what converts “fits closest” into a number. Everything above is that conversion, done to order $n$ instead of to order one.

Where the form came from #

Taylor published the series in 1715 and did not prove anything about the error; the question of whether a series converges to the function it was built from was not one anybody was asking in 1715, and the eighteenth century used these expansions freely and successfully without it.

Lagrange supplied the remainder in 1797, in a book whose entire purpose was to found the calculus on power series and remove limits from it altogether. The programme failed — the flat function of the rung below is one of the reasons — and the remainder survived it, which is a common enough fate for a technical result inside a failed programme.

Cauchy gave both the integral form and a second version of the Lagrange form in the 1820s, as part of the work that made convergence a question with an answer rather than an assumption. That is the right chronology to keep: the error term arrived eighty years after the series, and the reason to want it arrived with it. Nobody needs a remainder until somebody asks whether the series is right.

What the pictures cannot show #

Every figure here is a plot of quantities a machine computed, and two of them are quantities the theorem says exist without saying how to find. The $\xi$ curve is the output of two hundred bisections per point, and its accuracy depends on the remainder being computed as the tail of the series rather than as a difference — the first version of that figure subtracted, lost every significant digit at $x = 10^{-3}$, and reported a fraction of $1.86$ for a quantity that cannot exceed one.

The bound curves are maxima over 121 sampled points of each interval, which is a lower bound on the true supremum. For the three functions drawn the derivative is monotone or bounded by one, so the sample cannot miss anything; for a function whose derivative oscillates rapidly it could, and no figure would show it.

And no picture here shows the theorem. What is drawn is the theorem’s conclusion at particular degrees for particular functions, and the conclusion is a statement quantified over every $x$, every $n$ and every sufficiently differentiable $f$. The figures are 161 instances apiece, which is evidence that the implementation is right and not evidence for the result.

The ladder from here #

Rungs above: the centre as a choice, where expanding about a different point moves the interval and analytic continuation walks a series past its own radius. Complex differentiability, and why once implies always. Padé approximants, which allow a denominator and reach past the radius the polynomial cannot. The points that ruin the fit, where a polynomial through equally spaced samples fails on the same function for a different reason. And the asymptotic series, whose terms eventually grow and which is still the right tool — an expansion that never converges and whose first few terms beat any convergent series available.

An unknown that is not a defect #

The habit worth carrying out of this rung is about the shape of the theorem rather than about series.

An existence statement with an unnamed witness — there is a point where, there exists a constant such that — reads like a weakened version of a theorem that names the thing. It usually is not. Naming $\xi$ would require solving an equation involving the function’s high derivatives, which is harder than the original problem; the unnamed version is what makes the theorem short enough to be usable, because the next step throws the name away regardless.

The test of whether that is honest is whether the substitution loses anything that matters. Here it loses a constant factor and keeps the exponent, so nothing that matters is lost. Where the substitution would lose the exponent — where the worst case over the interval is a different order of magnitude from the typical case — the theorem is still true and the practice built on it is worthless, which is the situation the integral form exists for.