Symmetry forces a right angle

Rewriting a map against its own invariant directions turns it into two independent stretches, and the construction is only as good as the grid it produces. A skewed grid is a bad grid: lengths and angles measured against it are not the lengths and angles anyone means, and the matrix that translates to and from it can be arbitrarily badly behaved.

So the question is which maps have perpendicular invariant directions. The answer is a condition on the four entries that can be checked at a glance, and it is the strongest theorem in elementary linear algebra.

The condition #

A matrix is symmetric when it equals its own reflection across the leading diagonal: $b = c$ in

$$A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}.$$

The theorem is that a symmetric matrix has real eigenvalues and perpendicular eigenvectors, always, with no exceptions and no conditions attached. Neither half holds in general — a rotation has no real eigenvalue at all, and plenty of unsymmetric matrices have two perfectly good real eigenvalues at some awkward angle to each other.

Why the eigenvalues must be real #

The discriminant of the characteristic polynomial is $(a + d)^2 - 4(ad - bc)$, which for a symmetric matrix, with $c = b$, expands to

$$a^2 + 2ad + d^2 - 4ad + 4b^2 = (a - d)^2 + 4b^2.$$

A sum of two squares is never negative, so the discriminant is never negative, so both roots are real. That is the whole argument in two dimensions, and it is unusually satisfying because the answer arrives as a sum of squares — the same shape as the reason a distance is what it is, appearing here in an equation about a polynomial.

The corresponding statement in $n$ dimensions cannot be got at this way — there is no discriminant to expand — and the usual route is through the complex numbers. Allow the eigenvalue $\lambda$ and the eigenvector $v$ to be complex, and compute $\bar v^{\mathsf{T}} A v$ twice over: once as $\lambda\,\bar v^{\mathsf{T}} v$, and once by taking the conjugate transpose of the whole expression, which for a real symmetric $A$ returns $\bar\lambda\, \bar v^{\mathsf{T}} v$. Since $\bar v^{\mathsf{T}} v$ is a sum of squared moduli and so is a positive real number, $\lambda = \bar\lambda$, and a number equal to its own conjugate is real.

It is the same argument as the one about perpendicularity in the next section, run with a conjugate in it. That is not a coincidence, and it is the reason the two halves of the theorem are usually proved in the same breath.

It also says when the two eigenvalues coincide: the discriminant is zero only when $a = d$ and $b = 0$, which is to say only when the matrix is a multiple of the identity. So a symmetric matrix with a repeated eigenvalue is a plain scaling, and every direction is invariant. The defective case cannot happen here — there is no symmetric shear.

Why the directions must be perpendicular #

This half is one line and it uses the dot product as the measure of angle.

Take eigenvectors $u$ and $v$ with different eigenvalues $\lambda$ and $\mu$, and evaluate the quantity $u \cdot Av$ two ways. On one hand $Av = \mu v$, so it equals $\mu (u \cdot v)$. On the other, symmetry lets the matrix move across the dot product — $u \cdot Av = (Au)\cdot v$, because both sides are $\sum_{i,j} u_i a_{ij} v_j$ and swapping $a_{ij}$ for $a_{ji}$ changes nothing when the two are equal — so it also equals $\lambda(u\cdot v)$.

Therefore $\lambda(u \cdot v) = \mu(u \cdot v)$, and since $\lambda \neq \mu$ the only escape is $u \cdot v = 0$.

The step doing the work is the matrix can be moved from one side of the dot product to the other, and that is what symmetry means when it is stated about the map rather than about the array of numbers. It is the definition worth carrying, because it is the one that survives into infinite dimensions, where matrices do not.

The curve that shows it #

The theorem has a shape, and the shape is the level set of the associated quadratic form

$$Q(x, y) = ax^2 + (b + c)xy + dy^2,$$

which is the number $x \cdot Ax$ written out. The set where $Q = 1$ is a conic section, and when both eigenvalues are positive it is an ellipse.

The claim the picture makes is that the eigen-directions are the axes of that ellipse, with semi-axis lengths $1/\sqrt{\lambda}$. Both halves are measured off the drawn curve rather than derived: the curve is generated by walking the angle and solving $Q = 1$ for the radius, every point on it is checked to satisfy the equation, and its longest and shortest radii are found and compared with $1/\sqrt{\lambda}$.

Once stated that way the theorem is almost obvious, which is a good sign and a bad reason to believe it. An ellipse has two axes, they are perpendicular, and every ellipse has them. Since the level set of any positive symmetric form is an ellipse, the axes are there whether anyone looks for them or not, and the content of the theorem is that those axes are the eigen-directions rather than some unrelated pair.

That is the direction of the argument worth noticing. The geometry does not follow from the algebra here; the two are the same statement, and the reason the ellipse always has perpendicular axes is the reason the matrix always has perpendicular eigenvectors.

Where symmetric matrices come from #

A theorem about a special class of matrices is only worth having if the class turns up, and this one turns up almost everywhere, for a reason that is the same reason each time.

A symmetric matrix is what appears whenever a quantity is built from unordered pairs. The matrix of second derivatives of a smooth function is symmetric because the order of differentiation does not matter. A covariance matrix is symmetric because the covariance of $x$ with $y$ is the covariance of $y$ with $x$. The adjacency matrix of a graph whose edges have no direction is symmetric because an edge joins two things rather than pointing from one to the other. A matrix of distances, of resistances, of spring constants, of overlaps — all symmetric, all for the same grammatical reason.

Unsymmetric matrices appear where there is a direction: a transition from one state to another, a flow along an arrow, a derivative of one variable with respect to a different one. So the question is this matrix symmetric is usually not a question about the numbers at all. It is a question about whether the situation has a preferred direction, and the answer is available before any entry has been computed.

That is a useful thing to notice while modelling, and an uncomfortable one. It means the good behaviour is not a bonus discovered on inspection; it was decided by the shape of the question, and a problem whose matrix comes out unsymmetric is one where none of this is available and no amount of computation will produce it.

Reading the eigenvalues off the picture #

The correspondence goes both ways and it turns questions about matrices into questions about a drawing.

If both eigenvalues are positive, $Q$ is positive everywhere except the origin and the level set is an ellipse; such a form is called positive definite. If they have opposite signs, $Q$ takes both signs and the level set is a hyperbola, opening along the direction where $Q$ is positive. If one is zero, the form is a perfect square, the level set is a pair of parallel lines, and the matrix has flattened the plane onto a line.

So the sign pattern of a symmetric matrix’s eigenvalues is legible from the shape of one curve, without computing anything. Testing whether a form is positive definite is a common question — it is what decides whether a critical point of a surface is a minimum, a maximum or a saddle — and this is why the second-derivative test in two variables is a statement about a determinant and a trace rather than a statement about eigenvalues. The trace and determinant are the polynomial’s coefficients, and their signs already say which of the three shapes has been drawn.

What the form cannot see #

Here is the point at which the unsymmetric figure above earns its place.

The quadratic form $x \cdot Ax$ multiplies out to $ax^2 + bxy + cxy + dy^2$, in which $b$ and $c$ appear only as their sum. Two matrices differing by anything antisymmetric — the same diagonal, and off-diagonal entries $b + t$ and $c - t$ — give exactly the same function at every point of the plane.

So the level curve cannot distinguish them, and the curve drawn over an unsymmetric matrix is really the curve of its symmetric part $(A + A^{\mathsf{T}})/2$. That is why the eigen-directions in that figure are not the axes: the axes belong to a different matrix, one that agrees with $A$ on every value of the form and disagrees with it about which directions are invariant.

This is worth stating plainly because it is a trap with an inviting shape. A form and a matrix look like the same information written twice, and they are not: the form throws away the antisymmetric part, which for a two-by-two matrix is one number. When the matrix is symmetric that number is zero and nothing is lost, which is exactly why the theory of quadratic forms is a theory about symmetric matrices and mentions no others.

The two halves are one fact #

It is worth putting the algebra and the geometry side by side, because they are usually taught as separate results and they are not.

The algebraic statement is that $A = QDQ^{\mathsf{T}}$ for some matrix $Q$ whose columns are perpendicular unit arrows — the same $PDP^{-1}$ as the previous rung, with the extra information that $P$ can be taken to be a rotation, so that $P^{-1}$ is simply its transpose. No inversion is required and nothing is badly conditioned, because a rotation is as well behaved as a matrix gets.

The geometric statement is that the level curve of the form has axes. And the bridge is that $Q$ is the rotation taking the coordinate axes to those axes, so the two sentences differ only in whether the rotation is named or drawn.

That $P^{-1} = P^{\mathsf{T}}$ is the practical payoff and it is easy to undersell. Every worry in the previous rung — an inverse that might be enormous, a grid that might be nearly degenerate, a decomposition that exists but is useless to compute with — evaporates. A symmetric matrix is diagonalisable by a rotation, and a rotation never distorts anything.

What it is for #

The spectral theorem is the reason a great many computations are possible at all, and three uses are worth naming because they look unrelated.

Principal axes. Any measurement of spread in the plane or in space — a cloud of data, the mass distribution of a rigid body, the strain in a material — is a symmetric matrix, because it is built from products $x_ix_j$ that do not care about order. Its eigen-directions are perpendicular axes along which the spread decouples, and the eigenvalues are the spread along each. Nobody chooses those axes; the symmetry produces them.

Simultaneous diagonalisation, and why a conic has a normal form. Rotating the plane by the right angle turns $ax^2 + bxy + dy^2$ into $\lambda_1 x'^2 + \lambda_2 y'^2$ with no cross term. That is precisely the classification of the conic sections into ellipse, hyperbola and parabola, obtained without touching a cone.

A test that is not a test. The symmetry of the second-derivative matrix is why a surface has a well-defined pair of principal curvatures at each point, perpendicular to one another, and why a saddle is a saddle in a way that survives any smooth change of coordinates. The signs of two eigenvalues classify every non-degenerate critical point of every smooth surface, and the classification is exhaustive because a symmetric matrix has nothing else to offer.

Real spectra where they are needed. Many settings require the eigenvalues to be real for the answer to mean anything — a rate of growth, a frequency of vibration, a variance. In every one of those settings the matrix turns out to be symmetric, and the symmetry is not a convenience noticed afterwards. It comes from the same fact about the situation that makes the question meaningful.

What the picture cannot show #

Two things, and the second is the sharper.

The drawing works only when both eigenvalues are positive, because that is when the level set is a closed curve that fits on a page. A form with eigenvalues of opposite signs has a hyperbola for its level set, whose axes are still perpendicular and still the eigen-directions, but which runs off to infinity in both directions and cannot be shown whole. The generator refuses those matrices rather than drawing half of one.

And the theorem is about $n$ dimensions, where its content is much larger than anything two dimensions can suggest. In the plane, “two perpendicular directions” is nearly forced — any two directions that are not parallel can be straightened out by a change of scale, so perpendicularity is a small extra. In a hundred dimensions the claim is that a hundred mutually perpendicular directions exist, which is not nearly forced by anything, and the proof there is genuinely different: it maximises the form on the unit sphere, uses the fact that a continuous function on a closed bounded set attains its maximum, and then repeats the argument on what is left over.

The ladder from here #

Rungs above: the frames every map has, symmetric or not, and what they are good for when there is no eigenvector to be had. Simultaneous diagonalisation of two forms, which is where generalised eigenvalue problems come from. The variational characterisation, in which the eigenvalues are the successive maxima of the form on the sphere. Positive definiteness as an order on matrices. And the infinite-dimensional version, where the same proof runs and the matrix has become an operator on functions.