Matpic
mathematics, in pictures
Concept

Least squares

The rule that fits a model to data by minimising the total of the squared discrepancies. Squaring is what makes the answer a projection: the fitted values are the point of the subspace the model can reach that is nearest the data, and the leftover meets that subspace at a right angle.

Named by 4 essays across one field — each of them below, with the objects they name alongside it.

One subtraction clears a direction. Gram–Schmidt on two planar vectors, in 3 panels: the pair as given, the shadow of the second on the first, and the perpendicular pair that is left when the shadow is removed.

One subtraction clears a direction

A basis is a set of directions to measure along, and most bases are awkward because the directions get in each other's way. Removing one shadow at a time turns any basis into one where every coordinate is a shadow and nothing interferes.

algebra · Inner product The nearest point of the plane the columns span. A target vector in space, the plane spanned by two columns, the point of that plane nearest the target, and the residual joining them, which meets the plane at a right angle.

The nearest point of a flat thing

More equations than unknowns almost never have a solution. Asking instead for the point of a plane nearest to where the answer should have been turns an unanswerable question into a shadow, and the shadow is what a line of best fit is.

algebra · Inner product The map moved to the other side of the product. Two panels over the unit curve of the product u₁v₁ + u₂v₂. The left applies A to u and measures it against v, giving 2.328; the right applies the adjoint to v and measures it against u, giving the same number.

Moving a map across a product

The transpose looks like a fact about a matrix: reflect its entries in the diagonal. It is a fact about the inner product. Measure lengths and angles differently and the map that slides to the other side of the product is a different matrix, and a matrix that was symmetric stops being so.

algebra · Inner product Shrinking beats the nearest point at every true mean. p=3: ratio at |θ|²/p = 0, 1, 5: 0.667, 0.863, 0.976; p=5: ratio at |θ|²/p = 0, 1, 5: 0.400, 0.732, 0.931; p=10: ratio at |θ|²/p = 0, 1, 5: 0.200, 0.621, 0.886; p=25: ratio at |θ|²/p = 0, 1, 5: 0.080, 0.549, 0.855

The nearest point is not the best guess

Measure ten unrelated quantities once each, with noise, and report the measurements. Nothing seems better. Yet pulling all ten towards nought by an amount the data choose gives a smaller total error whatever the true values are — in three dimensions or more, and never in one or two. The reason is a right angle: in many dimensions noise is nearly perpendicular to the truth, so a measurement is always too long, and the best correction is the shadow of the truth on it.

algebra · Inner product

Named alongside it

The objects these essays reach for when they reach for this one.

Inner productOrthogonalityProjectionResidualSubspaceApproximationBasisDot productGram schmidtLinear independenceOrthonormal basisPositive definite

All concepts