Determinants, Eigenvalues & the Spectral Theorem

Units 1–3 gave us the machinery: matrices as machines that move space (Unit 1), the spaces they act on (Unit 2), and rulers to measure with (Unit 3). Now we interrogate the machine itself. If a matrix is a motion, what are its invariants — its fingerprints? We'll find exactly two kinds: numbers that summarize the whole motion (determinant, trace) and directions that pass through it unturned (eigenvectors, each wearing its own scale factor $\lambda$).

The route: ① the determinant — a volume dial with an alarm bell at zero; ② how to actually compute it (minors, cofactors, and the lazy-row trick); ③ the trace, an absurdly easy number with a hidden double life; ④ eigenvectors — hunting the directions a matrix cannot bend, first by dragging, then by algebra (the characteristic equation); ⑤ the Spectral Theorem — for symmetric matrices the fingerprints organize into a perfect perpendicular grain, $A=Q\Lambda Q^{\top}$; ⑥ two encores that ML uses daily: complex eigenvalues (rotations) and the Cholesky square root of a covariance matrix.

Before hunting special directions, we need one number for what a matrix does to space as a whole. Take the unit square, feed its corners through $A$, and watch it land as a parallelogram — remember from Unit 1 that the columns of $A$ are exactly where the basis arrows $\mathbf{e}_1,\mathbf{e}_2$ go. The determinant is the signed area of that parallelogram:

Compute one to see how cheap it is: the widget's opening matrix $A=\begin{bmatrix}2&1\\1&1.5\end{bmatrix}$ has $\det A=(2)(1.5)-(1)(1)=2$ — this machine doubles every area it touches. Now the three readings of the number. Magnitude: $|\det A|$ is the scaling factor for area (volume in 3D) — $\det=2$ doubles every area, $\det=\tfrac12$ halves it. Sign: negative means the motion flips space over, like a mirror — orientation reverses. Zero: the alarm bell. The parallelogram has collapsed to a line segment: the columns are linearly dependent, space has been squashed flat, information is destroyed, and no inverse can ever rebuild it. $\det A=0\;\Leftrightarrow\;A$ singular $\Leftrightarrow\;\operatorname{rank}A<n$ — Unit 1's "machine that squashes" now has a one-number detector.

Geometry says what the determinant means; the cofactor expansion is how we compute it. The idea is recursive: an $n\times n$ determinant is a weighted sum of $(n-1)\times(n-1)$ determinants. Delete row $i$ and column $j$ and the small determinant that remains is the minor $M_{ij}$; dress it with a checkerboard sign and it becomes the cofactor $C_{ij}=(-1)^{i+j}M_{ij}$. Then, expanding along any row $i$ — or any column $j$ — gives the same number:

That "any row, any column, same answer" is not obvious — it is a small miracle you can test below on the lecture's $3\times3$ example. And it hands you a strategy for free: expand along the line with the most zeros, because every zero entry deletes an entire cofactor's worth of work.

Two behaviours make determinants elimination-friendly, and the lecture proves both by induction: swapping two rows multiplies $\det$ by $-1$ (one more mirror), and adding a multiple of one row to another changes nothing. So you can run Unit 1's Gaussian elimination, reduce $A$ to a triangle $U$, and read the answer off the diagonal — because for triangular matrices the determinant is just the product of the diagonal entries:

The determinant's understated sibling costs almost nothing: the trace is the sum of the diagonal,

and it behaves beautifully: $\operatorname{tr}(A+B)=\operatorname{tr}A+\operatorname{tr}B$, $\operatorname{tr}(\alpha A)=\alpha\operatorname{tr}A$, $\operatorname{tr}(I_n)=n$, and — the workhorse — the cyclic property $\operatorname{tr}(AB)=\operatorname{tr}(BA)$, which holds even when $AB\ne BA$, even when the two products have different sizes. Countless ML derivations (gradients of $\operatorname{tr}(X^{\top}AX)$, for one) lean on that single identity.

Why should a doctor's quick pulse-check deserve a section? Because of a secret we'll prove in Section 6: the trace is the sum of the eigenvalues, and the determinant is their product. Two numbers you can read straight off the matrix silently summarize the spectrum you haven't computed yet.

Now the star of the unit. A matrix moves every vector — but a few special vectors it can only stretch or flip along their own line, never knock off it. Those unbendable directions are the eigenvectors, and each one carries a personal stretch factor, its eigenvalue:

Positive $\lambda$ stretches ($\lambda>1$) or squeezes ($0<\lambda<1$) along the direction; negative $\lambda$ flips it end-over-end and then scales. The zero vector is banned from the club — $A\mathbf{0}=\lambda\mathbf{0}$ holds for every $\lambda$, so it points at nothing. But if $\mathbf{x}$ qualifies, so does every non-zero multiple $c\mathbf{x}$: eigenvectors are really eigen-directions, whole lines the matrix owns.

You can verify a claimed eigenvector with one multiplication. The lecture's example: for $A=\begin{bmatrix}1&6\\5&2\end{bmatrix}$ and $\mathbf{u}=\begin{bmatrix}6\\-5\end{bmatrix}$,

— so $\mathbf{u}$ is an eigenvector with $\lambda=-4$: $A$ flips it and stretches it by 4, but the line through $\mathbf{u}$ is untouched. Try $\mathbf{v}=(3,-2)$ instead: $A\mathbf{v}=(-9,11)$, which is on nobody's line through $\mathbf{v}$ — not an eigenvector. Below, hunt eigen-directions by hand: drag the probe around the circle and feel for the angles where $A\mathbf{x}$ snaps into alignment.

The algebra takes three lines. Rearrange the eigen-equation:

A non-zero vector mapped to zero means $A-\lambda I$ is singular — full rank lost — and by the bridge theorem that happens precisely when

This is the characteristic equation. Its left side is a degree-$n$ polynomial in $\lambda$ — the characteristic polynomial $p_A(\lambda)$ — and its roots are the complete list of eigenvalues. For each root, the eigenvectors are the non-zero solutions of $(A-\lambda I)\mathbf{x}=\mathbf{0}$: Unit 1's elimination, pointed at the nullspace of the collapsed matrix. That nullspace is the eigenspace $E_\lambda$, and the set of all eigenvalues is the matrix's spectrum.

For $2\times2$ matrices the polynomial wears its meaning openly:

— trace and determinant are literally its coefficients. (In general $p_A(\lambda)=(-1)^n\lambda^n+(-1)^{n-1}\operatorname{tr}(A)\lambda^{n-1}+\cdots+\det A$: the top coefficients can only come from the diagonal product $\prod(a_{ii}-\lambda)$, and setting $\lambda=0$ gives the constant term $p_A(0)=\det A$.)

Run the lecture's example by hand once — it's the template for every eigen-computation you'll ever do. For $A=\begin{bmatrix}1&6\\5&2\end{bmatrix}$:

Roots: $\lambda_1=7,\ \lambda_2=-4$. Then two small eliminations find the directions. For $\lambda_1=7$: $(A-7I)\mathbf{x}=\begin{bmatrix}-6&6\\5&-5\end{bmatrix}\mathbf{x}=\mathbf{0}$ forces $x_1=x_2$, so $E_{7}=\operatorname{span}\{(1,1)\}$. For $\lambda_2=-4$: $\begin{bmatrix}5&6\\5&6\end{bmatrix}\mathbf{x}=\mathbf{0}$ gives $5x_1+6x_2=0$, so $E_{-4}=\operatorname{span}\{(6,-5)\}$ — exactly the $\mathbf{u}$ we verified in Section 5. Notice both collapsed matrices visibly lost rank: their rows became parallel. That's no accident — it's the definition of being at an eigenvalue.

Think of wood grain. A plank splits cleanly along its grain and resists across it: a hidden set of perpendicular directions built into the material. The Spectral Theorem says every symmetric matrix ($A=A^{\top}$) has exactly that structure:

Read $A=Q\Lambda Q^{\top}$ right-to-left and a "complicated" matrix becomes three innocent moves: turn into the eigenbasis ($Q^{\top}$ — a pure rotation, lengths untouched), stretch along the now-straight axes ($\Lambda$ — just numbers on a diagonal), turn back ($Q$). Like straightening a crooked picture frame before enlarging it, then hanging it back at its angle. Every symmetric matrix is a rotation away from being diagonal.

The worked example, start to finish: $A=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ has $p_A(\lambda)=(2-\lambda)^2-1=(\lambda-1)(\lambda-3)$, so $\lambda_1=3,\lambda_2=1$. Eigenvectors $(1,1)$ and $(-1,1)$ — orthogonal, as promised — normalise to

and multiplying back: $Q\Lambda Q^{\top}=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ ✓. Watch the three moves happen to a vector you control:

Everything scales up. A symmetric $3\times3$ matrix has three real eigenvalues on three mutually perpendicular eigen-directions; feed it the unit sphere and out comes an ellipsoid whose axes lie exactly along that grain, stretched by $|\lambda_1|,|\lambda_2|,|\lambda_3|$. And the fingerprints multiply: the ellipsoid's volume is $|\det A|=|\lambda_1\lambda_2\lambda_3|$ times the sphere's — Section 2's volume dial, now factored into per-direction stretches. Drag a $\lambda$ to zero and watch dimension die: the ellipsoid pancakes flat, $\det\to0$, the matrix goes singular in front of you.

Section 5 left a mystery: the rotation matrix resonated nowhere. Picture a merry-go-round — as it spins, not a single horse keeps pointing the same way. No real direction survives, so there is no real eigenvector to find. Does the theory just fail?

It doesn't — it expands. Run the machinery honestly for $R_\theta=\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}$:

The roots exist — they've just stepped off the real line into the complex plane, landing on the unit circle as a conjugate pair. And they're not nonsense: $e^{\pm i\theta}$ is precisely the complex number that means "turn by $\theta$". A complex eigenvalue $re^{\pm i\theta}$ from any real matrix always arrives with its conjugate twin and encodes rotation-by-$\theta$-while-scaling-by-$r$ inside some invariant plane. The eigenvalues didn't fail to describe the motion — they described it perfectly, in a language built for turning.

Complex vectors force one honest repair. The old squared length $\mathbf{x}^{\top}\mathbf{x}$ breaks: for $\mathbf{x}=(1+i,\,2+i)$ it gives $(1+i)^2+(2+i)^2=3+6i$ — a "length" that isn't even real. The fix is to conjugate one side: define $\mathbf{x}^{H}=\overline{\mathbf{x}}^{\top}$ (the conjugate-transpose, or Hermitian transpose), and then

behaves like a length again. With $H$ replacing $\top$, the whole symmetric story generalises: a matrix is Hermitian if $A^{H}=A$ — e.g. $\begin{bmatrix}1&3-i\\3+i&4\end{bmatrix}$ — and Hermitian matrices have real eigenvalues and orthogonal eigenvectors, by the very proofs in Section 7 (which, you may notice, were written with $H$ all along). This is why quantum mechanics writes every observable — energy, momentum, spin — as a Hermitian operator: measured values must come out real, and Hermitian spectra are the mathematical guarantee that they do.

One more factorisation, for the friendliest matrices of all: symmetric and positive-definite (all eigenvalues $>0$ — Unit 3's engine-room matrices, wearing this unit's vocabulary). Just as $9=3\times3$ gives the number 9 a square root, an SPD matrix has a tidy "square root" of its own:

with $L$ lower-triangular and positive on its diagonal. And like ordinary square roots, it only works for "positive" inputs. The entries of $L$ fall out one at a time, top-left to bottom-right — for $2\times2$:

The lecture's example: $A=\begin{bmatrix}4&2\\2&3\end{bmatrix}$ gives $\ell_{11}=2$, $\ell_{21}=1$, $\ell_{22}=\sqrt{3-1}=\sqrt2$, and indeed $\begin{bmatrix}2&0\\1&\sqrt2\end{bmatrix}\begin{bmatrix}2&1\\0&\sqrt2\end{bmatrix}=\begin{bmatrix}4&2\\2&3\end{bmatrix}$ ✓. Notice every diagonal step takes a real square root — if the matrix isn't positive-definite, some radicand goes negative and the algorithm halts. Cholesky doubles as the cheapest positive-definiteness test in practice: don't check eigenvalues, just try to factor.

Diagonalisability. The general recipe $A=PDP^{-1}$ needs $n$ independent eigenvectors to fill $P$'s columns. When do you get them? Always, if the $n$ eigenvalues are distinct — eigenvectors of different eigenvalues are automatically independent (a chain of them collapsing would force two $\lambda$'s to coincide). Always, if $A$ is symmetric — that's the spectral theorem, with the bonus $P^{-1}=Q^{\top}$. But with repeated eigenvalues and no symmetry, it can fail: the shear $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ has $\lambda=1$ counted twice by the polynomial (algebraic multiplicity 2) yet only the single eigen-line $\operatorname{span}\{(1,0)\}$ (geometric multiplicity 1). One direction short of a basis: the shear is defective, and no change of basis will ever make it diagonal. You met this in the widget — the missing second resonance was this fact, felt through your fingertips.

The bridge into the next units. Take any data matrix $A\in\mathbb{R}^{m\times n}$ with independent columns (rank $n$) and form the Gram matrix $A^{\top}A$. It is symmetric — $(A^{\top}A)^{\top}=A^{\top}A$ — and positive-definite: $\mathbf{x}^{\top}A^{\top}A\mathbf{x}=\|A\mathbf{x}\|^{2}>0$ for $\mathbf{x}\ne\mathbf{0}$, because full column rank means $A\mathbf{x}\ne\mathbf{0}$. So everything in this unit applies to it: real positive eigenvalues, orthonormal eigenbasis, Cholesky factor. That one observation is the engine of least squares (Unit 9's normal equations solve $A^{\top}A\hat{\mathbf x}=A^{\top}\mathbf{b}$), of SVD (Unit 5 strings together the eigen-structure of $A^{\top}A$ and $AA^{\top}$), and of PCA (Units 12–13 eigendecompose the covariance matrix, which is a scaled Gram matrix of centred data). The chapter you just finished is the load-bearing wall.

① The determinant is the volume dial — $|\det|$ scales area/volume, the sign records a mirror flip, and $\det=0$ is the alarm bell of collapse: singular, rank-deficient, irreversible. Compute it by cofactors along the laziest line, or by elimination to a triangle. ② The trace adds the diagonal in a heartbeat and moonlights as the sum of the eigenvalues, while the determinant is their product — the free pair of sanity checks. ③ Eigenvectors are the directions a matrix cannot turn, $A\mathbf{x}=\lambda\mathbf{x}$; the characteristic equation $\det(A-\lambda I)=0$ hunts every $\lambda$, and elimination on the collapsed matrix recovers each eigenspace. ④ For symmetric matrices the fingerprints organise perfectly: real eigenvalues on perpendicular directions, $A=Q\Lambda Q^{\top}$ — every symmetric matrix is just turn, stretch, turn back. Rotations opt out into the complex plane ($e^{\pm i\theta}$), where Hermitian matrices carry the theorem onward. ⑤ Cholesky $A=LL^{\top}$ is the SPD square root — the working end of covariance matrices, Gaussian sampling, and the fastest positive-definiteness test there is.