Unit 4 · Determinants, Eigenvalues & the Spectral Theorem

1

What survives the machine?

Imagine this

You open a photo on your phone and pull it wider with two fingers. Every face in it changes shape. Every point moves.

Still, two questions have clean answers. How much bigger did the photo get? And was there a direction that only got longer, without turning? Yes: the direction you pulled in.

This whole unit is those two questions, asked of a matrix.

In Unit 1, a matrix was a machine that moves every point of space. It stretches, squashes, slants and turns. Real data gives you big matrices with thousands of numbers inside. We want a few simple facts that sum up the whole machine.

There are two kinds of such facts:

#Numbers that describe the whole motion at once. The determinant says how much areas grow. The trace is a quick pulse check.
↗Directions that the machine only stretches and never turns. These are the eigenvectors. Each one comes with its own stretch number, the eigenvalue $\lambda$ (read it: "lambda").

Here is the path, one step at a time:

①The determinant — how much a matrix grows areas, with an alarm bell at zero. (Sections 2–3)
②The trace — add up the diagonal. Cheap, and secretly useful. (Section 4)
③Eigenvectors — the directions a matrix cannot turn, and a recipe to find them. (Sections 5–6)
④The Spectral Theorem — every symmetric matrix is just turn, stretch, turn back. (Sections 7–8)
⑤Three extras — matrices that only turn, the "square root" of a matrix, and the fine print. (Sections 9–11)

Why ML cares PCA looks for the few directions in which your data spreads the most. Those directions are eigenvectors. By the end of this unit, the formula $A=Q\Lambda Q^{\top}$ will read like a plain sentence.

In one sentence: a matrix moves everything, but it has simple fingerprints: one number for how much it grows space, and special directions it only stretches.

2

The determinant — how much a matrix grows area

Imagine this

You roll a ball of dough into a roti. Before rolling, you mark a small square on the dough. After rolling, that square covers twice the area. So does every other square you marked. The whole roti grew by ×2.

Now flip the roti over on the tawa. Same size, but the other side faces up. And if you roll one way far too hard, the dough becomes a thin string. Its area is zero, and you can never roll it back.

Grow, flip, or squash flat. That is everything the determinant tells you.

A matrix does the same thing to the flat plane. Start with the unit square: the 1-by-1 tile with one corner at the origin. Feed every point of it through the matrix $A$ . It comes out as a slanted tile, a parallelogram.

Which parallelogram? Remember Unit 1: the columns of $A$ are where the two edges of the square land. So the new tile has the two columns of $A$ as its sides.

The determinant is the area of that new tile, with a sign attached. For a 2×2 matrix it is one short formula:

\det\begin{bmatrix}a&b\\c&d\end{bmatrix}=ad-bc

Try it on $A=\begin{bmatrix}2&1\\1&1.5\end{bmatrix}$ : $\det A=(2)(1.5)-(1)(1)=3-1=2$ . So this machine doubles every area. The square of area 1 becomes a tile of area 2. A shape of area 5 becomes area 10.

That one number tells you three things:

Look at	In everyday words	In symbols
Its size	How much every area grows. $\det=2$ doubles areas. $\det=\tfrac12$ halves them.	area factor $=\|\det A\|$
Its sign	A minus sign means the tile got flipped over, like the roti on the tawa. The size still grows by $\|\det A\|$ .	$\det A<0$
Zero	The alarm bell. The tile is squashed flat onto a line. Nothing can undo that, so $A$ has no inverse.	$\det A=0\iff\operatorname{rank}A<n$

Keep this in mind the "squashed flat" alarm is exactly the tool that will find eigenvalues in Section 6.

Try press ▶ to watch the tile grow, slide, flip and squash. Then drag the glowing tip of a₂ across a₁: the tile flips over the moment it passes flat.

✋ drag the glowing tips a₁ and a₂ — they are the columns of A · drag anywhere else to orbit

drag a glowing tip to reshape · drag elsewhere to orbit

A =2111.5

ad − bc = (2)(1.5) − (1)(1) = 2

det A = 2 area × 2

—

The dashed square is the tile before the machine. The floor grid bends too: every small square grows by the same factor. Orange means flipped over; red means squashed flat.

The realization

$\det A$ is the area of the tile that the unit square becomes. Its size says how much every area grows. A minus sign means the tile was flipped over. Zero means it was squashed flat, and then nothing can undo $A$ .

Pause & predict

A matrix has $\det A=-3$ . What does it do to a shape of area 5?

Pause & predict

In the widget, the ones button sets both columns to $(1,1)$ . What is $\det A$ , and what does it mean?

In one sentence: the determinant is the signed area of the tile the unit square becomes: its size is the growth, a minus means flipped, and zero means squashed flat for good.

3

Computing it — small pieces and the lazy row

Imagine this

You know how to find a 2×2 determinant: $ad-bc$ . Now someone hands you a 3×3 matrix. Don't panic.

It is like a big restaurant bill for three tables. You don't add it all in one go. You add up each table's small bill, then combine the three. A 3×3 determinant is three small 2×2 determinants, each with a weight and a + or − sign.

Here is the recipe, going along the first row:

1Pick one entry of the row, say $a_{11}$ (row 1, column 1).
2Cover up its row and its column. A small 2×2 matrix is left. Its determinant is called the minor $M_{11}$ .
3Give it a sign from the checkerboard below: + in the top-left corner, then − and + taking turns. The signed minor is the cofactor $C_{11}$ .
4Multiply the entry by its cofactor. Do the same for every entry of the row. Add the answers.

In symbols, going along row $i$ :

\det A=\sum_{j=1}^{n}a_{ij}\,C_{ij},\qquad C_{ij}=(-1)^{i+j}M_{ij}

One example, slowly. Take the matrix used in the widget below, $A=\begin{bmatrix}3&1&0\\-2&-4&3\\5&4&-2\end{bmatrix}$ . Go along row 1:

\begin{aligned}\det A&=3\cdot(+1)\big[(-4)(-2)-(3)(4)\big]+1\cdot(-1)\big[(-2)(-2)-(3)(5)\big]+0 \\ &=3\cdot(-4)+(-1)\cdot(-11)+0=-12+11=-1\end{aligned}

The surprise: you may go along any row or any column, and the answer is always the same. So be lazy. Pick the line with the most zeros. A zero entry makes its whole term zero, so you skip it. Above, the 0 in row 1 saved us one small determinant.

The cofactor checkerboard. The sign never depends on the entries — only on the position

(i,j)

.

Try row 1, then row 2. Same answer, but row 1 gets one term for free because of its 0. Then show the + − checkerboard.

A =

Tap a term. Its row and column go dim. What stays bright is its 2×2 minor.

det A = −1 every road gives −1

Two moves that make it faster

For big matrices there is a quicker way, using Unit 1's row moves. Two facts make it work:

⇄Swap two rows → the determinant changes sign. The tile gets flipped over, like a mirror image.
↗Add a multiple of one row to another → nothing changes at all. The tile only slides, like pushing a deck of cards sideways. Same base, same height, same area.

Since $\det A^{\top}=\det A$ , the same is true for columns, and columns are easier to draw. Here are both moves on a 3D box:

drag the picture to orbit

The columns

a_1=(2,1,0)

,

a_2=(1,1.5,0)

,

a_3=(0,0,1)

make a box of volume 2. Swap the first two: the same box, but mirror-imaged, so the signed volume is

-2

(it glows orange). Slide: add

a_1

to

a_2

. The box leans over, but its base and height stay the same, so the volume is still 2:

(2)(2.5)-(1)(3)=2

.

So you can run Unit 1's elimination. Turn $A$ into a triangle $U$ (all zeros below the diagonal). Then just multiply the diagonal, and flip the sign once for every row swap you made:

\det A=(-1)^{s}\,u_{11}u_{22}\cdots u_{nn}\qquad(s=\text{number of row swaps})

Quick example: $\begin{bmatrix}2&1\\4&5\end{bmatrix}$ . Take 2 × row 1 away from row 2 to get $\begin{bmatrix}2&1\\0&3\end{bmatrix}$ . No swaps, so $\det=2\times3=6$ . Check with the formula: $10-4=6$ ✓.

Handy rule	In everyday words
$\det(AB)=\det A\cdot\det B$	Two machines in a row: their growth factors multiply.
$\det A^{\top}=\det A$	Flipping the matrix over its diagonal keeps the area.
$\det A^{-1}=1/\det A$	The undo machine shrinks things back by the same factor.
triangular $A$	The determinant is the product of the diagonal.

If you want the algebra · why a triangle's determinant is its diagonal, and why sliding changes nothing

Prove it · why a triangle's determinant is its diagonal product — derived

Claim. If $U$ is upper triangular (all zeros below the diagonal), then $\det U=u_{11}u_{22}\cdots u_{nn}$ .

1

Expand along the first column. Below

u_{11}

that column is all zeros, so only one term survives:

\det U=\sum_{i=1}^{n}u_{i1}C_{i1}=u_{11}C_{11}+0+\cdots+0=u_{11}\,(-1)^{1+1}M_{11}=u_{11}M_{11}

Every other term has a factor

u_{i1}=0

(

i\ge2

). The sign of

C_{11}

is

(-1)^{2}=+1

.

2

Look at what

M_{11}

is: delete row 1 and column 1 of

U

. What remains is an

(n-1)\times(n-1)

matrix that is still upper triangular, with diagonal

u_{22},\dots,u_{nn}

. Deleting the first row and column keeps every remaining entry in its relative position — zeros below the diagonal stay below the diagonal.

3

Apply step 1 to that smaller triangle, then again, and again:

\det U=u_{11}\,M_{11}=u_{11}\,u_{22}\,M'_{11}=\cdots=u_{11}u_{22}\cdots u_{n-1,n-1}\det\begin{bmatrix}u_{nn}\end{bmatrix}

Each round peels one diagonal entry off the front and leaves a triangle one size smaller.

4

The

1\times1

determinant is the entry itself,

\det[u_{nn}]=u_{nn}

. Hence

\det U=u_{11}u_{22}\cdots u_{nn}

. Done. ∎ For a lower triangle run the same argument along the first row (or use

\det L=\det L^{\top}

). Diagonal matrices are both, so their determinant is the diagonal product too.

Why does adding a row to another leave det unchanged? Expand the new determinant. It splits into the old $\det A$ plus $c$ times a determinant with two equal rows. Swap those two equal rows: the determinant must change sign, yet the matrix is the same. A number equal to its own negative is 0. So the extra piece is 0, and $\det$ survives elimination.

Full rank ⟺ det ≠ 0. Eliminate $A$ down to $U$ . Full rank means every column has a pivot, so every $u_{ii}\ne0$ , so $\det A=\pm\prod u_{ii}\ne0$ . And the other way round: $\det A\ne0$ forces every $u_{ii}\ne0$ , so all pivots are there and only $\mathbf{x}=\mathbf{0}$ solves $A\mathbf{x}=\mathbf{0}$ . Hold this thought: in Section 6, eigenvalues appear exactly where $A-\lambda I$ loses full rank, and $\det(A-\lambda I)=0$ is how we catch that.

The realization

A big determinant is a sum of small ones: entry × sign × minor, along any row or column. Pick the line with the most zeros. For big matrices, eliminate down to a triangle and multiply the diagonal. Every road gives the same number.

Pause & predict

The widget's matrix has one zero, at $a_{13}$ (row 1, column 3). Which roads take the least work?

In one sentence: break a big determinant into small signed pieces along the laziest line, or eliminate to a triangle and multiply the diagonal.

4

The trace — a quick pulse check

Imagine this

At a clinic, the nurse checks your pulse before anything else. It takes ten seconds and costs nothing. It does not tell the whole story. But it tells a lot, and it catches mistakes early.

The trace is the pulse check of a matrix. Just add the numbers on the diagonal, from top-left to bottom-right:

\operatorname{tr}(A)=a_{11}+a_{22}+\cdots+a_{nn}

Example: $\operatorname{tr}\begin{bmatrix}1&6\\5&2\end{bmatrix}=1+2=3$ . The 6 and the 5 are simply ignored.

The trace follows easy rules:

Rule	In everyday words
$\operatorname{tr}(A+B)=\operatorname{tr}A+\operatorname{tr}B$	Add the matrices first or add the traces later: same answer.
$\operatorname{tr}(\alpha A)=\alpha\operatorname{tr}A$	Scale the matrix, and the trace scales too.
$\operatorname{tr}(I_n)=n$	The identity has $n$ ones on its diagonal.
$\operatorname{tr}(AB)=\operatorname{tr}(BA)$	The order of a product does not matter to the trace. This even works when $AB$ and $BA$ are different sizes.

The last rule (the cyclic property) is the surprising one. Try it with a row $A=\begin{bmatrix}1&2\end{bmatrix}$ and a column $B=\begin{bmatrix}3\\4\end{bmatrix}$ :

AB=\begin{bmatrix}1\cdot3+2\cdot4\end{bmatrix}=\begin{bmatrix}11\end{bmatrix},\qquad BA=\begin{bmatrix}3&6\\4&8\end{bmatrix},\ \operatorname{tr}(BA)=3+8=11

A 1×1 matrix and a 2×2 matrix, with the same trace. Many machine-learning formulas are simplified with exactly this shuffle.

Why give a whole section to such a cheap number? Because of a fact we will meet in Section 6: the trace is the sum of the eigenvalues, and the determinant is their product. Two numbers you can read in seconds already know something about eigenvalues you have not found yet.

If you want the algebra · why tr(AB) = tr(BA), even for different sizes

Prove it · why tr(AB) = tr(BA) — derived

Claim. For $A$ of size $m\times n$ and $B$ of size $n\times m$ (so both products exist), $\operatorname{tr}(AB)=\operatorname{tr}(BA)$ — even though $AB$ is $m\times m$ and $BA$ is $n\times n$ .

1

Write one diagonal entry of

AB

with the row-times-column rule:

(AB)_{ii}=\sum_{k=1}^{n}a_{ik}\,b_{ki}

Row

i

of

A

dotted with column

i

of

B

. The index

k

runs over the inner dimension

n

.

2

Add the diagonal entries to get the trace — a double sum:

\operatorname{tr}(AB)=\sum_{i=1}^{m}\sum_{k=1}^{n}a_{ik}\,b_{ki}

Just the definition of trace applied to step 1. Every term is one product

a_{ik}b_{ki}

.

3

Do the same for

BA

, whose diagonal entries are

(BA)_{kk}=\sum_{i=1}^{m}b_{ki}a_{ik}

:

\operatorname{tr}(BA)=\sum_{k=1}^{n}\sum_{i=1}^{m}b_{ki}\,a_{ik}

Row

k

of

B

dotted with column

k

of

A

; now the inner index is

i

, running to

m

.

4

Compare: both are the sum of the same

mn

numbers

a_{ik}b_{ki}

, listed in a different order. Finite sums don't care about order, so

\operatorname{tr}(AB)=\operatorname{tr}(BA)

. Done. ∎

b_{ki}a_{ik}=a_{ik}b_{ki}

because scalars commute; swapping the two

\sum

signs is just re-grouping the same list of products. Nothing about the sizes

m,n

ever mattered.

The realization

$\operatorname{tr}A$ is the sum of the diagonal. It ignores everything off the diagonal, and $\operatorname{tr}(AB)=\operatorname{tr}(BA)$ always. Soon you will see it is also the sum of the eigenvalues.

Pause & predict

$A$ is $2\times 5$ and $B$ is $5\times 2$ . So $AB$ is $2\times2$ , but $BA$ is $5\times5$ . What about their traces?

In one sentence: the trace adds up the diagonal, ignores the order of a product, and turns out to be the sum of the eigenvalues.

5

Eigenvectors — the directions a matrix cannot turn

Imagine this

Before rolling a roti, draw a few arrows on the dough with a knife tip. Now roll only forwards and backwards.

Look at the arrows afterwards. Most of them now point a slightly different way: they got tilted. But the arrow drawn along the rolling direction still points the same way. It just got longer. And the arrow drawn straight across also kept its direction.

Those two special arrows are what this section is about.

A matrix knocks almost every arrow off its line. But a few special arrows only get stretched, shrunk or flipped along their own line. These are the eigenvectors ("eigen" is German for "own": the matrix's own directions). The stretch number is the eigenvalue $\lambda$ .

A\mathbf{x}=\lambda\mathbf{x}\qquad(\mathbf{x}\ne\mathbf{0})

Read it: "A times x is just a number times x." On this one arrow, the whole matrix acts like plain multiplication by $\lambda$ . What the number tells you:

>1 $\lambda>1$ : the arrow gets longer, like the rolling direction.
<1 $0<\lambda<1$ : the arrow gets shorter.
1 $\lambda=1$ : the arrow is left exactly as it was, like the axis of a spinning globe.
0 $\lambda=0$ : the arrow is squashed to nothing.
− $\lambda<0$ : the arrow flips to point backwards, then stretches by $|\lambda|$ .

Two small rules. The zero arrow does not count, because $A\mathbf{0}=\lambda\mathbf{0}$ is true for every $\lambda$ and tells us nothing. And if $\mathbf{x}$ works, so does $2\mathbf{x}$ or $-5\mathbf{x}$ . So an eigenvector is really a whole direction: a line that the matrix keeps.

Checking a guess takes one multiplication. Take $A=\begin{bmatrix}1&6\\5&2\end{bmatrix}$ and the arrow $\mathbf{u}=(6,-5)$ :

A\mathbf{u}=\begin{bmatrix}1(6)+6(-5)\\5(6)+2(-5)\end{bmatrix}=\begin{bmatrix}-24\\20\end{bmatrix}=-4\begin{bmatrix}6\\-5\end{bmatrix}

The answer is $-4$ times $\mathbf{u}$ . So $\mathbf{u}$ is an eigenvector with $\lambda=-4$ : flipped backwards and made 4 times longer, but still on its own line. Now try $\mathbf{v}=(3,-2)$ : $A\mathbf{v}=(-9,11)$ . That is not a multiple of $(3,-2)$ , so $\mathbf{v}$ is not an eigenvector. Go hunting yourself:

Try sweep x slowly around the circle and find both hidden directions of the first matrix. Then open rotation: nothing lines up anywhere.

✋ drag on the floor — the gold arrow follows your pointer · press ↻ to spin the view instead

drag on the floor to move the arrow · ↻ spins the view

alignment = — found 0/2

—

Gold is your arrow $\mathbf{x}$ . Blue is where $A$ sends it. The wall of light is tallest where $A\mathbf{x}$ lines up with $\mathbf{x}$ , and every line you find stays drawn.

Notice this

On the symmetric matrix, the two directions you find are at a right angle. They also sit exactly on the long and short axes of the blue oval. That is no accident. Section 7 turns it into a theorem.

On the rotation, nothing ever lines up. A merry-go-round leaves no direction alone. Section 9 solves that mystery.

The realization

An eigenvector is an arrow that $A$ only stretches: $A\mathbf{x}=\lambda\mathbf{x}$ . The stretch number $\lambda$ is its eigenvalue. To test a guess, multiply once and check if the answer is a multiple of the arrow.

Pause & predict

For $A=\begin{bmatrix}1&6\\5&2\end{bmatrix}$ , one multiplication gives $A\begin{bmatrix}1\\1\end{bmatrix}=\begin{bmatrix}7\\7\end{bmatrix}$ . What did we just learn?

Pause & predict

In the widget, the shear $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ lines up along just one line, the x-axis. How many independent eigen-directions does it have?

In one sentence: an eigenvector is a direction the matrix only stretches, and its eigenvalue says by how much: $A\mathbf{x}=\lambda\mathbf{x}$ .

6

Hunting eigenvalues — the characteristic equation

Imagine this

An old radio has a tuning dial. Turn it slowly: hiss, hiss, hiss… then, at one exact spot, a clear station. Turn a little more: hiss again. Then another station.

Finding eigenvalues works the same way. The dial is the number $\lambda$ . At most settings nothing happens. At a few exact settings, something "clicks". Those settings are the eigenvalues.

What exactly clicks? Move $\lambda\mathbf{x}$ to the left side of $A\mathbf{x}=\lambda\mathbf{x}$ :

A\mathbf{x}-\lambda\mathbf{x}=\mathbf{0}\quad\Longrightarrow\quad(A-\lambda I)\,\mathbf{x}=\mathbf{0}

Here $A-\lambda I$ just means: take $\lambda$ away from every number on the diagonal of $A$ . This new matrix sends a non-zero arrow $\mathbf{x}$ to zero. So it squashes space flat. And from Section 2, "squashed flat" means the determinant is zero:

\boxed{\;\det(A-\lambda I)=0\;}

This is the characteristic equation, our station finder. Its solutions are exactly the eigenvalues. Its left side is a polynomial in $\lambda$ , called the characteristic polynomial $p_A(\lambda)$ .

For a 2×2 matrix it is a quadratic, and it looks very familiar:

p_A(\lambda)=\lambda^2-\operatorname{tr}(A)\,\lambda+\det(A)

The trace and the determinant are its two coefficients! Now the full example, start to finish. For $A=\begin{bmatrix}1&6\\5&2\end{bmatrix}$ : the trace is $1+2=3$ and the determinant is $2-30=-28$ . So

p_A(\lambda)=\lambda^{2}-3\lambda-28=(\lambda-7)(\lambda+4)

The stations are $\lambda_1=7$ and $\lambda_2=-4$ . Now find each direction by solving $(A-\lambda I)\mathbf{x}=\mathbf{0}$ , just like Unit 1:

7 $A-7I=\begin{bmatrix}-6&6\\5&-5\end{bmatrix}$ . Both rows say $x_1=x_2$ . So the direction is $(1,1)$ .
−4 $A+4I=\begin{bmatrix}5&6\\5&6\end{bmatrix}$ . Both rows say $5x_1+6x_2=0$ . So the direction is $(6,-5)$ , the $\mathbf{u}$ from Section 5.

Notice that in both cases the two rows point along the same line. That is what "squashed flat" looks like in the numbers.

The free check

The eigenvalues always add up to the trace and multiply to the determinant. Here: $7+(-4)=3=\operatorname{tr}A$ ✓ and $7\times(-4)=-28=\det A$ ✓. Spend five seconds on this every time. It catches mistakes at once.

It also explains the alarm bell one level deeper: $\det A=0$ exactly when some eigenvalue is 0, that is, when some direction gets squashed to nothing.

Try press ▶ and watch the small tile of $A-\lambda I$ go flat exactly as the dial passes $\lambda=7$ and $\lambda=-4$ .

dial λ2

A =

tr = 3 det = −28 Σλ ✓ · Πλ ✓

—

The chips run the free check live: the roots must add to the trace and multiply to the determinant. If the curve never touches zero, the eigenvalues are complex (Section 9).

What "squashed flat" looks like. At an eigenvalue, both rows of

A-\lambda I

lie on one line. So

(A-\lambda I)\mathbf{x}=\mathbf{0}

is really just one condition: "

\mathbf{x}

is at a right angle to that row". The answers form a line (dashed), and the green eigenvector sits on it.

The recipe card

1Write $A-\lambda I$ : take $\lambda$ off the diagonal, touch nothing else.
2Solve $\det(A-\lambda I)=0$ . The roots are all the eigenvalues.
3For each root $\lambda_i$ , solve $(A-\lambda_i I)\mathbf{x}=\mathbf{0}$ by elimination. The answers form the eigenspace $E_{\lambda_i}$ , the eigen-directions of $\lambda_i$ .
4Free check: the roots add up to $\operatorname{tr}A$ and multiply to $\det A$ .

A shortcut worth stealing

A triangular matrix shows its eigenvalues on its diagonal. Why? $A-\lambda I$ is triangular too, and a triangle's determinant is its diagonal product: $p_A(\lambda)=(a_{11}-\lambda)(a_{22}-\lambda)\cdots(a_{nn}-\lambda)$ . So the roots are just $a_{11},a_{22},\dots$ . Two more freebies: $A$ and $A^{\top}$ have the same eigenvalues, and every eigenvalue of $I$ is 1.

If you want the algebra · where λ² − tr(A)λ + det A comes from, and why the eigenvalues add to the trace

Prove it · why the 2×2 characteristic polynomial is λ² − tr(A)λ + det A — derived

Claim. For $A=\begin{bmatrix}a&b\\c&d\end{bmatrix}$ , $\ \det(A-\lambda I)=\lambda^2-(a+d)\lambda+(ad-bc)$ .

1

Subtract

\lambda

down the diagonal and apply the

2\times2

formula

ad-bc

:

\det\begin{bmatrix}a-\lambda&b\\c&d-\lambda\end{bmatrix}=(a-\lambda)(d-\lambda)-bc

The off-diagonal entries

b,c

are untouched by the shift — only the diagonal moves.

2

Multiply out the product:

(a-\lambda)(d-\lambda)=ad-a\lambda-d\lambda+\lambda^2=\lambda^2-(a+d)\lambda+ad

Four terms from FOIL; the two

\lambda

terms collect into

-(a+d)\lambda

.

3

Put the

-bc

back and name the coefficients:

\lambda^2-(a+d)\lambda+(ad-bc)=\lambda^2-\operatorname{tr}(A)\,\lambda+\det A

. Done. ∎

a+d

is the trace by definition and

ad-bc

is the determinant by definition. Check on the lecture's matrix:

\lambda^2-3\lambda-28

.

Prove it · why the eigenvalues sum to the trace and multiply to the determinant — derived

Claim. If $\lambda_1,\lambda_2$ are the roots of $p_A$ , then $\lambda_1+\lambda_2=\operatorname{tr}A$ and $\lambda_1\lambda_2=\det A$ .

1

A degree-2 polynomial with roots

\lambda_1,\lambda_2

and leading coefficient 1 factors as

p_A(\lambda)=(\lambda-\lambda_1)(\lambda-\lambda_2)

Plugging in

\lambda=\lambda_1

or

\lambda_2

gives 0, and the

\lambda^2

coefficient is 1 — matching the box above. (If the roots are complex, this still holds over

\mathbb{C}

.)

2

Expand the factored form:

(\lambda-\lambda_1)(\lambda-\lambda_2)=\lambda^2-(\lambda_1+\lambda_2)\lambda+\lambda_1\lambda_2

Same FOIL as before, with the roots in place of

a,d

.

3

Two ways of writing the same polynomial must have the same coefficients. Compare with

\lambda^2-\operatorname{tr}(A)\lambda+\det A

:

\lambda_1+\lambda_2=\operatorname{tr}A

and

\lambda_1\lambda_2=\det A

. Done. ∎ For

n\times n

:

p_A(\lambda)=\prod_i(\lambda_i-\lambda)

; the

\lambda^{n-1}

coefficient of the product is

(-1)^{n-1}\sum_i\lambda_i

, which must equal

(-1)^{n-1}\operatorname{tr}A

; and setting

\lambda=0

gives

\prod_i\lambda_i=p_A(0)=\det A

. Same idea, more terms.

The realization

$\det(A-\lambda I)=0$

An eigenvalue is a setting of the dial $\lambda$ where $A-\lambda I$ squashes space flat. Solve this equation for the eigenvalues, then solve $(A-\lambda I)\mathbf{x}=\mathbf{0}$ for the directions. Check: sum = trace, product = determinant.

Pause & predict

No pencil allowed: what are the eigenvalues of $\begin{bmatrix}5&7\\0&3\end{bmatrix}$ ?

Pause & predict

A classmate works out the eigenvalues of $\begin{bmatrix}1&6\\5&2\end{bmatrix}$ (trace 3, det −28) and says $\lambda=6$ and $-3$ . Your ten-second verdict?

In one sentence: eigenvalues are the values of $\lambda$ where $\det(A-\lambda I)=0$ ; they always add up to the trace and multiply to the determinant.

7

The Spectral Theorem — symmetric matrices come apart cleanly

Imagine this

Tear a sheet of newspaper from top to bottom. It rips in a clean, straight line. Now tear it from side to side. The rip wanders all over.

The paper has a hidden grain, set when it was made in the factory. The grain has two directions at a right angle: an easy one and a hard one. You cannot see them, but they control everything.

Symmetric matrices have a hidden grain too.

A matrix is symmetric if it looks the same after flipping it over its diagonal: $A=A^{\top}$ . For example $\begin{bmatrix}2&1\\1&2\end{bmatrix}$ . Why care? Almost every matrix machine learning builds from data is symmetric: covariance matrices, $A^{\top}A$ , kernel matrices. So this is not a special case. It is the case you will meet most.

The Spectral Theorem makes three promises about any symmetric matrix:

The promise	In everyday words
① Real eigenvalues	Every stretch number is an ordinary real number. No complex numbers, ever.
② Right angles	Eigen-directions with different eigenvalues are at a right angle to each other: the grain.
③ Enough of them	There are always enough eigen-directions to make a full square grid of the space.

Put the grain directions (each of length 1) as the columns of a matrix $Q$ , and the stretch numbers on the diagonal of $\Lambda$ (read it: "capital lambda"). Then

A=Q\,\Lambda\,Q^{\top}

Read it from right to left, as three easy moves:

1 $Q^{\top}$ : turn the space so the grain lines up with the axes. Like Unit 3's stiff turns, it changes no length.
2 $\Lambda$ : stretch along each axis by its own $\lambda$ . Just numbers on a diagonal.
3 $Q$ : turn back.

So every symmetric matrix is just turn, stretch, turn back. And turning back is easy, because for such a $Q$ the inverse is simply the flip: $Q^{-1}=Q^{\top}$ .

The worked example: $A=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ . Trace 4, determinant 3, so $p_A(\lambda)=\lambda^2-4\lambda+3=(\lambda-3)(\lambda-1)$ . The eigenvalues are 3 and 1. Their directions are $(1,1)$ and $(-1,1)$ : at a right angle, as promised. Shrink each to length 1:

\mathbf{q}_1=\tfrac{1}{\sqrt2}\begin{bmatrix}1\\1\end{bmatrix},\quad \mathbf{q}_2=\tfrac{1}{\sqrt2}\begin{bmatrix}-1\\1\end{bmatrix},\qquad Q=\tfrac{1}{\sqrt2}\begin{bmatrix}1&-1\\1&1\end{bmatrix},\quad \Lambda=\begin{bmatrix}3&0\\0&1\end{bmatrix}

Multiply back and you get $Q\Lambda Q^{\top}=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ ✓. Watch the three moves happen:

Try press ▶ and follow the bright bead: it turns, slides out along the grain, and turns back. Then drag x onto the q₁ line and play again: x just ends up 3 times longer.

✋ drag the gold tip to choose x · drag anywhere else to orbit

drag the gold tip to choose x · drag elsewhere to orbit

QΛQᵀx = Ax ✓ — three simple moves, same answer

The ring of beads is every direction at once. The two turns keep it a circle. Only the stretch makes it an oval: 3 times longer along $\mathbf{q}_1$ , unchanged along $\mathbf{q}_2$ .

Careful: a second ellipse. This one is the set of points where

\mathbf{x}^{\top}A\mathbf{x}=1

. Along

\mathbf{q}_1

the value grows fastest (

\lambda_1=3

), so it reaches 1 sooner: the axis is only

1/\sqrt{3}

long. Big eigenvalue, short axis. That is the opposite of the widget, where

\mathbf{q}_1

gets stretched by 3.

If you want the algebra · why the eigenvalues are real, and why the directions meet at right angles

Prove it · why a symmetric matrix has real eigenvalues — derived

Claim. If $A$ is real and $A^{\top}=A$ , every eigenvalue of $A$ is a real number.

1

Allow

\lambda

and

\mathbf{x}\ne\mathbf{0}

to be complex for now, with

A\mathbf{x}=\lambda\mathbf{x}

. Multiply on the left by the conjugate-transpose

\mathbf{x}^{H}=\overline{\mathbf{x}}^{\top}

:

\mathbf{x}^{H}A\mathbf{x}=\lambda\,\mathbf{x}^{H}\mathbf{x}

Section 9 introduces

\mathbf{x}^{H}

properly; all we use is that it's a row vector, so both sides are single numbers.

2

The right-hand factor

\mathbf{x}^{H}\mathbf{x}

is a positive real number:

\mathbf{x}^{H}\mathbf{x}=\sum_i\overline{x_i}\,x_i=\sum_i|x_i|^2>0

Each

\overline{x_i}x_i=|x_i|^2\ge0

, and at least one

x_i\ne0

because

\mathbf{x}\ne\mathbf{0}

.

3

The left-hand side

s=\mathbf{x}^{H}A\mathbf{x}

equals its own conjugate. Take the conjugate-transpose of the

1\times1

"matrix"

s

:

\overline{s}=s^{H}=(\mathbf{x}^{H}A\mathbf{x})^{H}=\mathbf{x}^{H}A^{H}(\mathbf{x}^{H})^{H}=\mathbf{x}^{H}A\mathbf{x}=s

Reverse the order and conjugate-transpose each factor (the rule

(XYZ)^{H}=Z^{H}Y^{H}X^{H}

). Then

A^{H}=\overline{A}^{\top}=A^{\top}=A

because

A

is real and symmetric. A number equal to its own conjugate is real.

4

Divide step 1 by the positive real number from step 2:

\lambda=\dfrac{\mathbf{x}^{H}A\mathbf{x}}{\mathbf{x}^{H}\mathbf{x}}=\dfrac{\text{real}}{\text{positive real}}

, so

\lambda

is real. ∎ The same argument with

A^{H}=A

(Hermitian) instead of

A^{\top}=A

is what Section 9 uses.

Prove it · why eigenvectors of different eigenvalues are perpendicular — derived

Claim. If $A^{\top}=A$ , $A\mathbf{x}=\lambda\mathbf{x}$ , $A\mathbf{y}=\mu\mathbf{y}$ and $\lambda\ne\mu$ , then $\mathbf{y}^{\top}\mathbf{x}=0$ .

1

Start from the number

\mathbf{y}^{\top}A\mathbf{x}

and evaluate it using

\mathbf{x}

's eigen-equation:

\mathbf{y}^{\top}A\mathbf{x}=\mathbf{y}^{\top}(\lambda\mathbf{x})=\lambda\,\mathbf{y}^{\top}\mathbf{x}

Replace

A\mathbf{x}

by

\lambda\mathbf{x}

, then pull the scalar out front.

2

Evaluate the same number using

\mathbf{y}

's eigen-equation instead. First move

A

onto

\mathbf{y}

:

\mathbf{y}^{\top}A\mathbf{x}=(A^{\top}\mathbf{y})^{\top}\mathbf{x}=(A\mathbf{y})^{\top}\mathbf{x}=(\mu\mathbf{y})^{\top}\mathbf{x}=\mu\,\mathbf{y}^{\top}\mathbf{x}

\mathbf{y}^{\top}A=(A^{\top}\mathbf{y})^{\top}

is the transpose rule;

A^{\top}=A

is the only place symmetry is used.

3

Subtract the two expressions for the same number:

\lambda\,\mathbf{y}^{\top}\mathbf{x}-\mu\,\mathbf{y}^{\top}\mathbf{x}=(\lambda-\mu)\,\mathbf{y}^{\top}\mathbf{x}=0

Steps 1 and 2 both equal

\mathbf{y}^{\top}A\mathbf{x}

, so their difference is zero.

4

Since

\lambda-\mu\ne0

, divide by it:

\mathbf{y}^{\top}\mathbf{x}=0

— perpendicular. ∎ A product of two numbers is zero only if one of them is; the first factor isn't. With a repeated eigenvalue the eigenspace has extra dimensions, and Gram–Schmidt (Unit 3) builds an orthonormal basis inside it. The one fact we take on faith is that for symmetric matrices those dimensions always add up to

n

— no direction goes missing.

Where it powers ML

PCA: the main directions of a data set are the eigenvectors of its covariance matrix. Each eigenvalue says how much the data spreads along its direction. Keep the top few, and you compress the data while losing the least (Units 12–13). SVD: apply this theorem to $A^{\top}A$ and $AA^{\top}$ and you can break down any matrix (Unit 5). Shapes: $\mathbf{x}^{\top}A\mathbf{x}=c$ draws an ellipse or ellipsoid whose axes are the eigenvectors, like the figure above.

The realization

$A=Q\Lambda Q^{\top}$

A symmetric matrix is: turn into the grain, stretch each grain direction by its $\lambda$ , turn back. Its eigenvalues are real, its eigen-directions are at right angles, and undoing $Q$ is just flipping it.

Pause & predict

In $A=Q\Lambda Q^{\top}$ we used $Q^{\top}$ , where normally we would need the inverse $Q^{-1}$ . Why is that allowed here?

In one sentence: every symmetric matrix has a hidden right-angle grain, so it is just turn, stretch along the grain, turn back: $A=Q\Lambda Q^{\top}$ .

8

The theorem in 3D — a ball becomes an egg

Imagine this

Squeeze a round ball of clay with your hands. It becomes an egg shape, like a rugby ball: long one way, medium another way, short the third way.

Those three ways are at right angles to each other. They are the grain of your squeeze.

Everything from Section 7 works in 3D. A symmetric 3×3 matrix has three real eigenvalues and three eigen-directions, all at right angles. Feed it the round ball of all length-1 arrows. Out comes an ellipsoid (an egg shape). Its three axes lie along the three grain directions, with lengths $|\lambda_1|$ , $|\lambda_2|$ , $|\lambda_3|$ .

The volume grows by the product of the three stretches, which is the determinant again: $|\det A|=|\lambda_1\lambda_2\lambda_3|$ . For example, stretches 3, 1.5 and 0.6 give $\det A=3\times1.5\times0.6=2.7$ . And if one $\lambda$ is 0, the egg is pressed flat into a disc: $\det A=0$ , Section 2's alarm.

Why ML cares a covariance matrix is exactly such a symmetric $A$ . Its ellipsoid is the shape of your data cloud, and its axes are the principal components.

Try slide λ₃ to 0: the egg flattens into a disc. Watch the nine numbers of $A$ all change while the three tilted axes never move.

✋ drag to orbit the scene

drag the picture to orbit

λ₁3

λ₂1.5

λ₃0.6

A = QΛQᵀ =

det = λ₁λ₂λ₃ = 2.7 volume × 2.7

—

The axes are tilted on purpose: a symmetric matrix carries its own private right-angle axes. Its nine numbers look messy, but the three $\lambda$ 's are the simple truth underneath.

The realization

Pause & predict

In the widget, set $\lambda_1=3,\ \lambda_2=1.5,\ \lambda_3=-2$ . What happens to the ball's volume and to its handedness?

In one sentence: in 3D a symmetric matrix squeezes the ball into an egg along three right-angle axes, and the volume grows by the product of the three eigenvalues.

9

When nothing lines up — turns and complex eigenvalues

Imagine this

Look up at a ceiling fan. Every blade keeps turning. No blade stays pointing the same way.

A matrix that only turns is like that. It moves every arrow off its line. So in Section 5's widget, the rotation had no eigen-directions at all. Does the eigen-story just break here?

Let's run the recipe anyway, on a turn by the angle $\theta$ (read it: "theta"):

R_\theta=\begin{bmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{bmatrix},\qquad p(\lambda)=\lambda^{2}-2\cos\theta\,\lambda+1

(The trace is $2\cos\theta$ , and the determinant is $\cos^2\theta+\sin^2\theta=1$ .) Take a quarter turn, $\theta=90^\circ$ . Then $p(\lambda)=\lambda^2+1$ , and we need $\lambda^2=-1$ . No ordinary number squares to $-1$ .

Here is a friendly way to see it. Think of "multiply by $i$ " as "turn by 90°". Do it twice: you have turned by 180°, which just flips an arrow. Flipping is the same as multiplying by $-1$ . So $i\times i=-1$ . The number $i$ is a quarter turn.

So the eigenvalues of a quarter turn are $\lambda=\pm i$ . They are not nonsense. They are saying exactly "I turn things by 90°". For any angle, the recipe gives

\lambda=\cos\theta\pm i\sin\theta

These are complex numbers: an ordinary part plus an $i$ part. They come as a pair of twins with opposite $i$ parts. Draw them on a plane, with the ordinary part going across and the $i$ part going up. Both twins sit on the circle of radius 1, at the angles $\theta$ and $-\theta$ . The angle of the eigenvalue is the turn.

Try press ▶. The gold arrow on the turntable turns by θ. The eigenvalue λ₁ on the standing circle climbs by the same angle.

drag the picture to orbit

θ35°

p(λ) = λ² − 1.64λ + 1 λ = 0.82 ± 0.57i

—

At 0° and 180° the twins meet on the flat line, at 1 or at −1. Those are the only turns with real eigen-directions.

Turn and grow at once. An eigenvalue off the circle adds a stretch to the turn. Practice Problem 2 has $\lambda=1\pm2i$ . Its size is $|\lambda|=\sqrt{1^2+2^2}=\sqrt5\approx2.24$ , and its angle is about $63.4^\circ$ . So each step turns by $63.4^\circ$ and grows by $\sqrt5$ . Repeat it, and an arrow traces a spiral:

The spiral of

\lambda=1\pm2i

.

C=\begin{bmatrix}1&2\\-2&1\end{bmatrix}

has the same eigenvalues as Problem 2's matrix (it is the same machine seen from a different angle). The lengths go

1,\ \sqrt5,\ 5,\ 5\sqrt5

: the size of

\lambda

is the growth, and the angle of

\lambda

is the turn.

One repair for complex arrows. For complex arrows, the old length formula breaks. With $\mathbf{x}=(1+i,\,2+i)$ , $\mathbf{x}^{\top}\mathbf{x}=(1+i)^2+(2+i)^2=3+6i$ : a "length squared" that is not even a real number. The fix: flip the sign of every $i$ in one copy first. This is the conjugate transpose $\mathbf{x}^{H}$ . Then $\mathbf{x}^{H}\mathbf{x}=|1+i|^2+|2+i|^2=2+5=7$ , a proper length squared.

With $H$ in place of $\top$ , the symmetric story carries over. A matrix with $A^{H}=A$ is called Hermitian, for example $\begin{bmatrix}1&3-i\\3+i&4\end{bmatrix}$ . Hermitian matrices have real eigenvalues and right-angle eigen-directions, by the same proofs as Section 7. That is why quantum physics uses them: a measured energy must come out as a real number.

The realization

A pure turn has no real eigen-directions. Its eigenvalues are the twins $\cos\theta\pm i\sin\theta$ , which sit on the unit circle. The angle of $\lambda$ is the turn, and the size of $\lambda$ is the stretch.

Pause & predict

Set the widget to θ = 90°, the quarter turn $\begin{bmatrix}0&-1\\1&0\end{bmatrix}$ . What are its eigenvalues?

In one sentence: a matrix that turns has no real eigen-directions, and its complex eigenvalues $\cos\theta\pm i\sin\theta$ are the turn written as a number.

10

Cholesky — the square root of a matrix

Imagine this

A cricket video game needs 1000 made-up players. Their heights and weights must look real: taller players are usually heavier.

The game starts with plain random numbers, where height and weight have nothing to do with each other. Plotted, they make a round cloud of dots. Then it bends that cloud with one small matrix $L$ . Out comes a tilted oval cloud, exactly like real players.

That bending matrix $L$ is the "square root" of a matrix.

Every positive number has a square root: $9=3\times3$ . A symmetric positive-definite matrix (Unit 3's kind: all its eigenvalues are positive) has one too:

A=LL^{\top}

Here $L$ is lower-triangular: zeros above the diagonal, and positive numbers on it. You find its entries one at a time, starting from the top-left corner, like filling in a crossword. For a 2×2 matrix:

\ell_{11}=\sqrt{a_{11}},\qquad \ell_{21}=\frac{a_{21}}{\ell_{11}},\qquad \ell_{22}=\sqrt{a_{22}-\ell_{21}^{2}}

Example: $A=\begin{bmatrix}4&2\\2&3\end{bmatrix}$ . Then $\ell_{11}=\sqrt4=2$ , $\ell_{21}=2/2=1$ , and $\ell_{22}=\sqrt{3-1}=\sqrt2$ . Check by multiplying:

\begin{bmatrix}2&0\\1&\sqrt2\end{bmatrix}\begin{bmatrix}2&1\\0&\sqrt2\end{bmatrix}=\begin{bmatrix}4&2\\2&3\end{bmatrix}\;\checkmark

If $A$ is not positive-definite, one of the square roots gets a negative number inside, and the recipe stops. So Cholesky is also the quickest test for "positive-definite": don't hunt for eigenvalues, just try to factor.

Back to the cricket game. Say $\Sigma$ (read it: "sigma") is the covariance matrix you want: how much heights spread, how much weights spread, and how much they move together. Factor $\Sigma=LL^{\top}$ . Draw a round cloud $\mathbf{z}$ of plain random numbers. Output $\mathbf{x}=L\mathbf{z}$ . The new cloud has covariance exactly $\Sigma$ . Every simulation library does precisely this.

If you want the algebra · where the recipe comes from, and why Lz has covariance Σ

Prove it · where the 2×2 Cholesky recipe comes from — derived

Claim. If $A=\begin{bmatrix}a_{11}&a_{21}\\a_{21}&a_{22}\end{bmatrix}$ is SPD, then $A=LL^{\top}$ with $L=\begin{bmatrix}\ell_{11}&0\\\ell_{21}&\ell_{22}\end{bmatrix}$ , $\ \ell_{11}=\sqrt{a_{11}},\ \ell_{21}=a_{21}/\ell_{11},\ \ell_{22}=\sqrt{a_{22}-\ell_{21}^2}$ .

1

Multiply out

LL^{\top}

with unknown entries:

\begin{bmatrix}\ell_{11}&0\\\ell_{21}&\ell_{22}\end{bmatrix}\begin{bmatrix}\ell_{11}&\ell_{21}\\0&\ell_{22}\end{bmatrix}=\begin{bmatrix}\ell_{11}^2&\ell_{11}\ell_{21}\\\ell_{11}\ell_{21}&\ell_{21}^2+\ell_{22}^2\end{bmatrix}

Row-times-column, four times. The result is automatically symmetric — which is why this shape can only ever match a symmetric

A

.

2

Match the top-left entries:

\ell_{11}^2=a_{11}

, so

\ell_{11}=\sqrt{a_{11}}

. We take the positive root by convention. It exists because

a_{11}=\mathbf{e}_1^{\top}A\mathbf{e}_1>0

for a positive-definite

A

.

3

Match the off-diagonal entries:

\ell_{11}\ell_{21}=a_{21}

, so

\ell_{21}=a_{21}/\ell_{11}

. Division is legal because

\ell_{11}>0

from step 2. This is the only equation that involves

\ell_{21}

, so it pins it down.

4

Match the bottom-right entries:

\ell_{21}^2+\ell_{22}^2=a_{22}

, so

\ell_{22}=\sqrt{a_{22}-\ell_{21}^2}

. Done. ∎ The radicand is

a_{22}-a_{21}^2/a_{11}=\det A/a_{11}

, which is positive exactly when

\det A>0

— i.e. when both eigenvalues are positive. A failed square root here is the algorithm detecting "not positive-definite". The

n\times n

recipe repeats this entry by entry, top-left to bottom-right.

Prove it · why Lz has covariance Σ — derived

Claim. If $\mathbf{z}$ has mean $\mathbf{0}$ and covariance $I$ (independent standard normals), and $\Sigma=LL^{\top}$ , then $\mathbf{x}=L\mathbf{z}$ has covariance $\Sigma$ .

1

Covariance is the expected outer product of the centred vector. First,

\mathbf{x}

is centred:

\mathbb{E}[\mathbf{x}]=\mathbb{E}[L\mathbf{z}]=L\,\mathbb{E}[\mathbf{z}]=L\mathbf{0}=\mathbf{0}

Expectation is linear, and

L

is a constant matrix, so it slides outside.

2

Write the covariance of

\mathbf{x}

and substitute

\mathbf{x}=L\mathbf{z}

:

\operatorname{Cov}(\mathbf{x})=\mathbb{E}[\mathbf{x}\mathbf{x}^{\top}]=\mathbb{E}[(L\mathbf{z})(L\mathbf{z})^{\top}]=\mathbb{E}[L\,\mathbf{z}\mathbf{z}^{\top}L^{\top}]

(L\mathbf{z})^{\top}=\mathbf{z}^{\top}L^{\top}

— the transpose reverses the order.

3

Pull the constant matrices outside the expectation:

\mathbb{E}[L\,\mathbf{z}\mathbf{z}^{\top}L^{\top}]=L\,\mathbb{E}[\mathbf{z}\mathbf{z}^{\top}]\,L^{\top}=L\,I\,L^{\top}

Only

\mathbf{z}\mathbf{z}^{\top}

is random.

\mathbb{E}[\mathbf{z}\mathbf{z}^{\top}]=\operatorname{Cov}(\mathbf{z})=I

: each coordinate has variance 1, and different coordinates are uncorrelated.

4

LIL^{\top}=LL^{\top}=\Sigma

. ∎ Nothing used the triangular shape — any

M

with

MM^{\top}=\Sigma

works (e.g.

Q\Lambda^{1/2}

from Section 7). Cholesky is simply the cheapest such

M

to compute.

Try switch between round cloud z and shaped cloud Lz: the same 300 points. Then press not PD and see where the recipe stops.

✋ drag to orbit the cloud · the floor shows the flat 2D picture

drag the picture to orbit

Σ =

L =

—

1σ & 2σ shapes of Σ√λ₁·q₁ (long axis)√λ₂·q₂ (short axis)

The glass shell is the 1σ shape of Σ. Its axes are Σ's eigenvectors, straight from Section 7. Only L changes between the two views.

The realization

$A=LL^{\top}$

Fill in $L$ from the top-left corner, one entry at a time. It only works when $A$ is positive-definite, so it doubles as a test. And $\mathbf{x}=L\mathbf{z}$ bends a round random cloud into one with covariance $\Sigma$ .

Pause & predict

Factor $\Sigma=\begin{bmatrix}9&3\\3&5\end{bmatrix}$ in your head: $\ell_{11}=3$ , $\ell_{21}=1$ , and $\ell_{22}=$ ?

In one sentence: a positive-definite matrix has a triangular square root $L$ , found corner by corner, and $L$ is exactly the lens that turns plain random numbers into realistic data.

11

The fine print — missing directions, and the bridge to ML

Imagine this

Push the top of a deck of cards sideways. Every card slides flat, horizontally. A flat arrow stays flat. Every other arrow tilts over.

So the push (a shear) has just one eigen-direction. A 2×2 matrix usually has two. One is missing.

Does every $n\times n$ matrix have $n$ independent eigen-directions? No. But you are safe in two cases:

✓All $n$ eigenvalues are different. Then the eigen-directions are automatically independent (proof in the drawer below).
✓ $A$ is symmetric. That is the Spectral Theorem, even when eigenvalues repeat.

Trouble only comes when an eigenvalue repeats in a matrix that is not symmetric. The shear $\begin{bmatrix}1&1\\0&1\end{bmatrix}$ has $p(\lambda)=(1-\lambda)^2$ , so $\lambda=1$ counts twice. But $(A-I)\mathbf{x}=\begin{bmatrix}0&1\\0&0\end{bmatrix}\mathbf{x}=\mathbf{0}$ only says $x_2=0$ . That is just one line, through $(1,0)$ . So we have two ways of counting:

Name	In everyday words	For the shear
algebraic multiplicity	how many times $\lambda$ shows up as a root of $p_A$	$\operatorname{am}(1)=2$
geometric multiplicity	how many independent directions $\lambda$ really owns	$\operatorname{gm}(1)=1$

Always $1\le\operatorname{gm}\le\operatorname{am}$ . When $\operatorname{gm}<\operatorname{am}$ , the matrix is one direction short. We call it defective, and no change of axes will ever make it diagonal. You felt this in Section 5's widget: the shear's second direction was simply not there.

Every vertical line (orange) tilts to the blue slope under the shear; only the x-axis is left in place. One eigen-line, no second direction to be found.

Symmetric matrices never come up short: the repeated eigenvalue 3 owns a whole plane (two independent eigenvectors), and the eigenvalue 6 owns the plane's normal. Multiplicities match, and the eigenspaces are perpendicular.

If you want the algebra · why different eigenvalues give independent directions

Prove it · why distinct eigenvalues give independent eigenvectors — derived

Claim. If $A\mathbf{v}_1=\lambda_1\mathbf{v}_1$ , $A\mathbf{v}_2=\lambda_2\mathbf{v}_2$ , both non-zero, and $\lambda_1\ne\lambda_2$ , then $\mathbf{v}_1,\mathbf{v}_2$ are linearly independent. (The same trick, repeated, handles any number of distinct eigenvalues.)

1

Suppose some combination gives zero:

c_1\mathbf{v}_1+c_2\mathbf{v}_2=\mathbf{0}

. We must show

c_1=c_2=0

. That is the definition of independence: only the trivial combination gives zero.

2

Apply the matrix

A-\lambda_2I

to both sides:

c_1(A-\lambda_2I)\mathbf{v}_1+c_2(A-\lambda_2I)\mathbf{v}_2=\mathbf{0}

Multiplying

\mathbf{0}

by any matrix gives

\mathbf{0}

; on the left we distribute.

3

Evaluate each piece with the eigen-equations:

\begin{aligned}(A-\lambda_2I)\mathbf{v}_1&=\lambda_1\mathbf{v}_1-\lambda_2\mathbf{v}_1=(\lambda_1-\lambda_2)\mathbf{v}_1\\ (A-\lambda_2I)\mathbf{v}_2&=\lambda_2\mathbf{v}_2-\lambda_2\mathbf{v}_2=\mathbf{0}\end{aligned}

The shift by

\lambda_2

is chosen to send

\mathbf{v}_2

to zero and only rescale

\mathbf{v}_1

.

4

So step 2 collapses to

c_1(\lambda_1-\lambda_2)\mathbf{v}_1=\mathbf{0}

. Since

\lambda_1-\lambda_2\ne0

and

\mathbf{v}_1\ne\mathbf{0}

, we get

c_1=0

. A non-zero scalar times a non-zero vector is non-zero, so the only way out is

c_1=0

.

5

Back in step 1,

c_2\mathbf{v}_2=\mathbf{0}

with

\mathbf{v}_2\ne\mathbf{0}

forces

c_2=0

. Independent. ∎ For

k

distinct eigenvalues, apply

(A-\lambda_2I)(A-\lambda_3I)\cdots(A-\lambda_kI)

to kill every vector but

\mathbf{v}_1

, conclude

c_1=0

, and repeat. Distinct eigenvalues ⇒ a full eigenbasis ⇒ diagonalisable.

The bridge to machine learning

Your data sits in a table $A$ : 100 rows (one per example) and 8 columns (one per feature). Say the columns are independent. Now build $A^{\top}A$ , a small 8×8 table. It has two big properties:

⇆Symmetric: $(A^{\top}A)^{\top}=A^{\top}(A^{\top})^{\top}=A^{\top}A$ .
+Positive-definite: $\mathbf{x}^{\top}A^{\top}A\mathbf{x}=\|A\mathbf{x}\|^{2}>0$ for every $\mathbf{x}\ne\mathbf{0}$ , because independent columns mean $A\mathbf{x}\ne\mathbf{0}$ .

So everything in this unit works on it: real positive eigenvalues, a right-angle grain, and a Cholesky factor.

A hundred examples shrink into an 8×8 summary. Its

(i,j)

entry is the dot product of feature

i

with feature

j

. After centring the data, this is the covariance matrix, up to a scale.

So what this one fact drives least squares (Unit 9 solves $A^{\top}A\hat{\mathbf x}=A^{\top}\mathbf{b}$ ), the SVD (Unit 5 uses the eigenvectors of $A^{\top}A$ and $AA^{\top}$ ), and PCA (Units 12–13 break down the covariance matrix). This unit holds up all of them.

The realization

Different eigenvalues, or a symmetric matrix, guarantee a full set of eigen-directions. A repeated eigenvalue in a non-symmetric matrix can come up short ( $\operatorname{gm}<\operatorname{am}$ ). And the data matrix $A^{\top}A$ is always in the safe, symmetric, positive-definite club.

Pause & predict

Your data matrix $A$ has 100 rows (examples) and 8 independent columns (features). Which tools from this unit work on $A^{\top}A$ ?

In one sentence: only non-symmetric matrices with repeated eigenvalues can run short of eigen-directions, and the data matrix $A^{\top}A$ never does.

12

The whole story, in five lines

①The determinant is the area dial. Its size is how much areas (or volumes) grow. A minus sign means flipped over. Zero means squashed flat and impossible to undo. Compute it along the laziest line, or by elimination to a triangle.
②The trace adds the diagonal. It equals the sum of the eigenvalues, while the determinant equals their product: a free check.
③Eigenvectors are the directions a matrix only stretches: $A\mathbf{x}=\lambda\mathbf{x}$ . Find the eigenvalues from $\det(A-\lambda I)=0$ , then each direction by elimination.
④Symmetric matrices are turn, stretch, turn back: $A=Q\Lambda Q^{\top}$ , with real eigenvalues and right-angle directions. Pure turns have complex eigenvalues $\cos\theta\pm i\sin\theta$ .
⑤Cholesky $A=LL^{\top}$ is the square root of a positive-definite matrix. It tests positive-definiteness and turns random numbers into realistic data.

The one thing to remember

A matrix is its eigenvalues and eigenvectors, seen through a choice of axes. Find the directions it cannot turn and how much it stretches each one, and the grid of numbers turns into a simple picture. That is exactly what PCA, SVD and spectral clustering do for a living.

13

Practice arena — the unit's problem set, solved in full

These five problems come from Prof. Saurabh's practice set, which covers this unit and the next. The eigenvalue problems are here; the diagonalisation and SVD problems wait in Unit 5. Try each one on paper first. Then open the solution: every step is shown.

Problem 1Medium

Consider

A=\begin{bmatrix}4&1&1\\1&4&1\\1&1&4\end{bmatrix}.

(a) Find the characteristic polynomial and all eigenvalues with their algebraic multiplicities. (b) For every eigenvalue find a basis of its eigenspace and state the geometric multiplicity. (c) Hence decide, with reason, whether $A$ has a full basis of eigenvectors.

What this tests. Computing a $3\times 3$ characteristic polynomial cleanly, then comparing algebraic and geometric multiplicity eigenvalue by eigenvalue.

Show the worked solution — every step

Step 1 · Spot the structure.

A=3I+J

, where

J

is the all-ones matrix — every row sums to 6, a strong hint that

(1,1,1)^{\top}

will be an eigenvector with eigenvalue 6.

Step 2 · Characteristic polynomial by row reduction. Start from

\det(A-\lambda I)=\det\begin{bmatrix}4-\lambda&1&1\\1&4-\lambda&1\\1&1&4-\lambda\end{bmatrix}.

Subtract row 1 from rows 2 and 3 (determinant unchanged). Entry by entry, the new row 2 is

\big(1-(4-\lambda),\,(4-\lambda)-1,\,1-1\big)=(\lambda-3,\;3-\lambda,\;0)

and the new row 3 is

(\lambda-3,\;0,\;3-\lambda)

. Factor

(\lambda-3)

out of each of those rows:

\det(A-\lambda I)=(\lambda-3)^2\det\begin{bmatrix}4-\lambda&1&1\\1&-1&0\\1&0&-1\end{bmatrix}.

Expand the remaining determinant along row 3:

1\cdot\det\begin{bmatrix}1&1\\-1&0\end{bmatrix}+(-1)\cdot\det\begin{bmatrix}4-\lambda&1\\1&-1\end{bmatrix} = 1\cdot(0+1)+(-1)\big(-(4-\lambda)-1\big)=1+(5-\lambda)=6-\lambda.

Hence

\det(A-\lambda I)=(\lambda-3)^2(6-\lambda)=-(\lambda-6)(\lambda-3)^2.

So

\lambda=6

with algebraic multiplicity

\operatorname{am}(6)=1

, and

\lambda=3

with

\operatorname{am}(3)=2

.

Step 3 · Eigenspace of $\lambda=6$ .

(A-6I)\mathbf{v}=\mathbf{0}

reads

-2x+y+z=0,\; x-2y+z=0,\; x+y-2z=0

. Subtracting the second equation from the first gives

-3x+3y=0\Rightarrow x=y

; subtracting the third from the second gives

-3y+3z=0\Rightarrow y=z

. So

x=y=z

: basis

\{(1,1,1)^{\top}\}

, geometric multiplicity

\operatorname{gm}(6)=1

.

Step 4 · Eigenspace of $\lambda=3$ .

A-3I=J

(all ones), so all three equations collapse to the single condition

x+y+z=0

. Two free variables give

\mathbf{v}=\begin{bmatrix}-1\\1\\0\end{bmatrix},\quad\begin{bmatrix}-1\\0\\1\end{bmatrix}, \qquad \operatorname{gm}(3)=2.

Step 5 · Compare multiplicities.

\operatorname{gm}(6)=1=\operatorname{am}(6)

and

\operatorname{gm}(3)=2=\operatorname{am}(3)

: the three eigenvectors together form a basis of

\mathbb{R}^3

. (In fact

A

is symmetric, so the spectral theorem already guaranteed a full — even orthogonal — eigenbasis.)

Answer.

\det(A-\lambda I)=-(\lambda-6)(\lambda-3)^2

.

\lambda=6

: multiplicity 1, eigenvector

(1,1,1)^{\top}

;

\lambda=3

: multiplicity 2, eigenspace

\{x+y+z=0\}

with basis

(-1,1,0)^{\top},(-1,0,1)^{\top}

.

A

has a full eigenbasis.

Watch out

A repeated eigenvalue is a question, not a verdict: you must actually solve $(A-3I)\mathbf{v}=\mathbf{0}$ and count independent solutions. And keep the leading minus sign — for a $3\times 3$ , $\det(A-\lambda I)$ starts with $-\lambda^3$ , which is why the factorization reads $-(\lambda-6)(\lambda-3)^2$ .

Key takeaway

Sanity-check eigenvalues instantly: $\operatorname{tr}(A)=4+4+4=12=6+3+3$ and the row-sum trick hands you $(1,1,1)^{\top}\!\mapsto 6$ for free. For symmetric matrices, $\operatorname{gm}=\operatorname{am}$ always holds — the spectral theorem forbids "missing" eigenvectors.

Wait — really?

All three rows of $A-3I$ collapse into the single equation $x+y+z=0$ — a whole plane of eigenvectors. And the $\lambda=6$ eigenvector $(1,1,1)^{\top}$ is exactly the normal of that plane: orthogonality between eigenspaces showed up before anyone asked for it.

Problem 2Medium

Find all eigenvalues and corresponding eigenvectors of

A=\begin{bmatrix}3&-2\\4&-1\end{bmatrix},

and explain geometrically why no real eigenvector exists.

What this tests. Handling complex roots of the characteristic polynomial of a real $2\times 2$ — and reading the geometry behind them.

Show the worked solution — every step

Step 1 · Characteristic polynomial.

\det(A-\lambda I)=\det\begin{bmatrix}3-\lambda&-2\\4&-1-\lambda\end{bmatrix}=(3-\lambda)(-1-\lambda)+8.

Expand the product:

(3-\lambda)(-1-\lambda)=-3-3\lambda+\lambda+\lambda^{2}=\lambda^{2}-2\lambda-3

; adding 8 gives

\lambda^{2}-2\lambda+5=0.

Step 2 · Solve the quadratic.

\lambda=\frac{2\pm\sqrt{4-20}}{2}=\frac{2\pm\sqrt{-16}}{2}=1\pm 2i.

The discriminant is negative, so the eigenvalues are a complex-conjugate pair.

Step 3 · Eigenvectors. For

\lambda=1+2i

, the first row of

(A-\lambda I)\mathbf{v}=\mathbf{0}

gives

(3-\lambda)x-2y=0

, i.e.

(2-2i)x-2y=0\Rightarrow y=(1-i)x

. Taking

x=1

,

\mathbf{v}_1=\begin{bmatrix}1\\1-i\end{bmatrix}.

Its conjugate

\mathbf{v}_2=(1,\,1+i)^{\top}

belongs to the conjugate eigenvalue

\lambda=1-2i

.

Step 4 · Verify the pair. Entry by entry:

A\mathbf{v}_1=\big(3\cdot 1-2(1-i),\; 4\cdot 1-(1-i)\big)^{\top}=(1+2i,\;3+i)^{\top}

, while

\lambda_1\mathbf{v}_1=(1+2i)\,(1,\,1-i)^{\top}=\big(1+2i,\;(1+2i)(1-i)\big)^{\top}

and

(1+2i)(1-i)=1-i+2i-2i^{2}=3+i

. Both sides match. ✓

Step 5 · Geometric reason. A real

2\times 2

matrix with complex eigenvalues acts as a rotation combined with a scaling. A genuine rotation maps no nonzero real vector onto a multiple of itself — so no real eigenvector can exist.

Answer.

\lambda=1\pm 2i

with eigenvectors

\mathbf{v}=(1,\,1\mp i)^{\top}

. No real eigenvector exists because the eigenvalues are non-real: the map rotates the plane.

Watch out

The classic slip is expanding $(3-\lambda)(-1-\lambda)$ into $\lambda^{2}-2\lambda-3$ and then forgetting the $-(-2)(4)=+8$ from the off-diagonal — the constant term must be $\det A=-3+8=5$ . Also: use one row of $A-\lambda I$ to find the eigenvector; the other row is automatically dependent, and checking that it is doubles as a free error check.

Key takeaway

For real matrices, complex eigenvalues always arrive in conjugate pairs, and so do their eigenvectors: compute one eigenvector, conjugate every entry to get the other. One computation, two eigenpairs.

Wait — really?

$|\lambda|=\sqrt{1^{2}+2^{2}}=\sqrt{5}$ and $\det A=5=|\lambda|^{2}$ : this matrix is secretly "rotate, then stretch by $\sqrt5$ ". The determinant knew the stretch factor all along — even though no direction in the real plane is left pointing where it started.

Problem 3Easy

Consider

A=\begin{bmatrix}2&-1&1\\1&0&3\\1&-1&4\end{bmatrix}, \qquad\text{whose characteristic polynomial is } (\lambda-1)(\lambda-2)(\lambda-3).

Without forming any matrix power explicitly, compute (a) $\det(A^{8})$ , (b) $\operatorname{tr}(A^{6})$ , and (c) $\operatorname{tr}\!\left(A^{6}-7A^{2}\right)$ .

What this tests. The scalar shortcuts $\det(A^{k})=(\det A)^{k}$ and $\operatorname{tr}(A^{k})=\sum_i\lambda_i^{k}$ — handling huge powers with three numbers.

Show the worked solution — every step

Step 0 · Read off the eigenvalues. The characteristic polynomial factors as

(\lambda-1)(\lambda-2)(\lambda-3)

, so the eigenvalues are

\lambda_1=1,\ \lambda_2=2,\ \lambda_3=3

. Quick sanity check against the matrix:

\operatorname{tr}A=2+0+4=6=1+2+3

and

\det A=6=1\cdot 2\cdot 3

. ✓

Part (a) · Determinant of a power.

A^{k}

has eigenvalues

\lambda_i^{k}

, and the determinant is their product — which is

(\det A)^{k}

:

\det(A^{8})=(\det A)^{8}=6^{8}.

Compute it in two squarings:

6^{2}=36,\ 6^{4}=36^{2}=1296,\ 6^{8}=1296^{2}=1\,679\,616

.

Part (b) · Trace of a power. Trace is the sum of eigenvalues, taken after powering:

\operatorname{tr}(A^{6})=\sum_i \lambda_i^{6}=1^{6}+2^{6}+3^{6}=1+64+729=794.

Part (c) · Trace of a polynomial in $A$ . Trace is linear, and

A^{6},A^{2}

have eigenvalues

\lambda_i^{6},\lambda_i^{2}

, so

\operatorname{tr}\!\left(A^{6}-7A^{2}\right)=\sum_i\big(\lambda_i^{6}-7\lambda_i^{2}\big)=794-7\,(1+4+9)=794-98=696.

Every answer used only the three scalars

\lambda_i

— not a single

3\times 3

multiplication.

Answer.

\det(A^{8})=6^{8}=1\,679\,616

;

\operatorname{tr}(A^{6})=794

;

\operatorname{tr}(A^{6}-7A^{2})=696

.

Watch out

$\operatorname{tr}(A^{6})\neq(\operatorname{tr}A)^{6}$ — the trace of a power is the sum of powered eigenvalues, $\sum\lambda_i^{6}$ , not the power of the sum. The determinant is the multiplicative one: $\det(A^{k})=(\det A)^{k}$ really does hold.

Key takeaway

$\operatorname{tr}(A)=\sum_i\lambda_i$ and $\det(A)=\prod_i\lambda_i$ , and powers or polynomials of $A$ simply apply themselves to each $\lambda_i$ . To handle a high power, find the eigenvalues first — then work entirely with scalars.

Wait — really?

$A^{8}$ is a $3\times 3$ matrix of six-to-seven-digit entries that we never wrote down — yet its determinant, $1\,679\,616$ , came from the single number $\det A=6$ . The eigenvalues compress everything the power will ever do to volume and trace.

Problem 4Medium

Let $A$ be a $3\times 3$ matrix whose eigenvalues are $1,\,2,\,4$ , and define

B=A^{2}-5A+6I.

(a) Show that if $\lambda$ is an eigenvalue of $A$ (with eigenvector $\mathbf{v}$ ), then $\lambda^{2}-5\lambda+6$ is an eigenvalue of $B$ (with the same eigenvector). (b) Hence find the eigenvalues of $B$ , then $\det(B)$ and $\operatorname{tr}(B)$ , and state whether $B$ is invertible. (c) For which eigenvalues of $A$ would $B$ fail to be invertible?

What this tests. The spectral mapping idea $\lambda\mapsto p(\lambda)$ : polynomials in $A$ keep the eigenvectors and transform only the eigenvalues.

Show the worked solution — every step

Part (a) · Spectral mapping. Suppose

A\mathbf{v}=\lambda\mathbf{v}

. Then

A^{2}\mathbf{v}=A(\lambda\mathbf{v})=\lambda^{2}\mathbf{v}

, and so

B\mathbf{v}=(A^{2}-5A+6I)\mathbf{v}=\lambda^{2}\mathbf{v}-5\lambda\mathbf{v}+6\mathbf{v}=(\lambda^{2}-5\lambda+6)\,\mathbf{v}.

Thus

\mathbf{v}

is still an eigenvector — now with eigenvalue

p(\lambda)=\lambda^{2}-5\lambda+6

.

Part (b) · Apply $p$ to each eigenvalue.

p(1)=1-5+6=2,\qquad p(2)=4-10+6=0,\qquad p(4)=16-20+6=2.

So

B

has eigenvalues

\{2,\,0,\,2\}

, and therefore

\det(B)=2\cdot 0\cdot 2=0,\qquad \operatorname{tr}(B)=2+0+2=4.

Because

0

is an eigenvalue (equivalently

\det B=0

),

B

is not invertible — even though

A

itself is (none of

1,2,4

is zero).

Part (c) · When is $B$ singular?

B

is singular exactly when

p(\lambda)=0

for some eigenvalue

\lambda

of

A

. Factoring,

p(\lambda)=(\lambda-2)(\lambda-3)

: this vanishes iff

A

has

2

or

3

as an eigenvalue. Here

\lambda=2

occurs — which is precisely why

B

is singular.

Remark · Cayley–Hamilton. If

A

's characteristic polynomial were itself

\lambda^{2}-5\lambda+\text{const}

, then

p(A)

collapses to a multiple of

I

. E.g.

A=\begin{bmatrix}4&5\\0&1\end{bmatrix}

has eigenvalues

4,1

and characteristic polynomial

\lambda^{2}-5\lambda+4

; Cayley–Hamilton gives

A^{2}-5A+4I=0

, hence

A^{2}-5A+6I=2I

.

Answer.

p(\lambda)=\lambda^{2}-5\lambda+6

maps each eigenvalue;

B

has eigenvalues

2,\,0,\,2

.

\det(B)=0

,

\operatorname{tr}(B)=4

, so

B

is not invertible.

B

is singular iff

A

has eigenvalue

2

or

3

(the roots of

p

).

Watch out

The eigenvectors do not change — only the eigenvalues pass through $p$ . And invertibility does not transfer: an invertible $A$ can easily produce a singular $p(A)$ , as it just did.

Key takeaway

Eigenvalues of $p(A)$ are $p(\text{eigenvalues of }A)$ , with the same eigenvectors. So $p(A)$ is singular $\iff$ some eigenvalue of $A$ is a root of $p$ .

Wait — really?

$A$ is perfectly invertible, yet the innocent-looking combination $A^{2}-5A+6I$ is singular — because $p(t)=(t-2)(t-3)$ happens to have a root sitting exactly on one of $A$ 's eigenvalues. Matrix polynomials can create zero out of nonzero ingredients.

Problem 5Hard

For the symmetric tridiagonal matrix

A=\begin{bmatrix}2&1&0\\1&2&1\\0&1&2\end{bmatrix},

find its eigenvalues, an orthonormal set of eigenvectors, and hence write $A=PDP^{\top}$ with $P$ orthogonal.

What this tests. The spectral theorem in action: a symmetric matrix with irrational eigenvalues still yields an exact orthonormal eigenbasis and $A=PDP^{\top}$ .

Show the worked solution — every step

Step 1 · Eigenvalues. Expanding

\det(A-\lambda I)

along the first row:

\det(A-\lambda I)=(2-\lambda)\big[(2-\lambda)^{2}-1\big]-1\cdot(2-\lambda)=(2-\lambda)\big[(2-\lambda)^{2}-2\big]=0.

Hence

\lambda=2

, or

(2-\lambda)^{2}=2\Rightarrow\lambda=2\pm\sqrt{2}

. The eigenvalues are

\lambda_1=2+\sqrt{2},\qquad \lambda_2=2,\qquad \lambda_3=2-\sqrt{2}.

Step 2 · Eigenvector for $\lambda=2$ .

(A-2I)\mathbf{v}=\mathbf{0}

reads

y=0

(rows 1 and 3) and

x+z=0

(row 2), so

\mathbf{v}_2=(1,0,-1)^{\top}

.

Step 3 · Eigenvectors for $\lambda=2\pm\sqrt{2}$ . For

\lambda=2+\sqrt{2}

: row 1 gives

-\sqrt{2}\,x+y=0\Rightarrow y=\sqrt{2}\,x

, and row 3 gives

y-\sqrt{2}\,z=0\Rightarrow y=\sqrt{2}\,z

, so

x=z

. (Row 2 checks out:

x-\sqrt{2}\,y+z=x-2x+x=0

. ✓) Thus

\mathbf{v}_1=(1,\sqrt{2},1)^{\top}

. For

\lambda=2-\sqrt{2}

: the same computation with

-\sqrt2

gives

y=-\sqrt{2}\,x

and

x=z

, so

\mathbf{v}_3=(1,-\sqrt{2},1)^{\top}

.

Step 4 · Orthogonality check.

\mathbf{v}_1\cdot\mathbf{v}_2=1+0-1=0

,

\mathbf{v}_1\cdot\mathbf{v}_3=1-2+1=0

,

\mathbf{v}_2\cdot\mathbf{v}_3=1-0-1=0

— mutually orthogonal, exactly as the spectral theorem guarantees for distinct eigenvalues of a symmetric matrix.

Step 5 · Normalise and assemble.

\lVert\mathbf{v}_2\rVert=\sqrt{2}

and

\lVert\mathbf{v}_1\rVert=\lVert\mathbf{v}_3\rVert=\sqrt{1+2+1}=2

, so

P=\begin{bmatrix}\tfrac12&\tfrac{1}{\sqrt2}&\tfrac12\\[2pt] \tfrac{\sqrt2}{2}&0&-\tfrac{\sqrt2}{2}\\[2pt] \tfrac12&-\tfrac{1}{\sqrt2}&\tfrac12\end{bmatrix},\qquad D=\begin{bmatrix}2+\sqrt2&0&0\\0&2&0\\0&0&2-\sqrt2\end{bmatrix},\qquad A=PDP^{\top}.

Since the columns are orthonormal,

P^{\top}P=I

, i.e.

P^{-1}=P^{\top}

— no matrix inversion needed.

Answer. Eigenvalues

2+\sqrt2,\ 2,\ 2-\sqrt2

with orthonormal eigenvectors

\tfrac12(1,\sqrt2,1)^{\top}

,

\tfrac{1}{\sqrt2}(1,0,-1)^{\top}

,

\tfrac12(1,-\sqrt2,1)^{\top}

;

A=PDP^{\top}

with

P

orthogonal and

D=\operatorname{diag}(2+\sqrt2,\,2,\,2-\sqrt2)

.

Watch out

Normalise each eigenvector by its own length: here $\lVert\mathbf{v}_2\rVert=\sqrt2$ but $\lVert\mathbf{v}_1\rVert=\lVert\mathbf{v}_3\rVert=2$ — one common scale factor will not do. Orthogonality comes free from symmetry; orthonormality is your job.

Key takeaway

Symmetric matrix $\Rightarrow$ real eigenvalues, orthogonal eigenvectors, and $A=PDP^{\top}$ with $P^{-1}=P^{\top}$ . The transpose replaces the inverse — the single biggest computational gift of the spectral theorem.

Wait — really?

An integer matrix produced irrational eigenvalues $2\pm\sqrt2$ — and the eigenvector $(1,\sqrt2,1)^{\top}$ carries its own eigenvalue's $\sqrt2$ inside its entries. Yet the factorization $PDP^{\top}$ reproduces every integer entry of $A$ exactly: the irrationals conspire and cancel.

Next up

Unit 5 · Matrix Decompositions & SVD →

Eigenvalues work beautifully — right up until a matrix is not square, or lacks enough eigenvectors. Unit 5 climbs the ladder from the spectral theorem to the SVD, the decomposition that never fails, and ends with you compressing a real photograph with it, live.

← All units