Unit 3 · Analytic Geometry — The Math Behind the Machine

1

The quest for "similar"

Imagine this

You finish a film on Netflix. Two seconds later it says: "Because you watched this, try…". How does it choose?

Behind the screen, you are a list of numbers: how much you like action, romance, comedy, and so on. Every film is a list of numbers too. The app looks for films whose list is close to yours. Spotify does the same with songs. Google does it with web pages. ChatGPT does it with the meaning of words.

So everything rests on one small question: what does "close" actually mean?

At school, "close" meant one thing: put a ruler between the two points and read the distance. That works. But it is only one way to measure.

Think of choosing a film. Maybe the director matters to you far more than the running time. Then a good "closeness" score should care more about the director and less about the running time. That is a different ruler. It is not wrong. It just measures what you care about.

This unit teaches you how to build your own rulers, and how to make sure they never break. Here is the path, one step at a time:

①Length — what rules must any "length" follow? (norms)
②The dot product — one number that tells you if two lists agree.
③Your own ruler — the recipe for making new ones (inner products and the matrices behind them).
④Distance and angle — "how far apart?" and "pointing the same way?"
⑤Perfect grids — how to turn any set of directions into clean, square ones (Gram–Schmidt).

In one sentence: machine learning keeps asking "how similar are these two things?", and this unit is about the rulers that answer it.

2

Norms — what "length" is allowed to mean

Imagine this

You stand at a street corner. Your friend is 3 blocks east and 4 blocks north. How far away is she?

A crow flies straight over the buildings: 5 blocks. A taxi has to follow the streets, 3 blocks and then 4 blocks: 7 blocks.

Who is right? Both. They just measure "far" in different ways.

So "length" is not one fixed thing. It is a choice of rule. Mathematicians call such a rule a norm, and they write the length of an arrow $\mathbf{x}$ as $\|\mathbf{x}\|$ (read it: "the length of x").

You can invent any rule you like. But it must keep three common-sense promises, or it is not really a length:

The promise	In everyday words	In symbols
Scaling	Walk twice as far, and the length doubles. Walking backwards is still the same distance.	$\\|\lambda\mathbf{x}\\|=\|\lambda\|\,\\|\mathbf{x}\\|$
No shortcut	A detour is never shorter than going straight.	$\\|\mathbf{x}+\mathbf{y}\\|\le\\|\mathbf{x}\\|+\\|\mathbf{y}\\|$
Only "nowhere" is zero	Lengths are never negative. Only staying put has length 0.	$\\|\mathbf{x}\\|\ge0$ , and $=0$ only for $\mathbf{x}=\mathbf{0}$

The two rules from our story, written out. For the point $(3,4)$ :

\displaystyle \|\mathbf{x}\|_1=|x_1|+|x_2|+\cdots

taxi: add up the blocks ·

3+4=7

\displaystyle \|\mathbf{x}\|_2=\sqrt{x_1^2+x_2^2+\cdots}

crow: Pythagoras ·

\sqrt{9+16}=5

Each rule has a "fingerprint": the shape made by all the points at length exactly 1. We call it the unit ball. For the crow it is a round circle (a sphere in 3D). For the taxi it is a diamond. Why a diamond? Because for a taxi, one block east costs the same as one block north, so the points with "1 block total" form a tilted square.

Try: drag x onto an axis and watch the two rulers agree; then switch on "zoom to unit balls" to see the circle-vs-diamond fingerprint up close.

drag the picture to orbit

x₁3

x₂4

x₃0

‖x‖₂ (crow) = 5 ‖x‖₁ (taxi) = 7

—

The dashed shapes are each norm's unit ball — everything of length 1. The crow's is a circle; the taxi's is a diamond, because trading x-blocks for y-blocks costs the same. The shape of the unit ball is the norm's fingerprint — in Section 5 we bend the circle into an ellipse.

The realization

A norm is just a rule for measuring length. Different rules give different shapes of "all points at distance 1": a circle for the crow, a diamond for the taxi. When a machine-learning model measures error or closeness, it has quietly picked one of these rules, and the pick changes the answer.

Pause & predict

A new rule: "the length of x = how many of its numbers are not zero." So $(5,0,2)$ has length 2. Is this a real length (a norm)?

Pause & predict

A taxi drives from the corner to the point $(3,4)$ on a perfect street grid. How far does the meter run, and how far did the crow fly?

In one sentence: a norm is any rule for length that scales properly, allows no shortcuts, and gives zero only to "nowhere".

3

The dot product — one number that reads two vectors

Imagine this

You and a friend rate four films from $-2$ (hated it) to $+2$ (loved it). Do you have the same taste?

Here is a quick test. For each film, multiply your rating by your friend's rating. Then add the four answers.

😀You both loved it, or you both hated it → the product is positive. Agreement adds points.
😠One loved it, one hated it → the product is negative. A clash takes points away.
😐One of you didn't care (0) → the product is 0. That film doesn't count.

Big positive total: same taste. Big negative total: opposite taste. Near zero: your tastes are unrelated.

That score has a name: the dot product. You just invented it. For two lists $\mathbf{x}$ and $\mathbf{y}$ it is

\mathbf{x}\cdot\mathbf{y}\;=\;\mathbf{x}^T\mathbf{y}\;=\;x_1y_1+x_2y_2+\cdots+x_ny_n

Pair up, multiply, add. (The $\mathbf{x}^T\mathbf{y}$ form is Unit 1's "row times column", giving one number.) One example, slowly. Take $\mathbf{x}=(1,\,2,\,-1)$ and $\mathbf{y}=(3,\,-1,\,2)$ :

\mathbf{x}^T\mathbf{y}=\underbrace{(1)(3)}_{+3\ \text{agree}}\;+\;\underbrace{(2)(-1)}_{-2\ \text{clash}}\;+\;\underbrace{(-1)(2)}_{-2\ \text{clash}}\;=\;-1

One agreement, two clashes. The total is a little below zero, so these two lean slightly away from each other.

Try: press ▶ and watch each movie drop its weights. Then drag one friend's rating and see the scale tip. Can you make it balance at exactly 0?

drag the picture to orbit

Your ratings are fixed. Drag your friend's ratings. Same sign = agree (green cubes). Opposite signs = clash (orange cubes).

score xᵀy = — direction match cos ω = —

—

Aha — the ruler was hiding inside

Now compare a list with itself: $\mathbf{x}^T\mathbf{x}=x_1^2+x_2^2+\cdots$ . That is exactly the crow's length, squared! For $(3,4)$ : $9+16=25=5^2$ .

So length is just "how much you agree with yourself." This tiny fact is the key to the whole unit. Whoever decides how two lists are multiplied also decides what "length" means. Change the multiplication, and you get a new geometry.

There is also a picture behind the score. For two arrows with an angle $\omega$ between them:

\mathbf{x}^T\mathbf{y}\;=\;\|\mathbf{x}\|\,\|\mathbf{y}\|\cos\omega

Read it like this. Pointing the same general way (angle below 90°): positive. At a perfect right angle: exactly zero, because the agreements and the clashes cancel. Pointing opposite ways: negative. So the "rating game" number and the angle are one and the same thing. Sections 6 and 7 show why this is always allowed.

The realization

$\mathbf{x}^T\mathbf{y}=\sum_i x_iy_i$

The dot product is an agreement score. Every "people like you also watched…" system is built on this one line of arithmetic.

Pause & predict

Compute $(2,\,-1,\,3)\cdot(1,\,4,\,-2)$ . Would a film app say these two people have the same taste?

In one sentence: multiply matching entries and add them up; the total says how much two lists agree, and a list's agreement with itself is its length squared.

4

Inner products — the blueprint under the dot product

Imagine this

A chef tastes a great dish and asks: "What makes this work? Salt, acid, heat." Once she knows the three things that matter, she can cook new dishes that also work.

We will do the same with the dot product. What makes it such a good agreement score? Find the key ingredients. Then we can cook up new scores that work just as well.

It turns out only three ingredients matter. Call our score $\langle\mathbf{x},\mathbf{y}\rangle$ (read it: "the score of x with y").

①Fair with sums and scaling. Double one list, and the score doubles. Split a list into two parts, and the score splits too. Why we need it: it lets us do ordinary algebra with the score. (The formal name is bilinear: $\langle\lambda\mathbf{x}+\psi\mathbf{y},\mathbf{z}\rangle=\lambda\langle\mathbf{x},\mathbf{z}\rangle+\psi\langle\mathbf{y},\mathbf{z}\rangle$ , and the same for the second slot.)
②Same both ways. How much you agree with me equals how much I agree with you: $\langle\mathbf{x},\mathbf{y}\rangle=\langle\mathbf{y},\mathbf{x}\rangle$ . (Name: symmetric.)
③Agreeing with yourself is always positive. $\langle\mathbf{x},\mathbf{x}\rangle>0$ for every list that is not all zeros. Why we need it: this number is about to become "length squared", and a length squared can't be negative. (Name: positive definite.)

The realization

Any score with these three ingredients is called an inner product. The dot product is just the most famous one. Every inner product gives you its own length, its own distance and its own angle, and they all behave sensibly. The natural next question: how do we cook up new ones? (Next section.)

Pause & predict

Try this score on 2-number lists: $\langle\mathbf{x},\mathbf{y}\rangle=x_1y_1-x_2y_2$ (the second pair counts negative). It is fair with sums and the same both ways. Which ingredient does it break?

In one sentence: an inner product is any agreement score that is fair with sums, the same both ways, and always positive for a list with itself.

5

The engine room — symmetric positive-definite matrices

Imagine this

A mixing desk in a music studio has a slider for each channel. Push the bass slider up and the bass counts for more in the final sound.

A matrix can be a mixing desk for our agreement score. It decides how much each direction counts, and even how directions mix together.

Here is the trick. The ordinary dot product secretly has a matrix inside it, the identity $I$ (all sliders at 1):

\mathbf{x}\cdot\mathbf{y}=\mathbf{x}^T I\,\mathbf{y}

Now swap $I$ for another matrix $A$ . You get a new score:

\langle\mathbf{x},\mathbf{y}\rangle_A=\mathbf{x}^T A\,\mathbf{y}

Example: $A=\begin{pmatrix}2&0\\0&1\end{pmatrix}$ gives $\langle\mathbf{x},\mathbf{y}\rangle_A=2x_1y_1+x_2y_2$ . The first direction now counts double.

Does every matrix give a good score? No. It needs the two properties from the last section, turned into matrix language:

②Symmetric: $A=A^T$ (the matrix is its own mirror image across the diagonal). This gives "same both ways".
③Positive definite: $\mathbf{x}^TA\mathbf{x}>0$ for every non-zero $\mathbf{x}$ . This gives "lengths are positive".

Such a matrix is called SPD (symmetric positive-definite). The big fact from the slides: every inner product on $\mathbb{R}^n$ is $\mathbf{x}^TA\mathbf{y}$ for some SPD matrix $A$ , and every SPD matrix gives one. So "design a ruler" means "pick an SPD matrix".

Two quick facts follow straight away:

•The diagonal is positive. Test the first direction $\mathbf{e}_1=(1,0,\dots)$ against itself: $\mathbf{e}_1^TA\mathbf{e}_1=A_{11}$ . That is a length squared, so $A_{11}>0$ . Same for every diagonal entry.
• $A$ can be undone (it is invertible). If $A\mathbf{x}=\mathbf{0}$ for some non-zero $\mathbf{x}$ , then $\mathbf{x}^TA\mathbf{x}=0$ , which is not allowed. So no non-zero arrow is squashed to zero.

What does the matrix do to the picture? Watch the fingerprint, the shape of "all points at length 1", as you change $A$ :

Try: press diag(2, 1) and read the expansion of xᵀAy below the matrix — then type a₁₂ = 1.5 with a₁₁ = a₂₂ = 1 and watch the SPD test fail the moment det ≤ 0.

drag the picture to orbit

A = a₂₁ mirrors a₁₂
(symmetry, enforced)

det = 1 SPD ✓

—

The test vectors are the slides' pair $\mathbf{x}=(1,1)$ , $\mathbf{y}=(-1,1)$ ; the faint grey circle is the ordinary Euclidean unit ball, kept for comparison. Under $I$ the pair is orthogonal; under diag(2, 1) their inner product becomes −1 and $\cos\omega=-\tfrac13$ — orthogonality is a property of the ruler, not just the vectors. The broken engine shows why SPD is non-negotiable: some directions get a negative "squared length", and the unit ball tears open into a hyperbola.

The realization

The matrix $A$ reshapes the unit ball. The identity gives a round ball. A diagonal $A$ squashes it into an egg along the axes. Off-diagonal numbers tilt the egg. A matrix that is not positive-definite tears the ball open, because some arrows would get a negative length squared.

Pause & predict

Why must every number on the diagonal of a positive-definite matrix be positive?

How do you test a matrix for SPD?

"Positive for every arrow" sounds like infinitely many checks. For a 2×2 symmetric matrix $\begin{pmatrix}a&b\\b&d\end{pmatrix}$ it boils down to two quick checks:

1the top-left number is positive: $a>0$ ;
2the determinant is positive: $ad-b^2>0$ .

Example: $\begin{pmatrix}2&1\\1&2\end{pmatrix}$ : $2>0$ and $4-1=3>0$ . SPD ✓. And $\begin{pmatrix}1&2\\2&1\end{pmatrix}$ : $1>0$ but $1-4=-3$ . Not SPD ✗. For bigger matrices, check every top-left square block ("leading minor"). The drawer shows where this comes from: the school trick of completing the square.

If you want the algebra · the 2×2 test, and a 3×3 worked example

Prove it · the 2×2 SPD test (Sylvester's criterion) — derived

Claim. A symmetric $A=\begin{pmatrix}a&b\\b&d\end{pmatrix}$ is positive-definite iff $a>0$ and $ad-b^2>0$ (the two leading minors are positive).

1

Write out the quadratic form for

\mathbf{x}=(x_1,x_2)

:

\mathbf{x}^TA\mathbf{x}=a\,x_1^2+2b\,x_1x_2+d\,x_2^2

Multiply the sandwich out: the two off-diagonal

b

's give the same cross term

b\,x_1x_2

twice.

2

Test the first axis. Put

\mathbf{x}=(1,0)

: the form equals

a

. So positive-definiteness forces

a>0

. One well-chosen test vector settles the sign of

a

. From now on we may divide by

a

.

3

Complete the square in

x_1

:

a\,x_1^2+2b\,x_1x_2+d\,x_2^2=a\Big(x_1+\frac{b}{a}x_2\Big)^2+\Big(d-\frac{b^2}{a}\Big)x_2^2

Expand the right side to check:

a(x_1+\tfrac{b}{a}x_2)^2=a x_1^2+2b\,x_1x_2+\tfrac{b^2}{a}x_2^2

; the extra

\tfrac{b^2}{a}x_2^2

is cancelled by the second bracket. Nothing lost, nothing added.

4

Read the sign. The first term is

a\cdot(\text{square})\ge0

. If

d-\tfrac{b^2}{a}>0

, both terms are

\ge0

and they vanish together only when

x_2=0

and

x_1+\tfrac{b}{a}x_2=0

, i.e.

x_1=0

. So the form is

>0

for every

\mathbf{x}\neq\mathbf{0}

. A weighted sum of squares with positive weights is positive unless every square is zero — and that only happens at the zero vector.

5

Conversely, if

d-\tfrac{b^2}{a}\le0

, the vector

\mathbf{x}=(-\tfrac{b}{a},1)

kills the first square and leaves

\mathbf{x}^TA\mathbf{x}=d-\tfrac{b^2}{a}\le0

: not positive-definite. Multiply the condition by

a>0

:

d-\tfrac{b^2}{a}>0\iff ad-b^2>0

. So SPD

\iff a>0

and

\det A>0

. ∎ The witness vector in step 5 is exactly how the practice problems "exhibit a vector with negative squared length". The same completing-the-square idea, repeated column by column, gives the

n\times n

version: all leading minors positive.

Worked example · a 3×3 matrix, proved SPD by completing the square (the by-hand technique)

Take $A=\begin{pmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{pmatrix}$ . Multiply out the form and complete squares left to right, one variable at a time:

\begin{aligned}\mathbf{x}^TA\mathbf{x}&=2x_1^2-2x_1x_2+2x_2^2-2x_2x_3+2x_3^2\\&=2\Big(x_1-\tfrac12x_2\Big)^2+\tfrac32x_2^2-2x_2x_3+2x_3^2\\&=2\Big(x_1-\tfrac12x_2\Big)^2+\tfrac32\Big(x_2-\tfrac23x_3\Big)^2+\tfrac43x_3^2.\end{aligned}

Three squares with positive weights $2,\tfrac32,\tfrac43$ : the form is $>0$ unless $x_3=0$ , then $x_2=0$ , then $x_1=0$ . So $A$ is SPD. Cross-check with Sylvester: the leading minors are $2,\ 3,\ 4$ — all positive — and the three weights are their ratios $2,\ 3/2,\ 4/3$ . Completing the square and the minors test are the same computation in two costumes.

Pause & predict

Is $M=\begin{pmatrix}1&2\\2&1\end{pmatrix}$ positive-definite? (It is symmetric, and its diagonal is positive.)

In one sentence: to design a ruler, pick a symmetric matrix that gives every non-zero arrow a positive length squared, and test it with the top-left number and the determinant.

6

Lengths, distances — and the inequality that runs everything

Every inner product comes with a free ruler and a free tape measure:

📏Length: agree with yourself, then take the square root: $\|\mathbf{x}\|=\sqrt{\langle\mathbf{x},\mathbf{x}\rangle}$ .
📍Distance between two points: the length of the gap between them, $d(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|$ .

With the ordinary dot product these are the usual length and distance from school. Any sensible distance (a metric) keeps four promises: it is never negative; it is zero only from a point to itself; A to B equals B to A; and a stop along the way never makes a trip shorter.

Distance and agreement are linked

Expand the distance and the agreement score pops out:

\|\mathbf{x}-\mathbf{y}\|^2=\|\mathbf{x}\|^2+\|\mathbf{y}\|^2-2\langle\mathbf{x},\mathbf{y}\rangle

Keep the two lengths fixed. Then more agreement means less distance. Every point the score goes up, the squared distance goes down by two. So "find the nearest" and "find the most similar" are really the same search. Just don't mix the two up in the middle of a formula: one goes up when the other goes down.

One warning: it doesn't work the other way round. Every inner product gives a length, but not every length comes from an inner product. The taxi's length does not. The picture shows the tell-tale sign.

The tell-tale sign. Take $\mathbf{x}=(1,0)$ and $\mathbf{y}=(0,1)$ and look at the two diagonals of the square they make. A length that comes from an inner product always passes the parallelogram rule: $\|\mathbf{x}+\mathbf{y}\|^2+\|\mathbf{x}-\mathbf{y}\|^2=2(\|\mathbf{x}\|^2+\|\mathbf{y}\|^2)$ . The crow passes: $2+2=4$ ✓. The taxi fails: each diagonal costs 2 blocks, so $4+4=8\neq4$ ✗. No inner product can make the diamond.

Imagine this

At noon the sun is straight overhead. You hold a stick at a slant. Its shadow on the ground is never longer than the stick. It matches the stick only when you lay the stick flat.

That everyday fact is the most useful inequality in this unit. It is called Cauchy–Schwarz:

|\langle\mathbf{x},\mathbf{y}\rangle|\;\le\;\|\mathbf{x}\|\,\|\mathbf{y}\|

In words: the agreement score can never be bigger than the two lengths multiplied. Why is this the shadow fact? The score is $\|\mathbf{x}\|$ times the shadow of $\mathbf{y}$ on $\mathbf{x}$ 's line. The shadow is never longer than $\mathbf{y}$ itself. So the score is never more than $\|\mathbf{x}\|\,\|\mathbf{y}\|$ . The two sides are equal only when the stick lies flat, that is, when $\mathbf{y}$ points exactly along $\mathbf{x}$ (or exactly opposite).

Try: drag the angle. The stick's shadow on the road grows and shrinks, but it can never be longer than the stick — so the score can never pass the ceiling.

drag the picture to orbit

angle θ30°

⟨x,y⟩ = — ‖x‖‖y‖ = 6.00

—

Sun straight overhead. The shadow of a leaning stick is always shorter than the stick. That is the whole inequality.

The realization

$|\langle\mathbf{x},\mathbf{y}\rangle|\le\|\mathbf{x}\|\,\|\mathbf{y}\|\qquad\text{and}\qquad\|\mathbf{x}+\mathbf{y}\|\le\|\mathbf{x}\|+\|\mathbf{y}\|$

A shadow never beats the stick, and a detour never beats the straight road. The first fact makes angles possible (next section). The second fact makes sure every inner product gives a proper length.

If you want the algebra · 3 proofs, step by step

Prove it · distance and inner product are wired together — derived

Claim. $\|\mathbf{x}-\mathbf{y}\|^2=\|\mathbf{x}\|^2+\|\mathbf{y}\|^2-2\langle\mathbf{x},\mathbf{y}\rangle$ in every inner-product space.

1

Start from the definition of the induced norm, applied to the difference:

\|\mathbf{x}-\mathbf{y}\|^2=\langle\mathbf{x}-\mathbf{y},\,\mathbf{x}-\mathbf{y}\rangle

Squared length is the inner product of a vector with itself — no step yet, just the definition.

2

Open the left slot with bilinearity, then the right slot:

=\langle\mathbf{x},\mathbf{x}\rangle-\langle\mathbf{x},\mathbf{y}\rangle-\langle\mathbf{y},\mathbf{x}\rangle+\langle\mathbf{y},\mathbf{y}\rangle

Linearity in each argument lets the minus signs and the sums pass through, exactly like multiplying out

(p-q)(p-q)

.

3

Use symmetry to merge the two middle terms, and the definition of the norm on the outer two:

\|\mathbf{x}-\mathbf{y}\|^2=\|\mathbf{x}\|^2+\|\mathbf{y}\|^2-2\langle\mathbf{x},\mathbf{y}\rangle.

∎

\langle\mathbf{y},\mathbf{x}\rangle=\langle\mathbf{x},\mathbf{y}\rangle

by axiom ②. So what: with the lengths fixed, distance goes down exactly as the inner product goes up — the two gauges are one equation.

Prove it · Cauchy–Schwarz — derived from one clever vector

Claim. For any $\mathbf{u},\mathbf{v}$ in an inner-product space, $\langle\mathbf{u},\mathbf{v}\rangle^2\le\langle\mathbf{u},\mathbf{u}\rangle\langle\mathbf{v},\mathbf{v}\rangle$ , i.e. $|\langle\mathbf{u},\mathbf{v}\rangle|\le\|\mathbf{u}\|\|\mathbf{v}\|$ .

1

If

\mathbf{v}=\mathbf{0}

both sides are

0

and there is nothing to prove. So assume

\mathbf{v}\neq\mathbf{0}

, which gives

\langle\mathbf{v},\mathbf{v}\rangle>0

. Positive definiteness (axiom ③) is what lets us divide by

\langle\mathbf{v},\mathbf{v}\rangle

later.

2

For any number

\alpha

, the vector

\mathbf{u}-\alpha\mathbf{v}

has a squared length, and squared lengths are never negative:

0\le\langle\mathbf{u}-\alpha\mathbf{v},\,\mathbf{u}-\alpha\mathbf{v}\rangle

Axiom ③ again — this single inequality is the entire input; the rest is algebra.

3

Multiply out with bilinearity and symmetry (the same expansion as the identity above):

0\le\langle\mathbf{u},\mathbf{u}\rangle-2\alpha\langle\mathbf{u},\mathbf{v}\rangle+\alpha^2\langle\mathbf{v},\mathbf{v}\rangle

A quadratic in

\alpha

that is never negative — for every choice of

\alpha

, including the smartest one.

4

Choose

\alpha=\dfrac{\langle\mathbf{u},\mathbf{v}\rangle}{\langle\mathbf{v},\mathbf{v}\rangle}

(the shadow coefficient of Section 11) and substitute:

0\le\langle\mathbf{u},\mathbf{u}\rangle-2\frac{\langle\mathbf{u},\mathbf{v}\rangle^2}{\langle\mathbf{v},\mathbf{v}\rangle}+\frac{\langle\mathbf{u},\mathbf{v}\rangle^2}{\langle\mathbf{v},\mathbf{v}\rangle}=\langle\mathbf{u},\mathbf{u}\rangle-\frac{\langle\mathbf{u},\mathbf{v}\rangle^2}{\langle\mathbf{v},\mathbf{v}\rangle}

Check the middle term:

2\alpha\langle\mathbf{u},\mathbf{v}\rangle=2\langle\mathbf{u},\mathbf{v}\rangle^2/\langle\mathbf{v},\mathbf{v}\rangle

; and

\alpha^2\langle\mathbf{v},\mathbf{v}\rangle=\langle\mathbf{u},\mathbf{v}\rangle^2/\langle\mathbf{v},\mathbf{v}\rangle

. Two of the same thing minus one leaves minus one.

5

Multiply through by

\langle\mathbf{v},\mathbf{v}\rangle>0

and rearrange:

\langle\mathbf{u},\mathbf{v}\rangle^2\le\langle\mathbf{u},\mathbf{u}\rangle\langle\mathbf{v},\mathbf{v}\rangle

; take square roots:

|\langle\mathbf{u},\mathbf{v}\rangle|\le\|\mathbf{u}\|\|\mathbf{v}\|

. ∎ Equality holds exactly when step 2 was an equality, i.e.

\mathbf{u}-\alpha\mathbf{v}=\mathbf{0}

: u is a multiple of v. Nothing used the dot product specifically — the proof runs verbatim for every inner product, which is why Problem 8 can "verify" it under a weighted ruler.

Prove it · the triangle inequality follows from Cauchy–Schwarz — derived

Claim. $\|\mathbf{x}+\mathbf{y}\|\le\|\mathbf{x}\|+\|\mathbf{y}\|$ for the norm induced by any inner product — so every inner product really does produce a norm (and a metric).

1

Expand the squared length of the sum (same identity as before, with a plus sign):

\|\mathbf{x}+\mathbf{y}\|^2=\|\mathbf{x}\|^2+2\langle\mathbf{x},\mathbf{y}\rangle+\|\mathbf{y}\|^2

Bilinearity opens both slots; symmetry merges the two cross terms into

2\langle\mathbf{x},\mathbf{y}\rangle

.

2

Bound the cross term with Cauchy–Schwarz,

\langle\mathbf{x},\mathbf{y}\rangle\le|\langle\mathbf{x},\mathbf{y}\rangle|\le\|\mathbf{x}\|\|\mathbf{y}\|

:

\|\mathbf{x}+\mathbf{y}\|^2\le\|\mathbf{x}\|^2+2\|\mathbf{x}\|\|\mathbf{y}\|+\|\mathbf{y}\|^2=(\|\mathbf{x}\|+\|\mathbf{y}\|)^2

Replacing a term by something at least as large can only make the right side bigger; the result is a perfect square.

3

Both sides are non-negative, so taking square roots keeps the inequality:

\|\mathbf{x}+\mathbf{y}\|\le\|\mathbf{x}\|+\|\mathbf{y}\|

. ∎ For distances, apply this to

\mathbf{x}-\mathbf{y}=(\mathbf{x}-\mathbf{z})+(\mathbf{z}-\mathbf{y})

and promise ④ of a metric drops out.

Pause & predict

Why do we need Cauchy–Schwarz before we can talk about "the angle between x and y"?

Pause & predict

When is the shadow exactly as long as the stick, so $|\langle\mathbf{x},\mathbf{y}\rangle|=\|\mathbf{x}\|\,\|\mathbf{y}\|$ ?

In one sentence: length and distance come free with every inner product, and the score can never beat the product of the lengths, just as a shadow can never beat the stick.

7

Angles — the similarity dial

Imagine this

Two people type into a search box. One types "cheap flights to Goa". The other writes a long paragraph about cheap flights to Goa. Their word counts are very different. Their topic is the same.

We want a score that ignores how much someone says and looks only at which way they point. That score is the angle.

Divide the agreement score by the two lengths. Cauchy–Schwarz promises the answer lies between $-1$ and $+1$ , so it is the cosine of exactly one angle $\omega$ between 0° and 180°:

\cos\omega=\frac{\langle\mathbf{x},\mathbf{y}\rangle}{\|\mathbf{x}\|\,\|\mathbf{y}\|}

This is cosine similarity, machine learning's favourite similarity dial:

+1pointing the same way (angle 0°);
0unrelated, at a right angle (90°). We say the two arrows are orthogonal and write $\mathbf{x}\perp\mathbf{y}$ . This happens exactly when $\langle\mathbf{x},\mathbf{y}\rangle=0$ ;
−1pointing opposite ways (180°).

Stretching an arrow does not change where it points, so it does not change the cosine. The short query and the long paragraph get a cosine near +1.

Try: move only a length slider. The beads on the unit ring do not move, and cos ω does not change. Then press ▶ and watch y swing a full turn.

drag the picture to orbit

x angle18°

x length2.2

y angle74°

y length1.6

⟨x,y⟩ = — angle ω = — distance d(x,y) = —

—

cos ω is the shadow of y's bead on x's line: +1 same way, 0 perpendicular, −1 opposite. Lengths never change it — only directions do.

The realization

$\cos\omega$ is the shadow of a length-1 arrow. Shrink both arrows to length 1, then drop the shadow of one onto the other: its length is $\cos\omega$ . Full shadow: same direction. No shadow: right angle. Negative shadow: opposite. Length never enters. Only direction does.

If you want the algebra · why there is always exactly one angle

Prove it · why the angle is well-defined — derived

Claim. For non-zero $\mathbf{x},\mathbf{y}$ there is exactly one $\omega\in[0,\pi]$ with $\cos\omega=\dfrac{\langle\mathbf{x},\mathbf{y}\rangle}{\|\mathbf{x}\|\|\mathbf{y}\|}$ .

1

Divide Cauchy–Schwarz,

|\langle\mathbf{x},\mathbf{y}\rangle|\le\|\mathbf{x}\|\|\mathbf{y}\|

, by the positive number

\|\mathbf{x}\|\|\mathbf{y}\|

:

-1\;\le\;\frac{\langle\mathbf{x},\mathbf{y}\rangle}{\|\mathbf{x}\|\|\mathbf{y}\|}\;\le\;1

Both vectors are non-zero, so both norms are

>0

(axiom ③) and dividing keeps the inequality's direction.

2

On

[0,\pi]

, cosine decreases strictly from

1

to

-1

and takes every value in between exactly once. So the ratio in step 1 is

\cos\omega

for one and only one

\omega\in[0,\pi]

:

\omega=\arccos\big(\langle\mathbf{x},\mathbf{y}\rangle\,/\,\|\mathbf{x}\|\|\mathbf{y}\|\big)

. ∎ Without step 1 the formula could ask for

\arccos(1.7)

, which does not exist. Cauchy–Schwarz is the licence for the protractor.

Pause & predict

Two word-embedding arrows move closer and closer together. What happens to the distance and to the agreement score?

In one sentence: cosine similarity is the agreement score with the lengths divided out, so it measures direction only: +1 same, 0 unrelated, −1 opposite.

8

The high-dimension miracle

Imagine this

Two strangers each answer 1000 yes/no questions completely at random. How much will their answers agree?

On a few questions they will match by luck. On a few they will clash. Over 1000 questions the lucky matches and clashes almost perfectly cancel out. Their agreement score ends up very close to zero.

The slides end one page with a dare: "Pick two random arrows in high dimensions. What happens to the angle between them? Write a small program to see."

The program is below. It picks 300 random pairs of arrows (each number drawn at random), works out the cosine of each pair, and piles the answers into a bar chart. Drag the number of dimensions $n$ and watch.

Try: press ▶ and watch the pile squeeze into a needle at 0. At every stop, compare the measured spread with the predicted 1/√n.

dimension n2

measured spread (std of cos ω) = — predicted 1/√n = —

—

Pick two random arrows. In 2 dimensions any angle is common. In 1000 dimensions they are almost always nearly perpendicular — cos ω lands within about 1/√n of 0, so there is a huge amount of "perpendicular room" for unrelated things.

The realization

In 2D, two random arrows can point any which way. In 1000D, two random arrows are almost always at almost exactly 90°. The spread of the cosine shrinks like $1/\sqrt{n}$ : about 0.03 when $n=1000$ .

Why this matters: a 1000-number embedding space has room for millions of ideas that barely get in each other's way. It also means a cosine of 0.3 between two real embeddings is huge. Random chance would give about 0.

Pause & predict

You pick two random arrows with 1000 numbers each. The angle between them is almost certainly…

In one sentence: in many dimensions, random arrows are almost always at right angles, so real similarity stands out clearly.

9

Orthogonality is in the eye of the ruler

Imagine this

Draw a perfect square corner on a rubber sheet. Now stretch the sheet sideways. The corner is no longer square. It has opened up.

Nothing was redrawn. You only changed the sheet, that is, the way distances are measured. The same thing happens when you change the ruler.

Take $\mathbf{x}=(1,1)$ and $\mathbf{y}=(-1,1)$ . With the ordinary dot product, $(1)(-1)+(1)(1)=0$ . They are at a perfect right angle.

Now use the ruler $A=\begin{pmatrix}2&0\\0&1\end{pmatrix}$ , which counts the first direction double. Predict first:

Pause & predict

$\mathbf{x}=(1,1)$ and $\mathbf{y}=(-1,1)$ are at a right angle under the dot product. Under the ruler $A=\operatorname{diag}(2,1)$ , they are…

The numbers, step by step:

\langle\mathbf{x},\mathbf{y}\rangle_A=2x_1y_1+x_2y_2=2(1)(-1)+(1)(1)=-1

\|\mathbf{x}\|_A=\sqrt{2+1}=\sqrt3,\qquad\|\mathbf{y}\|_A=\sqrt3,\qquad\cos\omega=\frac{-1}{\sqrt3\cdot\sqrt3}=-\frac13

So the angle is about $109.5^\circ$ . The arrows did not move. Only the ruler changed. Here is the rubber-sheet picture, live:

Try: drag a. The floor stretches sideways like a photo, the two arrows tilt apart, and the little square corner disappears. Same arrows, new ruler, new angle.

drag the picture to orbit

weight a1.0

⟨x,y⟩_A = 0 cos ω = 0 ω = 90°

—

The ruler A = diag(a, 1) counts the first direction a times more. That is the same as stretching the floor by √a sideways and measuring normally.

The realization

"At a right angle" is not a fact about two arrows alone. It is a fact about two arrows and a ruler. Measuring with $A=\operatorname{diag}(2,1)$ is exactly the same as stretching the page sideways by $\sqrt2$ and then measuring normally. Choosing the ruler to fit the problem is what feature weighting and the Mahalanobis distance do in machine learning. (Section 5's lens shows the same thing: press diag(2, 1).)

In one sentence: change the ruler and the same two arrows can stop being at a right angle.

10

Orthonormal matrices — the motions that don't lie

Imagine this

Turn a photo on your phone. The faces in it keep their shapes. Nothing gets longer, nothing gets squashed. Now stretch the photo with two fingers. The faces get wider and the angles in them change.

Turning is an honest motion: every length and angle survives. Stretching or slanting is a lying motion: it moves things and changes their measurements.

Why machine learning cares: turn an embedding space and every similarity, every distance and every nearest neighbour stays exactly the same. So the honest motions are safe to use anywhere.

Which matrices are honest? The ones whose columns are at right angles to each other and each of length 1. They are called orthogonal (or orthonormal) matrices, and they satisfy one neat rule:

A^TA=I\qquad\text{so}\qquad A^{-1}=A^T

In words: to undo the motion, just flip the matrix (transpose it). No hard work needed. A turn by an angle $\theta$ , $\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix}$ , is one example. So is a mirror flip.

Why does $A^TA=I$ keep lengths? Because the new length squared is $\|A\mathbf{x}\|^2=\mathbf{x}^TA^TA\mathbf{x}=\mathbf{x}^TI\mathbf{x}=\|\mathbf{x}\|^2$ . The $A^TA$ in the middle just disappears. The same happens inside the angle formula, so angles survive too.

A slant (a shear) $\begin{pmatrix}1&k\\0&1\end{pmatrix}$ pushes every point sideways by $k$ times its height. Flat arrows stay put and everything else leans over. Watch both motions:

Try: press ▶ on Rotate — the tile spins and the chips never change. Then press Shear and watch the same tile smear and the chips drift.

drag the picture to orbit

turn θ35°

‖u‖: — angle(u,v): — AᵀA = I ✓

—

The two rings on the floor are the ruler: each arrow tip must stay on its ring if its length is honest. Stiff tile — tips ride the rings. Rubber tile — tips slide off.

The realization

$A^TA=I$

This one line is the test for an honest motion. Pass it, and every length, distance and angle is safe. The shear keeps the area (its determinant is 1), but it fails this test, and you can see its arrows slide off the measuring rings.

If you want the algebra · honest motions keep every length and angle

Prove it · orthonormal maps keep every length and every angle — derived

Claim. If $A^TA=I$ , then $\|A\mathbf{x}\|=\|\mathbf{x}\|$ for every $\mathbf{x}$ , and the angle between $A\mathbf{x}$ and $A\mathbf{y}$ equals the angle between $\mathbf{x}$ and $\mathbf{y}$ .

1

Write the dot product of two transformed vectors as a matrix sandwich and use

(A\mathbf{x})^T=\mathbf{x}^TA^T

:

\langle A\mathbf{x},A\mathbf{y}\rangle=(A\mathbf{x})^T(A\mathbf{y})=\mathbf{x}^T\!\left(A^TA\right)\mathbf{y}

Transpose of a product reverses the order (Unit 1); matrix multiplication is associative, so we may group the middle.

2

Replace

A^TA

by

I

:

\langle A\mathbf{x},A\mathbf{y}\rangle=\mathbf{x}^T\mathbf{y}=\langle\mathbf{x},\mathbf{y}\rangle

This is the whole cancellation. Inner products are untouched — every measurement built from them must be untouched too.

3

Lengths: put

\mathbf{y}=\mathbf{x}

in step 2:

\|A\mathbf{x}\|^2=\langle A\mathbf{x},A\mathbf{x}\rangle=\langle\mathbf{x},\mathbf{x}\rangle=\|\mathbf{x}\|^2

. Squared length is a special case of the inner product, so it is preserved; take square roots.

4

Angles: every factor in the cosine formula is preserved, so the ratio is too:

\cos\omega'=\frac{\langle A\mathbf{x},A\mathbf{y}\rangle}{\|A\mathbf{x}\|\,\|A\mathbf{y}\|}=\frac{\langle\mathbf{x},\mathbf{y}\rangle}{\|\mathbf{x}\|\,\|\mathbf{y}\|}=\cos\omega

Same cosine on

[0,\pi]

means the same angle. ∎ Note what the proof needs: only

A^TA=I

. A shear has

\det=1

yet

A^TA\neq I

— and the widget shows it bending both gauges.

Side note: if the columns are orthonormal, so are the rows. For square matrices a left inverse is also a right inverse: if $BA=I=AC$ then $B=B(AC)=(BA)C=C$ .

Pause & predict

Why does an orthonormal $A$ keep every length the same?

In one sentence: a matrix with $A^TA=I$ turns or flips space like a stiff tile, so every length and angle survives.

11

The shadow machine — projection

Imagine this

A streetlamp shines straight down on a road. You hold a stick $\mathbf{a}$ at a slant. The road runs in the direction $\mathbf{b}$ .

The stick's shadow on the road is the part of the stick that goes along the road. The rest of the stick points straight up, at a right angle to the road. So the stick = shadow + leftover.

That split is called projection. How long is the shadow? The shadow is some multiple of the road direction: $\alpha\mathbf{b}$ . We want the leftover $\mathbf{a}-\alpha\mathbf{b}$ to stand at a right angle to the road. That one wish fixes $\alpha$ :

\alpha=\frac{\langle\mathbf{a},\mathbf{b}\rangle}{\langle\mathbf{b},\mathbf{b}\rangle},\qquad \text{shadow}=\operatorname{proj}_{\mathbf{b}}(\mathbf{a})=\alpha\,\mathbf{b}

Quick example: $\mathbf{a}=(2,3)$ and the road $\mathbf{b}=(1,0)$ . Then $\alpha=\frac{2}{1}=2$ , the shadow is $(2,0)$ and the leftover is $(0,3)$ , pointing straight up. If $\mathbf{b}$ has length 1, it is even simpler: $\alpha=\langle\mathbf{a},\mathbf{b}\rangle$ .

Try: swing the stick past 90° from the road and watch α turn negative — the shadow falls behind the lamp post; then turn the road and see the split recomputed.

drag the picture to orbit

stick a: θ64°

stick a: ‖a‖2.4

stick a: lift φ28°

road b: θ12°

shadow coeff α = — ⟨a − αb, b⟩ = 0

—

Orange on the road = the shadow (α·b); the grey arrow is the road's unit direction b. Violet = the leftover, at a right angle to the road — the small square certifies it. This one picture is least squares, attention scores, and PCA's first move, all waiting to happen.

The realization

Any arrow = its shadow along a direction + a leftover at a right angle. The shadow amount is $\alpha=\langle\mathbf{a},\mathbf{b}\rangle/\langle\mathbf{b},\mathbf{b}\rangle$ . If the stick leans backwards (angle past 90°), $\alpha$ turns negative and the shadow falls behind you. This single split is the heart of least squares, attention scores and PCA.

If you want the algebra · where the shadow formula comes from

Prove it · the shadow coefficient — derived

Claim. The multiple of $\mathbf{b}$ whose leftover is orthogonal to $\mathbf{b}$ is $\alpha=\dfrac{\langle\mathbf{a},\mathbf{b}\rangle}{\langle\mathbf{b},\mathbf{b}\rangle}$ , so $\operatorname{proj}_{\mathbf{b}}(\mathbf{a})=\alpha\mathbf{b}$ .

1

State the demand as an equation:

\langle\mathbf{a}-\alpha\mathbf{b},\,\mathbf{b}\rangle=0

"The leftover is perpendicular to the road" — one unknown,

\alpha

, one equation.

2

Open the left slot with bilinearity:

\langle\mathbf{a},\mathbf{b}\rangle-\alpha\langle\mathbf{b},\mathbf{b}\rangle=0

Linearity in the first argument lets the subtraction and the scalar

\alpha

pass through.

3

Solve for

\alpha

(allowed because

\mathbf{b}\neq\mathbf{0}

makes

\langle\mathbf{b},\mathbf{b}\rangle>0

):

\alpha=\dfrac{\langle\mathbf{a},\mathbf{b}\rangle}{\langle\mathbf{b},\mathbf{b}\rangle}

, hence

\operatorname{proj}_{\mathbf{b}}(\mathbf{a})=\dfrac{\langle\mathbf{a},\mathbf{b}\rangle}{\langle\mathbf{b},\mathbf{b}\rangle}\,\mathbf{b}

and

\mathbf{a}-\operatorname{proj}_{\mathbf{b}}(\mathbf{a})\perp\mathbf{b}

. ∎ If

\mathbf{b}

is a unit vector the denominator is

1

and

\alpha

is just

\langle\mathbf{a},\mathbf{b}\rangle

. You met this

\alpha

already — it is the "smartest choice" inside the Cauchy–Schwarz proof. Same coefficient, same geometry.

In one sentence: to find a shadow, ask for the leftover to be at a right angle, and the amount comes out as $\langle\mathbf{a},\mathbf{b}\rangle/\langle\mathbf{b},\mathbf{b}\rangle$ .

12

Gram–Schmidt — cleaning a messy basis

Imagine this

In a city with square blocks, directions are easy: "3 blocks east, 2 blocks north." Now imagine a city whose streets meet at odd angles. Every direction turns into a puzzle.

Square streets make life easy. Gram–Schmidt is a machine that turns any crooked set of streets into square ones.

Unit 2 taught you to find a basis: a set of directions that can reach every point. The best kind is an orthonormal basis: every direction at a right angle to every other, and each of length 1.

Why is it the best? Because finding coordinates becomes easy. With a crooked basis, you must solve equations to find how much of each direction you need. With an orthonormal basis, you just take shadows: the amount of $\mathbf{e}_i$ in $\mathbf{v}$ is $\langle\mathbf{v},\mathbf{e}_i\rangle$ . No equations.

Coordinates by shadow. With the orthonormal pair $\mathbf{e}_1=(\cos30^\circ,\sin30^\circ)$ , $\mathbf{e}_2=(-\sin30^\circ,\cos30^\circ)$ , the vector $\mathbf{v}\approx(1.13,2.04)$ has coordinates $\lambda_1=\langle\mathbf{v},\mathbf{e}_1\rangle=2$ and $\lambda_2=\langle\mathbf{v},\mathbf{e}_2\rangle=1.2$ : each coordinate is just the shadow of $\mathbf{v}$ on that axis. In a non-orthonormal basis you would have to solve a $2\times2$ system instead.

If you want the algebra · why coordinates are just shadows

Prove it · coordinates in an orthonormal basis are inner products — derived

Claim. If $\{\mathbf{e}_1,\dots,\mathbf{e}_n\}$ is orthonormal and $\mathbf{v}=\lambda_1\mathbf{e}_1+\cdots+\lambda_n\mathbf{e}_n$ , then $\lambda_i=\langle\mathbf{v},\mathbf{e}_i\rangle$ .

1

Take the inner product of both sides of the expansion with one basis vector

\mathbf{e}_i

:

\langle\mathbf{v},\mathbf{e}_i\rangle=\Big\langle\sum_{j=1}^{n}\lambda_j\mathbf{e}_j,\ \mathbf{e}_i\Big\rangle=\sum_{j=1}^{n}\lambda_j\langle\mathbf{e}_j,\mathbf{e}_i\rangle

Bilinearity pulls the sum and the scalars

\lambda_j

out of the first slot.

2

Orthonormality kills every term but one:

\langle\mathbf{e}_j,\mathbf{e}_i\rangle=0

for

j\neq i

and

=1

for

j=i

. So

\langle\mathbf{v},\mathbf{e}_i\rangle=\lambda_i\cdot1=\lambda_i

. ∎ The "1" is where unit length earns its keep; with a merely orthogonal basis you would get

\lambda_i\|\mathbf{e}_i\|^2

and have to divide. The canonical basis of

\mathbb{R}^n

is one orthonormal basis; rotate it and you get infinitely many others.

The cleaning machine is the shadow machine used over and over. Take your crooked directions one at a time:

1Take the first direction. Shrink or stretch it to length 1. Done: that is $\mathbf{e}_1$ .
2Take the next direction. Cut off its shadow on every clean direction you already have. What is left stands at a right angle to all of them.
3Stretch that leftover to length 1. That is your next clean direction. Repeat step 2 for the rest.

For two directions it is just: $\mathbf{e}_1=\dfrac{\mathbf{x}_1}{\|\mathbf{x}_1\|}$ , then $\mathbf{u}_2=\mathbf{x}_2-\langle\mathbf{x}_2,\mathbf{e}_1\rangle\,\mathbf{e}_1$ , then $\mathbf{e}_2=\dfrac{\mathbf{u}_2}{\|\mathbf{u}_2\|}$ . Watch it happen in 3D:

Try: press ▶ and watch the leftover lift off the shadow; then step to 3 and orbit until you look straight down the e₁ axis — the green right-angle marker should open into a true L.

Step 0/4

drag the picture to orbit

—

The faint floor is the plane z = 0 (x₁ lies in it); the tinted sheet is the plane the two vectors span — Gram–Schmidt never leaves it. The final frame shows the receipt: $\langle\mathbf{e}_1,\mathbf{e}_2\rangle=-\tfrac16+\tfrac16+0=0$ and both lengths exactly 1 — a perfect grid for that plane, built from a lopsided pair that started $45^\circ$ apart.

The realization

Gram–Schmidt = cut off the shadows, then resize to 1. Each new direction loses the parts that point along the clean ones, so what is left must be at a right angle to them. Any basis at all can be cleaned this way.

Bonus · the slides' shortcut: Gram–Schmidt by elimination (and why it works)

The course adds a beautiful industrial route. Stack the messy basis as columns of $A$ , then run plain Gaussian elimination on the augmented matrix $[A^TA \mid A^T]$ — when the left half reaches echelon form, the rows on the right are an orthogonal basis. The slides' example: $\mathbf{v}_1=(3,1)$ , $\mathbf{v}_2=(2,2)$ give $A^TA=\begin{pmatrix}10&8\\8&8\end{pmatrix}$ , and elimination lands on

\left(\begin{array}{cc|cc}1&0.8&0.3&0.1\\0&1&-0.25&0.75\end{array}\right)

Check the right-hand rows: $(0.3,0.1)\cdot(-0.25,0.75)=-0.075+0.075=0$ — orthogonal (the first is $\mathbf{v}_1$ rescaled; the second is what's left of $\mathbf{v}_2$ after removing its $\mathbf{v}_1$ -shadow). Normalize and you have an orthonormal basis. Why it works, in three moves: ① full column rank makes $A^TA$ positive-definite ( $A\mathbf{x}=\mathbf{0}\iff A^TA\mathbf{x}=\mathbf{0}$ , so the blind spots match and are trivial) — elimination needs no row swaps; ② elimination is multiplication by elementary matrices, i.e. $A^TA=LU$ with $L$ lower-triangular (Unit 1's recorded elimination!), and the right half becomes $Q^T=L^{-1}A^T$ ; ③ then $Q^TQ=L^{-1}(A^TA)L^{-T}=UL^{-T}$ is upper-triangular and symmetric — hence diagonal — which is precisely the statement that $Q$ 's columns are orthogonal. Elimination doesn't just solve systems; run on $A^TA$ , it quietly performs Gram–Schmidt.

Same geometry, industrial route. $q_1=0.1\,\mathbf{v}_1$ and $q_2=0.625\times(-0.4,1.2)$ — the second row is precisely the leftover after removing $\mathbf{v}_2$ 's shadow $\tfrac{8}{10}\mathbf{v}_1=(2.4,0.8)$ , rescaled. Elimination found the shadow coefficient $0.8$ on its own: it is the multiplier it used to clear the first column.

Pause & predict

In Gram–Schmidt, why does cutting off the shadow $\langle\mathbf{x}_2,\mathbf{e}_1\rangle\mathbf{e}_1$ from $\mathbf{x}_2$ work?

In one sentence: Gram–Schmidt turns any crooked basis into a square one by cutting off shadows and resizing to length 1.

13

The whole story, in five lines

①Length is a choice. A norm is any length rule that scales, allows no shortcuts, and is zero only for "nowhere". The crow (round ball) and the taxi (diamond) both qualify.
②The dot product is an agreement score. Its blueprint (fair with sums, same both ways, positive with itself) defines an inner product. Every SPD matrix $A$ makes one: $\mathbf{x}^TA\mathbf{y}$ . Test SPD with the top-left numbers and determinants.
③Free gifts: length, distance, and (because a shadow never beats the stick) an angle: $\cos\omega=\langle\mathbf{x},\mathbf{y}\rangle/\|\mathbf{x}\|\|\mathbf{y}\|$ , ML's similarity dial.
④Right angles depend on the ruler. Honest motions ( $A^TA=I$ ) keep every measurement. Random arrows in high dimensions are almost always at right angles.
⑤Shadows build square grids. Projection splits an arrow into along + across. Gram–Schmidt repeats it to turn any basis into an orthonormal one.

The one thing to remember

Geometry is a choice. Pick the matrix, and you have picked what "long", "near" and "at a right angle" mean. Much of machine learning is the craft of picking that ruler well. And whatever ruler you pick, Gram–Schmidt can always build a perfect square grid for it.

14

Practice arena — the unit's problem set, solved in full

This is Prof. Saurabh's practice set for this unit — Analytic Geometry: eight problems on inner products, positive-definite matrices, norms, angles, orthogonality and projections. Attempt each one on paper before unfolding — the fold-out solution shows every step of the computation, nothing skipped.

Problem 1Easy

On $\mathbb{R}^3$ with the standard (Euclidean) inner product, let $\mathbf{u} = (1,1,0)^\top$ and $\mathbf{v} = (0,1,1)^\top$ .
(a) Find all vectors $\mathbf{w}\in\mathbb{R}^3$ orthogonal to both $\mathbf{u}$ and $\mathbf{v}$ , and write down the unit vectors of this form. [2 M]
(b) Compute the angle $\omega$ between $\mathbf{u}$ and $\mathbf{v}$ . [1 M]
(c) Show that $\{\mathbf{u},\mathbf{v},\mathbf{w}\}$ is a basis of $\mathbb{R}^3$ , and explain geometrically why $\mathbf{w}$ is perpendicular to the plane spanned by $\mathbf{u}$ and $\mathbf{v}$ . [1 M] (4 marks)

Show the worked solution — every step

Step 1 · Turn "orthogonal to both" into equations (a). Idea: "orthogonal to both" means two dot-product equations. Write

\mathbf{w} = (w_1, w_2, w_3)^\top

:

\mathbf{w}^\top\mathbf{u} = w_1 + w_2 = 0, \qquad \mathbf{w}^\top\mathbf{v} = w_2 + w_3 = 0.

Step 2 · Solve the little system. From these,

w_1 = -w_2

and

w_3 = -w_2

. Letting the free variable be

w_2 = t

,

\mathbf{w} = t\,(-1,\,1,\,-1)^\top, \qquad t\in\mathbb{R}.

Step 3 · Normalise for the unit vectors. Divide by the length

\|(-1,1,-1)^\top\| = \sqrt{(-1)^2 + 1^2 + (-1)^2} = \sqrt{3}

:

\mathbf{w} = \pm\tfrac{1}{\sqrt{3}}\,(-1,\,1,\,-1)^\top.

Both signs: the orthogonal line carries exactly two unit vectors.

Step 4 · The angle (b). Use

\cos\omega = \dfrac{\mathbf{u}^\top\mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}

. Here

\mathbf{u}^\top\mathbf{v} = 0 + 1 + 0 = 1

and

\|\mathbf{u}\| = \|\mathbf{v}\| = \sqrt{2}

, so

\cos\omega = \frac{1}{\sqrt{2}\cdot\sqrt{2}} = \frac{1}{2} \;\Longrightarrow\; \omega = 60^\circ.

Step 5 · Basis test (c). Three vectors in

\mathbb{R}^3

form a basis exactly when their determinant is nonzero. With

\mathbf{w} = (-1,1,-1)^\top

,

\det[\,\mathbf{u}\ \mathbf{v}\ \mathbf{w}\,] = \det\!\begin{bmatrix} 1 & 0 & -1 \\ 1 & 1 & 1 \\ 0 & 1 & -1 \end{bmatrix} = 1\,\big(1\cdot(-1) - 1\cdot 1\big) - 0 + (-1)\,\big(1\cdot 1 - 1\cdot 0\big) = -2 - 1 = -3 \neq 0,

so

\{\mathbf{u},\mathbf{v},\mathbf{w}\}

is a basis.

Step 6 · The geometric picture.

\mathbf{u}

and

\mathbf{v}

span a plane. Since

\mathbf{w}

was constructed to be orthogonal to both

\mathbf{u}

and

\mathbf{v}

, it is orthogonal to every vector in that plane — any such vector is

\alpha\mathbf{u} + \beta\mathbf{v}

, and

\mathbf{w}^\top(\alpha\mathbf{u} + \beta\mathbf{v}) = \alpha\cdot 0 + \beta\cdot 0 = 0

. So

\mathbf{w}

is the normal direction to the plane.

Answer. (a)

\mathbf{w} = t\,(-1,1,-1)^\top

,

t\in\mathbb{R}

; unit vectors

\pm\tfrac{1}{\sqrt{3}}(-1,1,-1)^\top

. (b)

\omega = 60^\circ

. (c)

\det[\,\mathbf{u}\ \mathbf{v}\ \mathbf{w}\,] = -3 \neq 0

, so it is a basis;

\mathbf{w}

is the plane's normal direction.

Watch out

A "unit vectors" answer needs both signs — $+\tfrac{1}{\sqrt3}(-1,1,-1)^\top$ and $-\tfrac{1}{\sqrt3}(-1,1,-1)^\top$ . Writing only one loses half the solution set, and forgetting to divide by $\sqrt{3}$ loses the "unit" part entirely.

Key takeaway

"Orthogonal to both" always translates into a homogeneous linear system — one dot-product equation per constraint. In $\mathbb{R}^3$ , two independent constraints leave a one-dimensional line of solutions: the normal line of the plane.

Wait — really?

You just computed a cross product without naming it: $\mathbf{u}\times\mathbf{v} = (1,-1,1)^\top$ , which is exactly the $t = -1$ member of your family $t\,(-1,1,-1)^\top$ . Solving "orthogonal to both" as a linear system is precisely what the cross product does for a living in $\mathbb{R}^3$ .

Problem 2Easy

(Similarity vs. dissimilarity of data objects.) Three data points live in $\mathbb{R}^3$ : $\mathbf{p}_1 = (2,1,0)^\top, \qquad \mathbf{p}_2 = (0,1,2)^\top, \qquad \mathbf{p}_3 = (2,1,1)^\top.$ (a) Build the $3\times 3$ dissimilarity matrix $D$ using Euclidean distance. [2 M]
(b) Which two points are the most similar? [1 M]
(c) Compute the cosine similarity between $\mathbf{p}_1$ and $\mathbf{p}_3$ . [1 M] (4 marks)

Show the worked solution — every step

Step 1 · Pairwise distances (a). Dissimilarity between two points is just the distance

d(\mathbf{p}_i,\mathbf{p}_j) = \|\mathbf{p}_i - \mathbf{p}_j\|

. Compute each pair:

d(\mathbf{p}_1,\mathbf{p}_2) = \|(2,0,-2)^\top\| = \sqrt{4+0+4} = \sqrt{8} = 2\sqrt{2}, \qquad d(\mathbf{p}_1,\mathbf{p}_3) = \|(0,0,-1)^\top\| = 1,

d(\mathbf{p}_2,\mathbf{p}_3) = \|(-2,0,1)^\top\| = \sqrt{4+0+1} = \sqrt{5}.

Step 2 · Assemble the matrix. Place these in a symmetric matrix with zeros on the diagonal (a point's distance to itself is

0

):

D = \begin{bmatrix} 0 & 2\sqrt{2} & 1 \\ 2\sqrt{2} & 0 & \sqrt{5} \\ 1 & \sqrt{5} & 0 \end{bmatrix}.

Step 3 · Most similar pair (b). "Most similar"

=

smallest dissimilarity. The smallest off-diagonal entry is

d(\mathbf{p}_1,\mathbf{p}_3) = 1

, so

\mathbf{p}_1

and

\mathbf{p}_3

are the most similar.

Step 4 · Cosine similarity (c). Cosine similarity is

\cos\theta = \dfrac{\mathbf{p}_1^\top\mathbf{p}_3}{\|\mathbf{p}_1\|\,\|\mathbf{p}_3\|}

:

\mathbf{p}_1^\top\mathbf{p}_3 = 2\cdot 2 + 1\cdot 1 + 0\cdot 1 = 5, \qquad \|\mathbf{p}_1\| = \sqrt{5}, \qquad \|\mathbf{p}_3\| = \sqrt{6},

\cos\theta = \frac{5}{\sqrt{5}\,\sqrt{6}} = \frac{5}{\sqrt{30}} = \frac{\sqrt{30}}{6} \approx 0.913.

(Close to

1

, confirming

\mathbf{p}_1

and

\mathbf{p}_3

point in nearly the same direction.)

Answer. (a)

D

as above, symmetric with zero diagonal. (b)

\mathbf{p}_1

and

\mathbf{p}_3

, at distance

1

. (c)

\cos\theta = 5/\sqrt{30} = \sqrt{30}/6 \approx 0.913

.

Watch out

In a dissimilarity matrix, small means similar — scanning for the largest entry answers the opposite question. And the diagonal zeros are not data: every point is at distance $0$ from itself, so only off-diagonal entries compete.

Key takeaway

Two different rulers for "similar": Euclidean distance measures closeness in position, cosine similarity measures alignment of direction. Here they happen to agree ( $\mathbf{p}_1,\mathbf{p}_3$ win on both); in general they need not.

Wait — really?

Cosine similarity is blind to length: $(2,1,0)^\top$ and $(4,2,0)^\top$ have cosine similarity exactly $1$ even though they sit $\sqrt{5}$ apart. That is why it is the default in text retrieval — a tweet and an essay about the same topic point the same way; they just differ in length.

Problem 3Medium

Let $A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$ and define a new inner product on $\mathbb{R}^2$ by $\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top A\,\mathbf{y}$ .
(a) Show $A$ is symmetric positive definite, so $\langle\cdot,\cdot\rangle_A$ is a genuine inner product. [1 M]
(b) The vectors $\mathbf{x} = (1,0)^\top$ and $\mathbf{y} = (0,1)^\top$ are orthogonal in the dot product. Are they still orthogonal under $\langle\cdot,\cdot\rangle_A$ ? Find the angle between them in this new inner product. [2 M]
(c) Find a vector $\mathbf{y}' = (a,1)^\top$ that is orthogonal to $\mathbf{x}$ with respect to $\langle\cdot,\cdot\rangle_A$ . [1 M] (4 marks)

Show the worked solution — every step

Step 1 · Certify the inner product (a).

A

is symmetric since

A^\top = A

. Sylvester's criterion (all leading minors positive):

D_1 = 2 > 0, \qquad D_2 = \det A = 2\cdot 2 - 1\cdot 1 = 3 > 0.

So

A

is positive definite. A symmetric, positive-definite matrix always makes

\mathbf{x}^\top A\,\mathbf{y}

a valid inner product: it is bilinear, symmetric, and

\mathbf{x}^\top A\,\mathbf{x} > 0

for

\mathbf{x} \neq \mathbf{0}

.

Step 2 · Test orthogonality (b). Compute

\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top A\,\mathbf{y}

. First

A\mathbf{y} = A(0,1)^\top = (1,2)^\top

, hence

\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top(A\mathbf{y}) = (1,0)\cdot(1,2) = 1 \neq 0.

They are not orthogonal here.

Step 3 · The $A$ -norms. To get the angle we also need

\|\mathbf{x}\|_A = \sqrt{\mathbf{x}^\top A\,\mathbf{x}} = \sqrt{A_{11}} = \sqrt{2}, \qquad \|\mathbf{y}\|_A = \sqrt{A_{22}} = \sqrt{2}.

Step 4 · The $A$ -angle.

\cos\theta = \frac{\langle\mathbf{x},\mathbf{y}\rangle_A}{\|\mathbf{x}\|_A\,\|\mathbf{y}\|_A} = \frac{1}{\sqrt{2}\cdot\sqrt{2}} = \frac{1}{2} \;\Longrightarrow\; \theta = 60^\circ.

Takeaway (from the source): orthogonality depends on which inner product you use.

Step 5 · Force orthogonality (c). We need

\langle\mathbf{x},\mathbf{y}'\rangle_A = 0

. Note

\mathbf{x}^\top A = (1,0)\,A = (2,1)

, so

\langle\mathbf{x},\mathbf{y}'\rangle_A = (2,1)\cdot(a,1) = 2a + 1 = 0 \;\Longrightarrow\; a = -\tfrac{1}{2}.

Thus

\mathbf{y}' = \left(-\tfrac{1}{2},\,1\right)^\top

.

Answer. (a) SPD by Sylvester:

D_1 = 2 > 0

,

D_2 = 3 > 0

. (b) No —

\langle\mathbf{x},\mathbf{y}\rangle_A = 1 \neq 0

; the angle in the

A

-geometry is

\theta = 60^\circ

. (c)

a = -\tfrac12

, i.e.

\mathbf{y}' = (-\tfrac12,\,1)^\top

.

Watch out

When you switch inner products, switch everything: the cosine formula needs the $A$ -inner product on top and the $A$ -norms below. Mixing $\langle\mathbf{x},\mathbf{y}\rangle_A = 1$ with the Euclidean norms $\|\mathbf{x}\| = \|\mathbf{y}\| = 1$ would give $\cos\theta = 1$ , i.e. a $0^\circ$ angle — badly wrong.

Key takeaway

Orthogonality is relative to the inner product. The same pair of vectors can be perpendicular in one geometry and at $60^\circ$ in another — and part (c) shows you can always re-engineer a vector to restore orthogonality in the new geometry.

Wait — really?

The coordinate axes themselves — $\mathbf{e}_1$ and $\mathbf{e}_2$ , the gridlines of the plane — meet at $60^\circ$ , not $90^\circ$ , the moment $A$ defines the geometry. Perpendicularity is not a property of two arrows; it is a property of the inner product that measures them.

Problem 4Medium

On $\mathbb{R}^3$ use the weighted inner product $\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top A\,\mathbf{y}$ with $A = \begin{bmatrix} 2 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 1 \end{bmatrix} \quad(\text{SPD}).$ Let $\mathbf{a} = (1,-1,0)^\top$ and $\mathbf{b} = (1,0,2)^\top$ .
(a) Compute $\langle\mathbf{b},\mathbf{a}\rangle_A$ , $\|\mathbf{a}\|_A$ , and $\|\mathbf{b}\|_A$ . [2 M]
(b) Find the orthogonal projection $\mathbf{p}$ of $\mathbf{b}$ onto the line spanned by $\mathbf{a}$ , in this inner product. [2 M]
(c) Verify that the residual $\mathbf{b} - \mathbf{p}$ is $A$ -orthogonal to $\mathbf{a}$ (this is what makes $\mathbf{p}$ the orthogonal projection). [1 M] (5 marks)

Show the worked solution — every step

Step 1 · One matrix–vector product does most of the work (a). Trick: compute the matrix–vector product once, then take dot products. With

A\mathbf{a} = A(1,-1,0)^\top = (2-1,\ 1-2,\ 0)^\top = (1,-1,0)^\top:

\langle\mathbf{b},\mathbf{a}\rangle_A = \mathbf{b}^\top(A\mathbf{a}) = (1,0,2)\cdot(1,-1,0) = 1, \qquad \|\mathbf{a}\|_A^2 = \mathbf{a}^\top(A\mathbf{a}) = (1,-1,0)\cdot(1,-1,0) = 2.

Step 2 · The norm of $\mathbf{b}$ . For

\|\mathbf{b}\|_A

, use

A\mathbf{b} = A(1,0,2)^\top = (2,1,2)^\top

:

\|\mathbf{b}\|_A^2 = \mathbf{b}^\top(A\mathbf{b}) = (1,0,2)\cdot(2,1,2) = 2 + 0 + 4 = 6.

So

\langle\mathbf{b},\mathbf{a}\rangle_A = 1

,

\|\mathbf{a}\|_A = \sqrt{2}

,

\|\mathbf{b}\|_A = \sqrt{6}

.

Step 3 · Project (b). Projection formula (same as usual, but every inner product is the

A

-one):

\mathbf{p} = \frac{\langle\mathbf{b},\mathbf{a}\rangle_A}{\langle\mathbf{a},\mathbf{a}\rangle_A}\,\mathbf{a} = \frac{1}{2}\,(1,-1,0)^\top = \left(\tfrac{1}{2},\,-\tfrac{1}{2},\,0\right)^\top.

Step 4 · Check the residual (c). The residual is

\mathbf{r} = \mathbf{b} - \mathbf{p} = (1,0,2)^\top - \left(\tfrac{1}{2},\,-\tfrac{1}{2},\,0\right)^\top = \left(\tfrac{1}{2},\ \tfrac{1}{2},\ 2\right)^\top.

Check

A

-orthogonality using the same

A\mathbf{a} = (1,-1,0)^\top

:

\langle\mathbf{r},\mathbf{a}\rangle_A = \mathbf{r}^\top(A\mathbf{a}) = \left(\tfrac{1}{2},\tfrac{1}{2},2\right)\cdot(1,-1,0) = \tfrac{1}{2} - \tfrac{1}{2} + 0 = 0. \;\checkmark

Since

\mathbf{r} \perp_A \mathbf{a}

,

\mathbf{p}

is indeed the orthogonal projection of

\mathbf{b}

onto the line through

\mathbf{a}

.

Answer. (a)

\langle\mathbf{b},\mathbf{a}\rangle_A = 1

,

\|\mathbf{a}\|_A = \sqrt{2}

,

\|\mathbf{b}\|_A = \sqrt{6}

. (b)

\mathbf{p} = \left(\tfrac12, -\tfrac12, 0\right)^\top

. (c)

\langle\mathbf{b}-\mathbf{p},\,\mathbf{a}\rangle_A = 0

✓ — the residual is

A

-orthogonal to

\mathbf{a}

.

Watch out

Verify with the same inner product you projected in: the defining property is $\langle\mathbf{b}-\mathbf{p},\,\mathbf{a}\rangle_A = 0$ . In this particular problem a plain dot-product check happens to give $0$ too — pure luck, because $A\mathbf{a} = \mathbf{a}$ here (see below). In a general weighted geometry the residual is $A$ -orthogonal, not dot-orthogonal, to $\mathbf{a}$ .

Key takeaway

The projection recipe never changes: $\mathbf{p} = \dfrac{\langle\mathbf{b},\mathbf{a}\rangle}{\langle\mathbf{a},\mathbf{a}\rangle}\,\mathbf{a}$ — swap in whichever inner product defines the geometry. The residual-orthogonality check is the certificate that $\mathbf{p}$ really is the closest point on the line.

Wait — really?

$A\mathbf{a} = \mathbf{a}$ : the vector $\mathbf{a} = (1,-1,0)^\top$ is an eigenvector of $A$ with eigenvalue $1$ . That is why every $A$ -computation involving $\mathbf{a}$ collapsed to a plain dot product — and why the Euclidean projection of $\mathbf{b}$ onto $\mathbf{a}$ would land on the same $\mathbf{p}$ here. Move $\mathbf{a}$ off an eigendirection and the two projections part ways.

Problem 5Medium

Consider the symmetric matrix $A(t) = \begin{bmatrix} t & 3 \\ 3 & t \end{bmatrix}$ , $t\in\mathbb{R}$ .
(a) Classify $A(t)$ for all real $t$ as positive definite, positive semidefinite, indefinite, negative semidefinite, or negative definite. [3 M]
(b) For which $t$ does $\langle\mathbf{x},\mathbf{y}\rangle = \mathbf{x}^\top A(t)\,\mathbf{y}$ define a valid inner product? At the borderline value $t = 3$ , exhibit a nonzero vector $\mathbf{v}$ with $\langle\mathbf{v},\mathbf{v}\rangle = 0$ . [2 M] (5 marks)

Show the worked solution — every step

Looking ahead

This solution uses eigenvalues, which Unit 4 introduces properly: for a symmetric matrix, positive-definite means all eigenvalues are positive. If you have not met them yet, you can classify $A(t)$ with this unit's tools instead — Sylvester's minors $D_1=t$ and $D_2=t^2-9$ give "positive definite $\iff t>3$ " — and read the eigenvalue route as a preview.

Step 1 · Find the eigenvalues (a). Method: a symmetric matrix is classified by the signs of its eigenvalues. Solve

\det(A - \lambda I) = 0

:

(t-\lambda)^2 - 9 = 0 \;\Longrightarrow\; t - \lambda = \pm 3 \;\Longrightarrow\; \lambda = t - 3 \ \text{ or } \ \lambda = t + 3.

Step 2 · Read off the signs. Now just read off the signs of the two eigenvalues

t-3

and

t+3

:

\begin{aligned} t > 3 &: \ \text{both positive} \;\Rightarrow\; \text{positive definite}\\ t = 3 &: \ \lambda = 6,\,0 \;\Rightarrow\; \text{positive semidefinite}\\ -3 < t < 3 &: \ \text{one } +,\ \text{one } - \;\Rightarrow\; \text{indefinite}\\ t = -3 &: \ \lambda = 0,\,-6 \;\Rightarrow\; \text{negative semidefinite}\\ t < -3 &: \ \text{both negative} \;\Rightarrow\; \text{negative definite} \end{aligned}

Step 3 · Inner-product range and the degenerate vector (b). An inner product needs a positive-definite matrix, so we need

t > 3

. At

t = 3

the matrix is only semidefinite, which fails the rule

\langle\mathbf{v},\mathbf{v}\rangle > 0

for

\mathbf{v}\neq\mathbf{0}

. Concretely, take

\mathbf{v} = (1,-1)^\top

. With

A(3) = \begin{bmatrix} 3 & 3 \\ 3 & 3 \end{bmatrix}

,

\langle\mathbf{v},\mathbf{v}\rangle = \mathbf{v}^\top A(3)\,\mathbf{v} = (1,-1)\begin{bmatrix} 3 & 3 \\ 3 & 3 \end{bmatrix}\begin{bmatrix} 1 \\ -1 \end{bmatrix} = (1,-1)\cdot(0,0) = 0,

even though

\mathbf{v}\neq\mathbf{0}

. So

t = 3

does not give an inner product.

Answer. (a) Positive definite for

t > 3

; positive semidefinite at

t = 3

(

\lambda = 6, 0

); indefinite for

-3 < t < 3

; negative semidefinite at

t = -3

(

\lambda = 0, -6

); negative definite for

t < -3

. (b) Valid inner product

\iff t > 3

; at

t = 3

,

\mathbf{v} = (1,-1)^\top \neq \mathbf{0}

has

\langle\mathbf{v},\mathbf{v}\rangle = 0

.

Watch out

Semidefinite is not "almost good enough": at $t = 3$ the axiom that dies is exactly the strict inequality $\langle\mathbf{v},\mathbf{v}\rangle > 0$ for $\mathbf{v}\neq\mathbf{0}$ . A geometry in which a nonzero vector has length zero cannot measure distances — so the borderline value is excluded, not included.

Key takeaway

For a symmetric matrix, definiteness = the sign pattern of its eigenvalues, full stop. For the pattern $\begin{bmatrix} t & c \\ c & t \end{bmatrix}$ the eigenvalues are always $t + c$ and $t - c$ — diagonal plus-or-minus off-diagonal — which makes the whole classification a one-line read-off.

Wait — really?

One dial sweeps all five classes — and the eigenvectors never move. For every $t$ , $A(t)\,(1,1)^\top = (t+3)(1,1)^\top$ and $A(t)\,(1,-1)^\top = (t-3)(1,-1)^\top$ . Turning $t$ just slides the two eigenvalues $t \pm 3$ along the number line past zero, while the axes of the quadratic form stay frozen at $45^\circ$ .

Problem 6Hard

(Positive definiteness over $\mathbb{C}$ .) For a real parameter $k$ , consider $A(k) = \begin{bmatrix} k & 1+i \\ 1-i & 2 \end{bmatrix}.$ For $\mathbf{u},\mathbf{v}\in\mathbb{C}^2$ , let $\mathbf{u}^*$ denote the conjugate transpose and set $\langle\mathbf{u},\mathbf{v}\rangle = \mathbf{u}^* A(k)\,\mathbf{v}$ .
(a) Show $A(k)$ is Hermitian for every real $k$ . [1 M]
(b) Find all real $k$ for which $A(k)$ is positive definite — equivalently, for which $\langle\mathbf{u},\mathbf{v}\rangle = \mathbf{u}^* A(k)\,\mathbf{v}$ is a valid inner product on $\mathbb{C}^2$ . [2 M]
(c) At the borderline value $k = 1$ the matrix is only positive semidefinite. Find a nonzero vector $\mathbf{z}\in\mathbb{C}^2$ with $\mathbf{z}^* A(1)\,\mathbf{z} = 0$ , showing the form degenerates there. [2 M] (5 marks)

Show the worked solution — every step

Step 1 · Hermitian check (a). A matrix is Hermitian when

A^* = A

, i.e. the diagonal entries are real and the two off-diagonal entries are complex conjugates of each other. Here the diagonal is

k,\,2

(real, because

k

is real) and the off-diagonals satisfy

\overline{1+i} = 1-i

. Hence

A(k)^* = A(k)

for every real

k

.

Step 2 · Sylvester over $\mathbb{C}$ (b). Sylvester's criterion also works for Hermitian matrices: positive definite

\iff

all leading principal minors are positive (they come out real automatically). The minors are

D_1 = k, \qquad D_2 = \det A(k) = 2k - (1+i)(1-i).

Step 3 · The crucial complex product. It is a squared modulus:

(1+i)(1-i) = 1 - i^2 = 2 = |1+i|^2, \qquad\text{so}\qquad D_2 = 2k - 2 = 2(k-1).

Step 4 · Solve the inequalities. Requiring

D_1 > 0

and

D_2 > 0

gives

k > 0

and

k > 1

, so

A(k) \ \text{is positive definite} \iff k > 1.

Step 5 · The degenerate direction at $k = 1$ (c). At

k = 1

we have

D_2 = 0

, so

A(1) = \begin{bmatrix} 1 & 1+i \\ 1-i & 2 \end{bmatrix}

is singular (

D_1 = 1 > 0

,

D_2 = 0

: positive semidefinite). Idea: any nonzero null-space vector works, since

A(1)\mathbf{z} = \mathbf{0} \Rightarrow \mathbf{z}^* A(1)\,\mathbf{z} = \mathbf{z}^*\mathbf{0} = 0

. From the first row,

z_1 + (1+i)z_2 = 0

; choosing

z_2 = 1

gives

z_1 = -(1+i)

, so

\mathbf{z} = \begin{bmatrix} -(1+i) \\ 1 \end{bmatrix}.

Step 6 · Verify it is a null vector. Using

(1-i)(1+i) = 2

:

A(1)\,\mathbf{z} = \begin{bmatrix} 1\cdot\big({-(1+i)}\big) + (1+i)\cdot 1 \\ (1-i)\cdot\big({-(1+i)}\big) + 2\cdot 1 \end{bmatrix} = \begin{bmatrix} 0 \\ -2+2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}.

Hence

\mathbf{z}^* A(1)\,\mathbf{z} = 0

with

\mathbf{z} \neq \mathbf{0}

: at

k = 1

the form is not positive definite, so it is not an inner product.

Answer. (a)

A(k)^* = A(k)

for all real

k

. (b)

A(k)

is positive definite

\iff k > 1

. (c)

\mathbf{z} = (-(1+i),\,1)^\top \neq \mathbf{0}

satisfies

A(1)\mathbf{z} = \mathbf{0}

, hence

\mathbf{z}^* A(1)\,\mathbf{z} = 0

.

Watch out

Over $\mathbb{C}$ , transpose is not enough — use the conjugate transpose $\mathbf{u}^*$ . And the off-diagonal product is $(1+i)(1-i) = |1+i|^2 = 2$ , not $(1+i)^2 = 2i$ . Writing $D_2 = 2k - 2i$ (a complex determinant for a Hermitian matrix — impossible) is the classic slip.

Key takeaway

Hermitian + positive definite is the complex analogue of symmetric + positive definite: Sylvester's criterion still runs, the leading minors are automatically real, and only $k > 1$ — both minors strictly positive — buys a genuine inner product on $\mathbb{C}^2$ .

Wait — really?

At $k = 1$ there exists a nonzero vector of length zero: $\langle\mathbf{z},\mathbf{z}\rangle = 0$ although $\mathbf{z} \neq \mathbf{0}$ . The geometry has a direction it simply cannot see — which is precisely the disease the positive-definiteness axiom exists to rule out.

Problem 7Medium

For each matrix $M$ below, decide whether $\langle\mathbf{x},\mathbf{y}\rangle = \mathbf{x}^\top M\,\mathbf{y}$ defines a valid inner product on $\mathbb{R}^2$ . Justify using the three requirements (bilinear, symmetric, positive-definite); when it fails, exhibit a witness. [2 M each] $\text{(a)}\ M = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix} \qquad \text{(b)}\ M = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix} \qquad \text{(c)}\ M = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}$ (6 marks)

Show the worked solution — every step

Step 1 · Bilinearity is free. Every

\mathbf{x}^\top M\,\mathbf{y}

is automatically bilinear, so only symmetry and positive-definiteness can fail.

Step 2 · (a) Symmetric but indefinite.

M

is symmetric, so symmetry holds. Check positive-definiteness by Sylvester:

D_1 = 1 > 0

but

D_2 = \det M = 1 - 4 = -3 < 0

. Fails. A witness:

\mathbf{x} = (1,-1)^\top

gives

\mathbf{x}^\top M\,\mathbf{x} = (1,-1)\begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\ -1 \end{bmatrix} = (1,-1)\cdot(-1,1) = -2 < 0.

So

\langle\mathbf{x},\mathbf{x}\rangle < 0

is possible — not an inner product.

Step 3 · (b) All three requirements hold.

M

is symmetric. Sylvester:

D_1 = 2 > 0

and

D_2 = \det M = 6 - 1 = 5 > 0

, so

M

is positive definite. All three requirements hold — this is a valid inner product.

Step 4 · (c) Symmetry fails.

M

is not symmetric, and that breaks the symmetry requirement directly. Test on the basis vectors:

\langle\mathbf{e}_1,\mathbf{e}_2\rangle = \mathbf{e}_1^\top M\,\mathbf{e}_2 = M_{12} = 1, \qquad \langle\mathbf{e}_2,\mathbf{e}_1\rangle = \mathbf{e}_2^\top M\,\mathbf{e}_1 = M_{21} = 0.

Since

\langle\mathbf{e}_1,\mathbf{e}_2\rangle \neq \langle\mathbf{e}_2,\mathbf{e}_1\rangle

, symmetry fails — not an inner product (even though the quadratic form

\mathbf{x}^\top M\,\mathbf{x} = x_1^2 + x_1x_2 + x_2^2

happens to be positive; symmetry alone already rules it out).

Answer. (a) Not an inner product — indefinite; witness

\mathbf{x} = (1,-1)^\top

with

\mathbf{x}^\top M\,\mathbf{x} = -2 < 0

. (b) Valid inner product — symmetric with

D_1 = 2 > 0

,

D_2 = 5 > 0

. (c) Not an inner product — symmetry fails:

\langle\mathbf{e}_1,\mathbf{e}_2\rangle = 1 \neq 0 = \langle\mathbf{e}_2,\mathbf{e}_1\rangle

.

Watch out

$D_1 > 0$ alone proves nothing — in (a) the first minor passes and the matrix is still indefinite; Sylvester needs all leading minors positive. And check symmetry before reaching for Sylvester: the criterion is only meaningful for symmetric matrices.

Key takeaway

The three-point checklist, cheapest first: bilinearity is automatic for $\mathbf{x}^\top M\,\mathbf{y}$ ; then verify symmetry $M^\top = M$ ; only then test positive definiteness (Sylvester or eigenvalue signs). Failing any single one ends the game.

Wait — really?

Matrix (c) satisfies $\mathbf{x}^\top M\,\mathbf{x} = x_1^2 + x_1x_2 + x_2^2 = \left(x_1 + \tfrac{x_2}{2}\right)^2 + \tfrac{3}{4}x_2^2 > 0$ for every $\mathbf{x} \neq \mathbf{0}$ — genuinely positive "lengths" — and it is still not an inner product. Positivity cannot buy back broken symmetry: with this $M$ , the "angle between $\mathbf{x}$ and $\mathbf{y}$ " would depend on the order in which you name them.

Problem 8Medium

On $\mathbb{R}^2$ use $\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top A\,\mathbf{y}$ with $A = \begin{bmatrix} 3 & 1 \\ 1 & 2 \end{bmatrix}$ (SPD), and let $\mathbf{x} = (1,1)^\top$ , $\mathbf{y} = (1,-1)^\top$ .
(a) Compute $\|\mathbf{x}\|_A$ , $\|\mathbf{y}\|_A$ , and $\langle\mathbf{x},\mathbf{y}\rangle_A$ . [2 M]
(b) Verify the Cauchy–Schwarz inequality $|\langle\mathbf{x},\mathbf{y}\rangle_A| \le \|\mathbf{x}\|_A\,\|\mathbf{y}\|_A$ . [1 M]
(c) Compute the distance $d(\mathbf{x},\mathbf{y}) = \|\mathbf{x}-\mathbf{y}\|_A$ and verify the triangle inequality $\|\mathbf{x}-\mathbf{y}\|_A \le \|\mathbf{x}\|_A + \|\mathbf{y}\|_A$ . [2 M] (5 marks)

Show the worked solution — every step

Step 1 · Two matrix–vector products (a). Compute

A\mathbf{x}

and

A\mathbf{y}

once:

A\mathbf{x} = A(1,1)^\top = (3+1,\ 1+2)^\top = (4,3)^\top, \qquad A\mathbf{y} = A(1,-1)^\top = (3-1,\ 1-2)^\top = (2,-1)^\top.

Step 2 · Norms and inner product.

\|\mathbf{x}\|_A^2 = \mathbf{x}^\top(A\mathbf{x}) = (1,1)\cdot(4,3) = 7, \qquad \|\mathbf{y}\|_A^2 = \mathbf{y}^\top(A\mathbf{y}) = (1,-1)\cdot(2,-1) = 2 + 1 = 3,

\langle\mathbf{x},\mathbf{y}\rangle_A = \mathbf{x}^\top(A\mathbf{y}) = (1,1)\cdot(2,-1) = 1.

So

\|\mathbf{x}\|_A = \sqrt{7}

,

\|\mathbf{y}\|_A = \sqrt{3}

,

\langle\mathbf{x},\mathbf{y}\rangle_A = 1

.

Step 3 · Cauchy–Schwarz (b).

|\langle\mathbf{x},\mathbf{y}\rangle_A| = 1 \ \le\ \|\mathbf{x}\|_A\,\|\mathbf{y}\|_A = \sqrt{7}\cdot\sqrt{3} = \sqrt{21} \approx 4.58. \;\checkmark

Step 4 · Distance and triangle inequality (c). Distance uses the same norm on the difference. With

\mathbf{x}-\mathbf{y} = (0,2)^\top

and

A(0,2)^\top = (2,4)^\top

:

\|\mathbf{x}-\mathbf{y}\|_A^2 = (0,2)\cdot(2,4) = 8 \;\Longrightarrow\; d(\mathbf{x},\mathbf{y}) = \sqrt{8} = 2\sqrt{2} \approx 2.83.

Triangle inequality:

\|\mathbf{x}-\mathbf{y}\|_A = 2\sqrt{2} \approx 2.83 \ \le\ \|\mathbf{x}\|_A + \|\mathbf{y}\|_A = \sqrt{7} + \sqrt{3} \approx 4.38. \;\checkmark

Answer. (a)

\|\mathbf{x}\|_A = \sqrt{7}

,

\|\mathbf{y}\|_A = \sqrt{3}

,

\langle\mathbf{x},\mathbf{y}\rangle_A = 1

. (b)

1 \le \sqrt{21} \approx 4.58

✓. (c)

d(\mathbf{x},\mathbf{y}) = 2\sqrt{2} \approx 2.83 \le \sqrt{7}+\sqrt{3} \approx 4.38

✓.

Watch out

The distance must be measured with $A$ on the difference: $(\mathbf{x}-\mathbf{y})^\top A\,(\mathbf{x}-\mathbf{y}) = 8$ , so $d = 2\sqrt{2}$ — not the Euclidean $\|(0,2)^\top\| = 2$ . Every length in this problem, the distance included, is an $A$ -length.

Key takeaway

Cauchy–Schwarz and the triangle inequality are theorems of every inner-product space: if $A$ is genuinely SPD they cannot fail. So "verifying" them is really a sanity check on your arithmetic — if such a check ever fails, either the computation or the claimed positive-definiteness is wrong.

Wait — really?

In the plain dot product these two vectors are orthogonal — $\mathbf{x}^\top\mathbf{y} = 1 - 1 = 0$ — yet under $A$ their inner product is $1$ , giving $\cos\theta_A = 1/\sqrt{21} \approx 0.218$ : an angle of about $77.4^\circ$ . The matrix $A$ literally bends the right angle.

Next up

Unit 4 · Determinants, Eigenvalues & the Spectral Theorem →

With lengths and angles in hand, Unit 4 goes looking for a matrix's fingerprints: the determinant as a volume dial, the directions a matrix cannot turn, and the theorem that lets a symmetric matrix be rebuilt as turn · stretch · turn back.

← All units