Analytic Geometry

Your school reflex says: straight-line (Euclidean) distance. Perfectly valid — and just one option among many. If "director" matters more than "runtime" when comparing movies, you'd want a measure that weights the director dimension more heavily. The standard ruler isn't wrong; it's merely default. This unit is about building custom rulers — safely, so that geometry never breaks.

The route: ① pin down what any notion of length must satisfy (norms); ② reverse-engineer the dot product into a blueprint and generalize it (inner products); ③ meet the engine that powers every legal ruler (symmetric positive-definite matrices); ④ collect the payoff — distances, angles, orthogonality, and the cleanest coordinate systems in mathematics (orthonormal bases, via Gram–Schmidt).

A norm is any function $\|\cdot\|:V\to\mathbb{R}$ that behaves the way length must. Three non-negotiables: absolute homogeneity $\|\lambda\mathbf{x}\|=|\lambda|\,\|\mathbf{x}\|$ (double the vector, double the length — flipping sign changes nothing); the triangle inequality $\|\mathbf{x}+\mathbf{y}\|\le\|\mathbf{x}\|+\|\mathbf{y}\|$ (no detour is shorter than the direct route); and positive definiteness $\|\mathbf{x}\|\ge 0$, with $\|\mathbf{x}\|=0$ only for $\mathbf{x}=\mathbf{0}$ (nothing real has negative size; only nothing has size zero).

Two rulers you already own:

The dot product $\mathbf{x}^T\mathbf{y}=\sum_i x_iy_i$ looks like mere arithmetic, but it's secretly an alignment meter: positive when vectors point the same general way, negative when opposed, zero when perpendicular. Take it apart and exactly three load-bearing properties appear. Bilinearity: linear in each slot — $\Omega(\lambda\mathbf{x}+\psi\mathbf{y},\mathbf{z})=\lambda\,\Omega(\mathbf{x},\mathbf{z})+\psi\,\Omega(\mathbf{y},\mathbf{z})$, and the same in the second argument. Symmetry: $\Omega(\mathbf{x},\mathbf{y})=\Omega(\mathbf{y},\mathbf{x})$ — x's similarity to y is y's similarity to x. Positive definiteness: $\Omega(\mathbf{x},\mathbf{x})>0$ for every $\mathbf{x}\neq\mathbf{0}$ — self-similarity is positive, because it's about to play the role of squared length.

Here's the master recipe. Notice first that the humble dot product is a matrix sandwich in disguise: $\mathbf{x}\cdot\mathbf{y}=\mathbf{x}^T I\,\mathbf{y}$ — the identity matrix is its engine. Now swap engines. The slides' theorem: on a finite-dimensional space with a chosen basis, $\langle\cdot,\cdot\rangle$ is an inner product iff there is a symmetric, positive-definite (SPD) matrix $A$ with

Symmetric: $A=A^T$ (inherits the symmetry axiom). Positive-definite: $\mathbf{x}^TA\mathbf{x}>0$ for all $\mathbf{x}\neq\mathbf{0}$ (inherits the length axiom). Two facts fall out immediately, both from Unit 1's toolbox. SPD ⇒ full rank: $\mathbf{x}^TA\mathbf{x}>0$ for every non-zero $\mathbf{x}$ means no non-zero vector is allowed in the null space — the blind spot is $\{\mathbf{0}\}$, so $A$ is invertible. Diagonals are positive: feed the canonical basis vector $\mathbf{e}_i$ into both slots: $\mathbf{e}_i^TA\mathbf{e}_i=A_{ii}>0$.

What does the engine do, geometrically? It reshapes space's sense of size. Watch the fingerprint — the unit ball — as you tune the engine:

Every inner product ships with a free ruler: the induced norm $\|\mathbf{x}\|=\sqrt{\langle\mathbf{x},\mathbf{x}\rangle}$. (The traffic doesn't flow backwards: the Manhattan norm is a perfectly legal norm that comes from no inner product — inner-product geometry is the smoother, "round-ball" subfamily.) From the norm comes a distance: $d(\mathbf{x},\mathbf{y})=\|\mathbf{x}-\mathbf{y}\|=\sqrt{\langle\mathbf{x}-\mathbf{y},\,\mathbf{x}-\mathbf{y}\rangle}$ — with the dot product, that's the Euclidean distance. A distance function (a metric) keeps four promises: $d\ge 0$; $d(\mathbf{x},\mathbf{y})=0\iff\mathbf{x}=\mathbf{y}$; symmetry; and the triangle inequality $d(\mathbf{x},\mathbf{z})\le d(\mathbf{x},\mathbf{y})+d(\mathbf{y},\mathbf{z})$.

And now the inequality the whole unit leans on — Cauchy–Schwarz:

Alignment can never exceed the product of the sizes. It's the fact that makes "the angle between two vectors" a legal phrase — coming up next.

Cauchy–Schwarz says the ratio below always lands in $[-1,1]$, so there is a unique angle $\omega\in[0,\pi]$ with

— and this number, cosine similarity, is ML's favorite similarity dial: +1 same direction, 0 unrelated (perpendicular), −1 opposite. It measures orientation agreement while ignoring size — which is exactly why search engines compare a three-word query to a thousand-word document with it. Two vectors with $\langle\mathbf{x},\mathbf{y}\rangle=0$ are orthogonal, written $\mathbf{x}\perp\mathbf{y}$ (the zero vector, ever the freeloader, is orthogonal to everything).

The slides end one page with a dare: pick two vectors at random in high dimensions — what happens to the angle between them? Write a small program to see… The program is written; it's below. Sample pairs of random vectors (each coordinate an independent Gaussian), compute their cosine similarity, and pile up a histogram. Then drag the dimension slider and watch a law of nature emerge.

Here's the unit's sneakiest lesson, straight from the slides. Take $\mathbf{x}=(1,1)$ and $\mathbf{y}=(-1,1)$. Under the dot product: $\langle\mathbf{x},\mathbf{y}\rangle=(1)(-1)+(1)(1)=0$ — perfectly orthogonal. Now switch the engine to $A=\begin{pmatrix}2&0\\0&1\end{pmatrix}$:

Same two arrows, new ruler — and they are no longer perpendicular (the angle is about $109.5^\circ$). Nothing moved except our definition of measurement. If you haven't already, hit the diag(2, 1) preset in Section 4's lens and watch the unit circle deform while the verdict flips. The deep point: "perpendicular" is not a property of two vectors; it's a property of two vectors plus a ruler. Choose the ruler to match the problem — that's feature weighting, Mahalanobis distance, and attention temperature, all in embryo.

A square matrix whose columns are orthonormal (mutually $\perp$, each of unit length) is called an orthogonal matrix, and it earns the cleanest identity in the subject:

The inverse is free — just the transpose. (And if the columns are orthonormal, so are the rows: for square matrices a left inverse is automatically the right inverse — if $BA=I=AC$ then $B=B(AC)=(BA)C=C$.) The payoff is behavioral: these transformations cannot distort measurement. Lengths survive: $\|A\mathbf{x}\|^2=(A\mathbf{x})^TA\mathbf{x}=\mathbf{x}^TA^TA\mathbf{x}=\mathbf{x}^T\mathbf{x}=\|\mathbf{x}\|^2$. Angles survive by the same cancellation inside $\cos\omega$. They are geometry's rigid motions — rotations and reflections, like the rotation matrix $\begin{pmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{pmatrix}$ — and everything else (shears, stretches) is a liar that bends the ruler as it moves things.

The formula falls out of one demand: the leftover must be orthogonal to $\mathbf{b}$. Write the shadow as $\alpha\mathbf{b}$ and require $\langle\mathbf{a}-\alpha\mathbf{b},\,\mathbf{b}\rangle=0$; linearity gives $\alpha=\frac{\langle\mathbf{a},\mathbf{b}\rangle}{\langle\mathbf{b},\mathbf{b}\rangle}$, so

(You met $\alpha$ already — it's the "smartest choice" from the Cauchy–Schwarz proof. Same coefficient, same geometry.)

Unit 2 taught you to find a basis. This unit's finale upgrades it to the best kind: an orthonormal basis $\{\mathbf{e}_1,\dots,\mathbf{e}_n\}$ with $\langle\mathbf{e}_i,\mathbf{e}_j\rangle=0$ for $i\neq j$ and $\langle\mathbf{e}_i,\mathbf{e}_i\rangle=1$ — every direction perpendicular, every direction unit length. (Only orthogonal, not unit length? That's an orthogonal basis — one normalization away.) Why chase it? Because in an orthonormal basis, coordinates stop requiring equation-solving: the address of any $\mathbf{v}$ is just $\lambda_i=\langle\mathbf{v},\mathbf{e}_i\rangle$ — read off by inner product. The canonical basis of $\mathbb{R}^n$ is one example; rotate it and you get infinitely many others.

The cleaning algorithm is the shadow machine on repeat: take your messy basis in order; normalize the first vector; from each later vector, subtract its shadows on everything already cleaned, then normalize what's left. Step through the companion's example in 3D:

① A norm is any length that scales, obeys the triangle inequality, and vanishes only at $\mathbf{0}$ — the taxi's ℓ₁ and the crow's ℓ₂ both qualify. ② An inner product is the dot product's blueprint made general — bilinear, symmetric, positive-definite — and every SPD matrix $A$ builds one: $\mathbf{x}^TA\mathbf{y}$. ③ Each inner product induces a norm, a distance, and — thanks to Cauchy–Schwarz — an angle: $\cos\omega=\langle\mathbf{x},\mathbf{y}\rangle/\|\mathbf{x}\|\|\mathbf{y}\|$, ML's similarity dial. ④ Orthogonality is relative to the ruler, orthonormal matrices are the motions that preserve every measurement, and random high-dimensional vectors are nearly orthogonal by default. ⑤ Gram–Schmidt (shadow-subtraction, or elimination on $[A^TA|A^T]$) turns any basis into an orthonormal one — the cleanest grid a geometry can have.